| *** spzala has quit IRC | 00:00 | |
| *** spzala has joined #openstack-infra | 00:00 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add support for requiring github pr head status https://review.openstack.org/449390 | 00:01 |
|---|---|---|
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Adds github triggering from status updates https://review.openstack.org/453844 | 00:01 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement pipeline requirement on github reviews https://review.openstack.org/453845 | 00:01 |
| *** nicolasbock has quit IRC | 00:03 | |
| *** Apoorva_ has joined #openstack-infra | 00:04 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] use networkx https://review.openstack.org/467419 | 00:04 |
| *** spzala has quit IRC | 00:05 | |
| *** jaypipes has quit IRC | 00:05 | |
| *** wolverineav has joined #openstack-infra | 00:05 | |
| *** esikache1 has joined #openstack-infra | 00:06 | |
| *** Apoorva has quit IRC | 00:07 | |
| *** sflanigan has joined #openstack-infra | 00:08 | |
| *** sflanigan has joined #openstack-infra | 00:08 | |
| fungi | thanks again cloudnull!!! | 00:08 |
| fungi | and we're up to 50gb free for logs now | 00:08 |
| *** dingyichen has joined #openstack-infra | 00:09 | |
| *** esikache1 has quit IRC | 00:10 | |
| clarkb | mordred: will you approve the second shade change? | 00:12 |
| clarkb | mordred: then we can try to get a release out in the morning? | 00:12 |
| *** esikache1 has joined #openstack-infra | 00:13 | |
| *** salv-orlando has quit IRC | 00:16 | |
| *** dave-mccowan has quit IRC | 00:17 | |
| *** esikache1 has quit IRC | 00:17 | |
| mtreinish | fungi: well my first attempt was less than successful: http://paste.openstack.org/show/610507/ | 00:20 |
| mtreinish | when I find the time, I'll have to dive in on figuring out where paho and websockify aren't speaking the same language | 00:20 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] use networkx https://review.openstack.org/467419 | 00:22 |
| *** Apoorva_ has quit IRC | 00:23 | |
| *** salv-orlando has joined #openstack-infra | 00:29 | |
| *** spzala has joined #openstack-infra | 00:31 | |
| *** dimtruck is now known as zz_dimtruck | 00:33 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add openstack/ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 00:33 |
| *** marst_ has joined #openstack-infra | 00:34 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add openstack/ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 00:34 |
| *** kaisers has quit IRC | 00:36 | |
| *** spzala has quit IRC | 00:36 | |
| *** spzala has joined #openstack-infra | 00:36 | |
| *** esikache1 has joined #openstack-infra | 00:38 | |
| *** kaisers has joined #openstack-infra | 00:43 | |
| *** esikache1 has quit IRC | 00:43 | |
| *** zz_dimtruck is now known as dimtruck | 00:44 | |
| *** ihrachys|afk has left #openstack-infra | 00:46 | |
| *** dave-mccowan has joined #openstack-infra | 00:51 | |
| *** esikache1 has joined #openstack-infra | 00:51 | |
| *** cuongnv has joined #openstack-infra | 00:52 | |
| openstackgerrit | Merged openstack-infra/shade master: Allow a user to submit start and end time as strings https://review.openstack.org/467257 | 00:52 |
| *** Julien-zte has joined #openstack-infra | 00:54 | |
| *** dave-mccowan has quit IRC | 00:55 | |
| *** pvaneck has quit IRC | 01:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/467428 | 01:02 |
| openstackgerrit | Merged openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient" https://review.openstack.org/462144 | 01:02 |
| *** Julien-zte has quit IRC | 01:03 | |
| *** esikache1 has quit IRC | 01:04 | |
| *** Apoorva has joined #openstack-infra | 01:04 | |
| *** esikache1 has joined #openstack-infra | 01:09 | |
| *** wolverineav has quit IRC | 01:10 | |
| *** ijw_ has joined #openstack-infra | 01:10 | |
| *** yuanying_ has joined #openstack-infra | 01:13 | |
| *** yuanying has quit IRC | 01:13 | |
| *** ijw has quit IRC | 01:13 | |
| *** esikache1 has quit IRC | 01:14 | |
| *** Julien-zte has joined #openstack-infra | 01:16 | |
| *** Apoorva_ has joined #openstack-infra | 01:16 | |
| *** salv-orlando has quit IRC | 01:17 | |
| *** Apoorva has quit IRC | 01:19 | |
| *** salv-orlando has joined #openstack-infra | 01:20 | |
| fungi | 73gb free now | 01:20 |
| *** spzala has quit IRC | 01:21 | |
| *** Apoorva_ has quit IRC | 01:21 | |
| mmedvede | Is it possible to add more packages to http://tarballs.openstack.org/etcd/ ? | 01:24 |
| mmedvede | namely, for other arches? This https://review.openstack.org/#/c/466817 breaks everything other than x86 | 01:25 |
| *** gongysh has joined #openstack-infra | 01:28 | |
| *** hongbin has joined #openstack-infra | 01:28 | |
| clarkb | mmedvede: we only use x86-64, you should set your own value for oyher arches | 01:29 |
| mmedvede | right, I would, but the patch makes it hard to overrride the mirror | 01:29 |
| clarkb | you can use localrc passthrough for that or we cpuld make that properly configurable in d-f | 01:29 |
| clarkb | I do t think we eant to be a mirror for evwryone so wouldnt host packages for arches we dont use ourselves | 01:30 |
| *** jascott1 has joined #openstack-infra | 01:31 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/devstack-gate master: Allow overriding ETCD_DOWNLOAD_URL https://review.openstack.org/467437 | 01:35 |
| mmedvede | clarkb: ^ | 01:35 |
| *** jascott1 has quit IRC | 01:35 | |
| *** ramishra has joined #openstack-infra | 01:35 | |
| rm_work | hey, seeing problems with our multinode testing due to etcd service? | 01:36 |
| rm_work | http://logs.openstack.org/54/466454/9/check/gate-octavia-v1-dsvm-scenario-multinode/8445901/logs/subnode-2/devstacklog.txt.gz#_2017-05-24_00_35_59_061 | 01:36 |
| rm_work | any known bug here? | 01:36 |
| rm_work | is there something we have to fix? | 01:37 |
| rm_work | main node runs fine, second node fails with the above error | 01:37 |
| *** esikache1 has joined #openstack-infra | 01:39 | |
| clarkb | mmedvede: couppe minor things on review | 01:39 |
| clarkb | rm_work: best guess is etcd shouldnt run on subnode and attempting to is breaking as a result | 01:41 |
| *** dhinesh has quit IRC | 01:41 | |
| rm_work | clarkb: so do WE need to disable that somehow? | 01:41 |
| rm_work | it's not like we're explicitly enabling it, since it just appeared under us suddenly :P | 01:41 |
| clarkb | its not in the nabled services list so likely devstack bug | 01:42 |
| rm_work | (this is a gate blocker for Octavia BTW) | 01:42 |
| *** priteau has joined #openstack-infra | 01:42 | |
| *** baoli has joined #openstack-infra | 01:42 | |
| clarkb | dims us your best bet | 01:42 |
| *** esikache1 has quit IRC | 01:43 | |
| *** salv-orlando has quit IRC | 01:45 | |
| rm_work | OK, poking people in #openstack-qa | 01:45 |
| *** liujiong has joined #openstack-infra | 01:46 | |
| *** priteau has quit IRC | 01:46 | |
| *** baoli has quit IRC | 01:46 | |
| *** spzala has joined #openstack-infra | 01:47 | |
| *** esikache1 has joined #openstack-infra | 01:51 | |
| rm_work | doing this temporarily (I think this should work?) https://review.openstack.org/#/c/467440/ | 01:51 |
| rm_work | clarkb: tell me if that seems sane to you | 01:51 |
| rm_work | oh ffff missed a newline | 01:52 |
| *** spzala has quit IRC | 01:52 | |
| rm_work | ok PS2 | 01:52 |
| rm_work | augh no i'm dumb, PS3 | 01:54 |
| rm_work | glad I asked you, because it forced me to look again :P | 01:54 |
| *** SumitNaiksatam has joined #openstack-infra | 01:54 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/devstack-gate master: Allow overriding ETCD_DOWNLOAD_URL https://review.openstack.org/467437 | 01:55 |
| mmedvede | thanks clarkb, addressed your comments. Although it makes the etcd mirror config appear less of a temporary measure | 01:56 |
| *** yuanying_ has quit IRC | 01:57 | |
| fungi | 90gb free now | 01:57 |
| *** esikache1 has quit IRC | 01:59 | |
| *** ramishra has quit IRC | 02:00 | |
| *** adam_g has quit IRC | 02:00 | |
| *** yuanying has joined #openstack-infra | 02:01 | |
| *** adam_g has joined #openstack-infra | 02:04 | |
| openstackgerrit | gongysh proposed openstack-infra/irc-meetings master: Change tacker team meeting time https://review.openstack.org/467443 | 02:08 |
| *** zhurong has joined #openstack-infra | 02:11 | |
| *** baoli has joined #openstack-infra | 02:14 | |
| dims | rm_work : clarkb : fungi : filed https://review.openstack.org/467445 to stop the problem on multinode scenarios | 02:14 |
| *** jamesdenton has joined #openstack-infra | 02:14 | |
| *** sflanigan has quit IRC | 02:18 | |
| *** jamesdenton has quit IRC | 02:21 | |
| *** spzala has joined #openstack-infra | 02:25 | |
| fungi | thanks dims! looks reasonable to be, but i'm not on devstack-core | 02:25 |
| *** adam_g has quit IRC | 02:25 | |
| fungi | i wonder if devstack changes really don't have any multinode job running against them? | 02:25 |
| *** bobh has quit IRC | 02:26 | |
| *** adam_g has joined #openstack-infra | 02:26 | |
| *** bobh has joined #openstack-infra | 02:27 | |
| *** thorst_afk has joined #openstack-infra | 02:27 | |
| *** sflanigan has joined #openstack-infra | 02:30 | |
| *** nicolasbock has joined #openstack-infra | 02:31 | |
| *** gouthamr has quit IRC | 02:33 | |
| *** adam_g has quit IRC | 02:33 | |
| *** adam_g has joined #openstack-infra | 02:34 | |
| *** thorst_afk has quit IRC | 02:35 | |
| *** thorst_afk has joined #openstack-infra | 02:37 | |
| *** gildub has quit IRC | 02:42 | |
| *** masber has joined #openstack-infra | 02:45 | |
| *** marst_ has quit IRC | 02:49 | |
| *** yamamoto_ has joined #openstack-infra | 02:51 | |
| *** thorst_afk has quit IRC | 02:52 | |
| *** jamesmca_ has joined #openstack-infra | 02:54 | |
| *** esikache1 has joined #openstack-infra | 02:57 | |
| *** gildub has joined #openstack-infra | 02:58 | |
| *** jamesmca_ has quit IRC | 02:59 | |
| *** yamahata has quit IRC | 03:00 | |
| *** spzala has quit IRC | 03:00 | |
| *** esikache1 has quit IRC | 03:02 | |
| *** s-shiono has quit IRC | 03:05 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Ensure PRs arent rejected for stale negative reviews https://review.openstack.org/460700 | 03:07 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Represent github change ID in status page by PR number https://review.openstack.org/460716 | 03:09 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Comment on PRs if a remote call to merge a change failed https://review.openstack.org/460762 | 03:10 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include exc_info in reporter failure https://review.openstack.org/460765 | 03:11 |
| *** mriedem has quit IRC | 03:11 | |
| *** Julien-zte has quit IRC | 03:11 | |
| *** aeng has joined #openstack-infra | 03:12 | |
| *** bobh has quit IRC | 03:15 | |
| *** esikache1 has joined #openstack-infra | 03:16 | |
| *** bobh has joined #openstack-infra | 03:16 | |
| *** cody-somerville has joined #openstack-infra | 03:18 | |
| *** cody-somerville has joined #openstack-infra | 03:18 | |
| *** csomerville has quit IRC | 03:18 | |
| *** Julien-zte has joined #openstack-infra | 03:19 | |
| *** bobh has quit IRC | 03:19 | |
| *** esikache1 has quit IRC | 03:22 | |
| *** sflanigan has quit IRC | 03:30 | |
| *** spzala has joined #openstack-infra | 03:31 | |
| *** vikrant has joined #openstack-infra | 03:35 | |
| *** spzala has quit IRC | 03:36 | |
| *** krtaylor has joined #openstack-infra | 03:39 | |
| *** baoli has quit IRC | 03:39 | |
| *** boris-42_ has quit IRC | 03:39 | |
| *** nicolasbock has quit IRC | 03:42 | |
| *** sflanigan has joined #openstack-infra | 03:43 | |
| *** sflanigan has joined #openstack-infra | 03:43 | |
| *** hongbin has quit IRC | 03:44 | |
| *** spzala has joined #openstack-infra | 03:46 | |
| *** esikache1 has joined #openstack-infra | 03:46 | |
| *** aeng has quit IRC | 03:48 | |
| *** spzala has quit IRC | 03:50 | |
| *** esikache1 has quit IRC | 03:51 | |
| *** kornicameister has joined #openstack-infra | 03:51 | |
| *** ykarel has joined #openstack-infra | 03:52 | |
| *** esikache1 has joined #openstack-infra | 03:56 | |
| *** winggundamth has joined #openstack-infra | 03:57 | |
| *** stakeda has joined #openstack-infra | 03:59 | |
| *** jamesmca_ has joined #openstack-infra | 04:00 | |
| *** esikache1 has quit IRC | 04:00 | |
| *** jamesmca_ has quit IRC | 04:01 | |
| *** jamesmca_ has joined #openstack-infra | 04:01 | |
| *** jamesmca_ has quit IRC | 04:04 | |
| fungi | #status ok Sufficient free space has been reclaimed that jobs are passing again; any POST_FAILURE results can now be rechecked. | 04:04 |
| openstackstatus | fungi: sending ok | 04:04 |
| fungi | up around 130gb free, so should be okay while i get some sleep | 04:05 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 04:06 | |
| -openstackstatus- NOTICE: Sufficient free space has been reclaimed that jobs are passing again; any POST_FAILURE results can now be rechecked. | 04:06 | |
| *** esikache1 has joined #openstack-infra | 04:07 | |
| *** psachin has joined #openstack-infra | 04:09 | |
| *** vhosakot has joined #openstack-infra | 04:09 | |
| openstackstatus | fungi: finished sending ok | 04:10 |
| *** ramishra has joined #openstack-infra | 04:10 | |
| *** zhurong has quit IRC | 04:11 | |
| *** vhosakot has quit IRC | 04:11 | |
| *** esikache1 has quit IRC | 04:11 | |
| *** zhurong has joined #openstack-infra | 04:15 | |
| *** yamamoto_ has quit IRC | 04:18 | |
| *** esikache1 has joined #openstack-infra | 04:18 | |
| *** e0ne has joined #openstack-infra | 04:19 | |
| *** gongysh has quit IRC | 04:20 | |
| *** esikache1 has quit IRC | 04:23 | |
| *** e0ne has quit IRC | 04:25 | |
| *** armax has quit IRC | 04:28 | |
| *** armax has joined #openstack-infra | 04:28 | |
| *** armax has quit IRC | 04:29 | |
| *** yamahata has joined #openstack-infra | 04:32 | |
| *** spzala has joined #openstack-infra | 04:32 | |
| *** psachin has quit IRC | 04:35 | |
| *** spzala has quit IRC | 04:37 | |
| *** csomerville has joined #openstack-infra | 04:46 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add cachecontrol to requests to github https://review.openstack.org/461587 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include exc_info in reporter failure https://review.openstack.org/460765 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Adds github triggering from status updates https://review.openstack.org/453844 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement github trigger requirement status https://review.openstack.org/463386 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement pipeline requirement on github reviews https://review.openstack.org/453845 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Ensure PRs arent rejected for stale negative reviews https://review.openstack.org/460700 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Represent github change ID in status page by PR number https://review.openstack.org/460716 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Comment on PRs if a remote call to merge a change failed https://review.openstack.org/460762 | 04:48 |
| *** cody-somerville has quit IRC | 04:48 | |
| *** gildub has quit IRC | 04:52 | |
| *** psachin has joined #openstack-infra | 04:52 | |
| *** jaosorior_away is now known as jaosorior | 04:55 | |
| *** yamamoto_ has joined #openstack-infra | 04:56 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use networkx for digraph https://review.openstack.org/467419 | 04:58 |
| *** flepied has joined #openstack-infra | 05:02 | |
| openstackgerrit | Rajat Sharma proposed openstack-infra/shade master: Replace assertRaisesRegexp with assertRaisesRegex https://review.openstack.org/467486 | 05:08 |
| *** gildub has joined #openstack-infra | 05:08 | |
| *** sree has joined #openstack-infra | 05:09 | |
| *** gongysh has joined #openstack-infra | 05:10 | |
| *** winggundamth has quit IRC | 05:12 | |
| *** yolkfull has joined #openstack-infra | 05:15 | |
| *** prateek__ has joined #openstack-infra | 05:17 | |
| *** gongysh has quit IRC | 05:18 | |
| *** bobh has joined #openstack-infra | 05:20 | |
| *** spzala has joined #openstack-infra | 05:20 | |
| *** links has joined #openstack-infra | 05:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set manifest permissions in the image https://review.openstack.org/466593 | 05:21 |
| *** gongysh has joined #openstack-infra | 05:25 | |
| *** bobh has quit IRC | 05:25 | |
| *** spzala has quit IRC | 05:25 | |
| *** esikache1 has joined #openstack-infra | 05:26 | |
| *** ltomasbo|away is now known as ltomasbo | 05:28 | |
| *** aeng has joined #openstack-infra | 05:29 | |
| *** esikache1 has quit IRC | 05:31 | |
| *** esikache1 has joined #openstack-infra | 05:32 | |
| *** isaacb has joined #openstack-infra | 05:33 | |
| *** dchen has joined #openstack-infra | 05:35 | |
| *** dimtruck is now known as zz_dimtruck | 05:35 | |
| *** dingyichen has quit IRC | 05:36 | |
| *** esikache1 has quit IRC | 05:36 | |
| *** sflanigan has quit IRC | 05:37 | |
| *** esikache1 has joined #openstack-infra | 05:37 | |
| *** gcb has quit IRC | 05:39 | |
| *** gildub has quit IRC | 05:39 | |
| *** gcb has joined #openstack-infra | 05:40 | |
| *** esikache1 has quit IRC | 05:41 | |
| *** flepied has quit IRC | 05:47 | |
| *** sflanigan has joined #openstack-infra | 05:49 | |
| *** sflanigan has joined #openstack-infra | 05:49 | |
| *** dingyichen has joined #openstack-infra | 05:49 | |
| *** spzala has joined #openstack-infra | 05:51 | |
| *** csomerville has quit IRC | 05:52 | |
| *** esikache1 has joined #openstack-infra | 05:53 | |
| *** psachin has quit IRC | 05:53 | |
| *** markus_z has joined #openstack-infra | 05:55 | |
| *** spzala has quit IRC | 05:56 | |
| *** esikache1 has quit IRC | 05:57 | |
| *** rcernin has quit IRC | 06:00 | |
| *** andreas_s has joined #openstack-infra | 06:03 | |
| *** psachin has joined #openstack-infra | 06:05 | |
| *** isaacb has quit IRC | 06:05 | |
| *** cshastri has joined #openstack-infra | 06:08 | |
| *** lewo has quit IRC | 06:16 | |
| *** lewo has joined #openstack-infra | 06:17 | |
| *** gyee has quit IRC | 06:17 | |
| *** rcernin has joined #openstack-infra | 06:17 | |
| *** lewo has quit IRC | 06:18 | |
| *** lewo has joined #openstack-infra | 06:18 | |
| *** psachin has quit IRC | 06:23 | |
| *** psachin has joined #openstack-infra | 06:23 | |
| *** gongysh has quit IRC | 06:31 | |
| *** spzala has joined #openstack-infra | 06:32 | |
| *** esikache1 has joined #openstack-infra | 06:32 | |
| *** mandre|away is now known as mandre | 06:35 | |
| *** spzala has quit IRC | 06:36 | |
| *** Hal has quit IRC | 06:37 | |
| *** florianf has joined #openstack-infra | 06:40 | |
| openstackgerrit | Nisha Agarwal proposed openstack-infra/project-config master: Add py35 experimental gates for ironic-lib https://review.openstack.org/462701 | 06:43 |
| *** dhajare has joined #openstack-infra | 06:43 | |
| *** gildub has joined #openstack-infra | 06:44 | |
| *** liujiong has quit IRC | 06:49 | |
| *** liujiong has joined #openstack-infra | 06:50 | |
| *** matrohon has joined #openstack-infra | 06:52 | |
| *** adam_g has quit IRC | 06:52 | |
| *** adam_g has joined #openstack-infra | 06:53 | |
| *** DangerousDaren has joined #openstack-infra | 06:56 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Set manifest permissions in the image https://review.openstack.org/466593 | 06:58 |
| *** rossella_ has joined #openstack-infra | 07:01 | |
| *** spzala has joined #openstack-infra | 07:01 | |
| *** gildub has quit IRC | 07:02 | |
| *** spzala has quit IRC | 07:06 | |
| *** gongysh has joined #openstack-infra | 07:09 | |
| *** dizquierdo_afk has joined #openstack-infra | 07:10 | |
| *** jroll has quit IRC | 07:12 | |
| *** dizquierdo_afk has quit IRC | 07:14 | |
| *** dizquierdo has joined #openstack-infra | 07:14 | |
| *** shardy has joined #openstack-infra | 07:14 | |
| *** yamahata has quit IRC | 07:15 | |
| *** pcaruana has joined #openstack-infra | 07:16 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** alexchadin has joined #openstack-infra | 07:32 | |
| *** spzala has joined #openstack-infra | 07:32 | |
| *** adam_g has quit IRC | 07:33 | |
| *** adam_g has joined #openstack-infra | 07:34 | |
| *** dtantsur|afk is now known as dtantsur | 07:36 | |
| *** oanson has quit IRC | 07:36 | |
| *** oanson has joined #openstack-infra | 07:37 | |
| *** spzala has quit IRC | 07:37 | |
| *** jpena|off is now known as jpena | 07:38 | |
| *** Hal has joined #openstack-infra | 07:45 | |
| *** aarefiev_afk is now known as aarefiev | 07:46 | |
| *** jpich has joined #openstack-infra | 07:48 | |
| *** sbezverk_ has joined #openstack-infra | 07:56 | |
| *** binbincong has joined #openstack-infra | 07:57 | |
| *** sbezverk has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** priteau has joined #openstack-infra | 08:03 | |
| *** tushar__ has quit IRC | 08:05 | |
| *** e0ne has joined #openstack-infra | 08:07 | |
| *** gildub has joined #openstack-infra | 08:08 | |
| *** isaacb has joined #openstack-infra | 08:10 | |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Disable rally jobs for stable/newton https://review.openstack.org/467515 | 08:16 |
| *** alexchadin has quit IRC | 08:17 | |
| *** spzala has joined #openstack-infra | 08:18 | |
| *** sambetts|afk is now known as sambetts | 08:20 | |
| *** ralonsoh has joined #openstack-infra | 08:23 | |
| *** alexchadin has joined #openstack-infra | 08:23 | |
| *** sflanigan has quit IRC | 08:23 | |
| *** spzala has quit IRC | 08:24 | |
| *** namnh has joined #openstack-infra | 08:25 | |
| *** Nilesh has joined #openstack-infra | 08:26 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change tacker team meeting time https://review.openstack.org/467443 | 08:26 |
| Nilesh | Hello All, Our CI is reporting failures due to it unable to connect to pgp.mit.edu | 08:27 |
| *** sree_ has joined #openstack-infra | 08:27 | |
| *** isaacb has quit IRC | 08:27 | |
| Nilesh | http://52.35.51.88/05/467505/1/check/dsvm-veritas-hyperscale-cinder-driver/6da8dd6/logs/devstacklog.txt.gz#_2017-05-24_07_52_00_422 | 08:28 |
| *** sree_ is now known as Guest25627 | 08:28 | |
| Nilesh | is anyone facing same issue ? | 08:28 |
| *** markmcd has joined #openstack-infra | 08:29 | |
| *** pbourke has quit IRC | 08:29 | |
| *** pbourke has joined #openstack-infra | 08:30 | |
| *** sree has quit IRC | 08:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Separate Puppet OpenStack tarball jobs from Python jobs https://review.openstack.org/467294 | 08:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable Python 2.7 checks for Almanach Client https://review.openstack.org/467265 | 08:34 |
| frickler | Nilesh: do you have outgoing filters defined for your servers? see https://bugs.launchpad.net/devstack/+bug/1693092 | 08:35 |
| openstack | Launchpad bug 1693092 in devstack "lib/etcd3:_install_etcd fails to receive key from keyserver on servers behind firewall" [Undecided,In progress] - Assigned to Sam Wan (sam-wan) | 08:35 |
| *** dtantsur is now known as dtantsur|lunch | 08:36 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Fix typos (34 vs 35) https://review.openstack.org/467523 | 08:37 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: grafana: Remove python34 stuff https://review.openstack.org/467524 | 08:37 |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** dingyichen has quit IRC | 08:39 | |
| *** dizquierdo has quit IRC | 08:40 | |
| *** Julien-zte has quit IRC | 08:44 | |
| *** Julien-zte has joined #openstack-infra | 08:44 | |
| *** stakeda has quit IRC | 08:45 | |
| *** Julien-zte has quit IRC | 08:48 | |
| *** Julien-zte has joined #openstack-infra | 08:49 | |
| *** dchen has quit IRC | 08:49 | |
| *** spzala has joined #openstack-infra | 08:50 | |
| *** gongysh has quit IRC | 08:54 | |
| *** spzala has quit IRC | 08:55 | |
| *** Julien-zte has quit IRC | 08:57 | |
| *** Julien-zte has joined #openstack-infra | 08:59 | |
| *** sree has joined #openstack-infra | 09:01 | |
| *** CynicGalahad has joined #openstack-infra | 09:01 | |
| *** Guest25627 has quit IRC | 09:03 | |
| openstackgerrit | Nisha Agarwal proposed openstack-infra/project-config master: [WIP] disable swift for py3 enablement in ironic https://review.openstack.org/467538 | 09:05 |
| *** electrofelix has joined #openstack-infra | 09:06 | |
| *** Julien-zte has quit IRC | 09:08 | |
| *** tinwood has quit IRC | 09:09 | |
| *** tinwood has joined #openstack-infra | 09:09 | |
| *** Julien-zte has joined #openstack-infra | 09:10 | |
| *** Julien-zte has quit IRC | 09:17 | |
| *** Julien-zte has joined #openstack-infra | 09:18 | |
| *** spzala has joined #openstack-infra | 09:21 | |
| *** spzala has quit IRC | 09:25 | |
| *** Julien-zte has quit IRC | 09:27 | |
| *** Julien-zte has joined #openstack-infra | 09:28 | |
| *** markmcd has quit IRC | 09:34 | |
| *** Julien-zte has quit IRC | 09:34 | |
| *** gongysh has joined #openstack-infra | 09:34 | |
| *** Julien-zte has joined #openstack-infra | 09:34 | |
| *** salv-orlando has joined #openstack-infra | 09:35 | |
| *** markmcd has joined #openstack-infra | 09:36 | |
| *** Julien-zte has quit IRC | 09:36 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Retire Gnocchi https://review.openstack.org/466320 | 09:37 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Update Gnocchi devstack URL https://review.openstack.org/466321 | 09:37 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Stop gating Gnocchi https://review.openstack.org/466318 | 09:37 |
| *** Julien-zte has joined #openstack-infra | 09:37 | |
| *** sdague has joined #openstack-infra | 09:44 | |
| *** hashar has joined #openstack-infra | 09:47 | |
| *** salv-orlando has quit IRC | 09:48 | |
| *** Douhet has quit IRC | 09:48 | |
| *** Douhet has joined #openstack-infra | 09:48 | |
| *** Julien-zte has quit IRC | 09:49 | |
| *** Julien-zte has joined #openstack-infra | 09:50 | |
| Nilesh | frickler : We need to wait till this fix ? | 09:52 |
| *** spzala has joined #openstack-infra | 09:52 | |
| Nilesh | https://bugs.launchpad.net/devstack/+bug/1693092 | 09:52 |
| openstack | Launchpad bug 1693092 in devstack "lib/etcd3:_install_etcd fails to receive key from keyserver on servers behind firewall" [Undecided,In progress] - Assigned to Sam Wan (sam-wan) | 09:52 |
| *** Julien-zte has quit IRC | 09:53 | |
| *** Julien-zte has joined #openstack-infra | 09:54 | |
| *** links has quit IRC | 09:55 | |
| *** tristanC has joined #openstack-infra | 09:56 | |
| *** spzala has quit IRC | 09:56 | |
| *** Julien-zte has quit IRC | 09:58 | |
| *** Julien-zte has joined #openstack-infra | 09:58 | |
| *** adriant has quit IRC | 09:58 | |
| frickler | Nilesh: well, you could also just fix the firewalling for your CI nodes. that might still be necessary after https://review.openstack.org/467480 is merged, depending on the details of your setup. you may also want to continue this discussion in #openstack-qa, as it is a devstack issue | 10:02 |
| *** matrohon has quit IRC | 10:04 | |
| *** zhurong has quit IRC | 10:04 | |
| *** gildub has quit IRC | 10:05 | |
| *** links has joined #openstack-infra | 10:08 | |
| *** cuongnv has quit IRC | 10:11 | |
| Nilesh | frickler : I did tried new commands on setup that worked fine for me. | 10:18 |
| Nilesh | frickler : I have one more question regarding the logs we stored for CI, | 10:19 |
| Nilesh | Is it okay to delete the older log files ? as our log server is getting full ? | 10:20 |
| *** jroll has joined #openstack-infra | 10:25 | |
| *** dtantsur|lunch is now known as dtantsur | 10:26 | |
| *** jkilpatr has quit IRC | 10:38 | |
| *** mat128 has joined #openstack-infra | 10:38 | |
| *** spzala has joined #openstack-infra | 10:38 | |
| *** liujiong has quit IRC | 10:40 | |
| *** Julien-zte has quit IRC | 10:40 | |
| *** Hal has quit IRC | 10:40 | |
| *** yamamoto_ has quit IRC | 10:43 | |
| *** spzala has quit IRC | 10:43 | |
| *** gongysh has quit IRC | 10:43 | |
| *** nicolasbock has joined #openstack-infra | 10:45 | |
| *** adam_g has quit IRC | 10:47 | |
| *** adam_g has joined #openstack-infra | 10:49 | |
| *** namnh has quit IRC | 10:51 | |
| *** jkilpatr has joined #openstack-infra | 10:56 | |
| *** flepied has joined #openstack-infra | 10:57 | |
| *** hashar has quit IRC | 10:59 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-dev/cookiecutter master: Update test-requirements.txt to latest https://review.openstack.org/467585 | 11:06 |
| *** rhallisey has joined #openstack-infra | 11:13 | |
| *** sree has quit IRC | 11:14 | |
| *** tinwood is now known as tinwood_afk | 11:16 | |
| *** lucasagomes is now known as lucas-pto | 11:17 | |
| *** yolkfull has quit IRC | 11:21 | |
| *** zz_dimtruck is now known as dimtruck | 11:23 | |
| *** bobh has joined #openstack-infra | 11:23 | |
| *** spzala has joined #openstack-infra | 11:24 | |
| *** bobh has quit IRC | 11:27 | |
| *** spzala has quit IRC | 11:29 | |
| *** esikache1 has quit IRC | 11:29 | |
| *** yamamoto has joined #openstack-infra | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| *** bobh has joined #openstack-infra | 11:34 | |
| *** adam_g has quit IRC | 11:41 | |
| *** nicolasbock has quit IRC | 11:43 | |
| *** esikachev has joined #openstack-infra | 11:45 | |
| *** Julien-zte has joined #openstack-infra | 11:46 | |
| *** hemna has quit IRC | 11:47 | |
| yolanda | hi, infra-roots . I wanted to cut a release of diskimage-builder, that wil include the latest changes to block device. Are you ok with that? | 11:48 |
| *** dansmith has quit IRC | 11:48 | |
| *** waynr has quit IRC | 11:49 | |
| *** zeus has quit IRC | 11:49 | |
| *** hemna has joined #openstack-infra | 11:49 | |
| *** dansmith has joined #openstack-infra | 11:51 | |
| *** dansmith is now known as Guest36154 | 11:51 | |
| *** waynr has joined #openstack-infra | 11:51 | |
| *** jaypipes has joined #openstack-infra | 11:52 | |
| *** zeus has joined #openstack-infra | 11:53 | |
| *** zeus is now known as Guest41638 | 11:53 | |
| *** mat128 has quit IRC | 11:58 | |
| *** hashar has joined #openstack-infra | 12:01 | |
| *** annegentle has joined #openstack-infra | 12:02 | |
| *** esikachev has quit IRC | 12:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Display command/shell results in a banner https://review.openstack.org/467603 | 12:05 |
| *** spzala has joined #openstack-infra | 12:11 | |
| *** esikachev has joined #openstack-infra | 12:14 | |
| *** trown|outtypewww is now known as trown | 12:15 | |
| *** spzala has quit IRC | 12:16 | |
| *** annegentle has quit IRC | 12:16 | |
| *** dprince has joined #openstack-infra | 12:18 | |
| *** Hal has joined #openstack-infra | 12:19 | |
| *** spzala has joined #openstack-infra | 12:22 | |
| *** esikachev has quit IRC | 12:23 | |
| *** esikachev has joined #openstack-infra | 12:23 | |
| *** krtaylor has quit IRC | 12:24 | |
| *** bobh has quit IRC | 12:25 | |
| *** esikachev has quit IRC | 12:28 | |
| *** wolverineav has joined #openstack-infra | 12:28 | |
| *** esikachev has joined #openstack-infra | 12:31 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 12:32 |
| *** HeOS has joined #openstack-infra | 12:32 | |
| *** rlandy has joined #openstack-infra | 12:34 | |
| *** zhurong has joined #openstack-infra | 12:35 | |
| *** esikachev has quit IRC | 12:35 | |
| *** zhurong has quit IRC | 12:36 | |
| *** tushar has joined #openstack-infra | 12:38 | |
| *** annegentle has joined #openstack-infra | 12:38 | |
| *** dave-mccowan has joined #openstack-infra | 12:41 | |
| *** rossella_ has quit IRC | 12:41 | |
| *** jpena is now known as jpena|lunch | 12:44 | |
| *** eharney has joined #openstack-infra | 12:45 | |
| *** ggnel_t has quit IRC | 12:46 | |
| *** annegentle has quit IRC | 12:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 12:47 |
| *** fguillot has joined #openstack-infra | 12:51 | |
| openstackgerrit | Merged openstack-infra/shade master: Pick most recent rather than first fixed address https://review.openstack.org/467385 | 12:52 |
| *** Goneri has joined #openstack-infra | 12:52 | |
| *** trown is now known as trown|brb | 12:56 | |
| *** gcb has quit IRC | 12:57 | |
| *** gcb has joined #openstack-infra | 12:57 | |
| *** jamesmca_ has joined #openstack-infra | 12:58 | |
| *** baoli has joined #openstack-infra | 12:58 | |
| *** kgiusti has joined #openstack-infra | 12:58 | |
| *** efoley has joined #openstack-infra | 13:00 | |
| *** nicolasbock has joined #openstack-infra | 13:00 | |
| *** mdrabe has joined #openstack-infra | 13:04 | |
| *** xyang1 has joined #openstack-infra | 13:04 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix failures in prepare ovb servers scripts https://review.openstack.org/467620 | 13:06 |
| *** rfolco has joined #openstack-infra | 13:06 | |
| fungi | yolanda: no objections from me. is this the first release which includes the fix for the permissions issues/information leakage vulnerability too? | 13:08 |
| *** jamesmca_ has quit IRC | 13:08 | |
| *** pradk has quit IRC | 13:11 | |
| *** tinwood_afk is now known as tinwood | 13:12 | |
| *** thorst_afk has joined #openstack-infra | 13:12 | |
| *** jamesmca_ has joined #openstack-infra | 13:14 | |
| *** rossella_ has joined #openstack-infra | 13:14 | |
| *** trown|brb is now known as trown | 13:16 | |
| yolanda | yes, it will be included | 13:18 |
| *** jamesmca_ has quit IRC | 13:18 | |
| *** mriedem has joined #openstack-infra | 13:20 | |
| *** jamesdenton has joined #openstack-infra | 13:21 | |
| mriedem | is etcd in jobs newish? http://logs.openstack.org/88/391288/57/gate/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/acc31d0/logs/subnode-2/devstacklog.txt.gz#_2017-05-24_03_20_11_510 | 13:26 |
| mriedem | and failing? | 13:26 |
| *** esikachev has joined #openstack-infra | 13:27 | |
| *** dabukalam has joined #openstack-infra | 13:27 | |
| fungi | mriedem: it was breaking multinode jobs for the latter part of yesterday because of not being excluded from subnodes, but that should be fixed now | 13:28 |
| *** rossella__ has joined #openstack-infra | 13:28 | |
| mriedem | ok yeah this was on a subnode | 13:28 |
| *** esberglu has joined #openstack-infra | 13:28 | |
| *** prateek__ has quit IRC | 13:30 | |
| *** ykarel has quit IRC | 13:30 | |
| *** vhosakot has joined #openstack-infra | 13:30 | |
| *** rossella_ has quit IRC | 13:31 | |
| *** vikrant has quit IRC | 13:31 | |
| *** esikachev has quit IRC | 13:31 | |
| *** esikachev has joined #openstack-infra | 13:37 | |
| *** sree has joined #openstack-infra | 13:41 | |
| *** dimtruck is now known as zz_dimtruck | 13:42 | |
| *** felipemonteiro has joined #openstack-infra | 13:42 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: [multinode] Remove t-i-e from the overcloud after bootstrap https://review.openstack.org/467627 | 13:43 |
| *** felipemonteiro_ has joined #openstack-infra | 13:44 | |
| andreas_s | hi sdague, would you mind worflowing https://review.openstack.org/#/c/467597/ again? I fixed the indent issue that bashate complained about - thanks! | 13:44 |
| *** jpena|lunch is now known as jpena | 13:45 | |
| *** mat128 has joined #openstack-infra | 13:47 | |
| *** felipemonteiro has quit IRC | 13:47 | |
| *** krtaylor has joined #openstack-infra | 13:47 | |
| openstackgerrit | zgdeng proposed openstack/diskimage-builder master: WIP: Add zip element as s390 architecture bootloader https://review.openstack.org/443548 | 13:48 |
| *** gouthamr has joined #openstack-infra | 13:50 | |
| *** srobert has joined #openstack-infra | 13:52 | |
| *** ZZelle has quit IRC | 13:52 | |
| *** yamahata has joined #openstack-infra | 13:53 | |
| *** jamesmca_ has joined #openstack-infra | 13:54 | |
| *** prateek has joined #openstack-infra | 13:57 | |
| *** rbrndt has joined #openstack-infra | 13:57 | |
| *** alexchadin has quit IRC | 13:57 | |
| openstackgerrit | Hari proposed openstack-infra/project-config master: Add openstack/ranger and openstack/ranger-agent project https://review.openstack.org/467374 | 13:58 |
| *** Guest36154 is now known as dansmith | 14:00 | |
| *** jpena is now known as jpena|brb | 14:02 | |
| *** zz_dimtruck is now known as dimtruck | 14:07 | |
| *** CynicGalahad has quit IRC | 14:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 14:18 |
| *** armax has joined #openstack-infra | 14:22 | |
| *** Julien-zte has quit IRC | 14:22 | |
| *** pradk has joined #openstack-infra | 14:23 | |
| *** Julien-zte has joined #openstack-infra | 14:23 | |
| *** sree_ has joined #openstack-infra | 14:23 | |
| *** sree_ is now known as Guest88303 | 14:23 | |
| *** lin_yang has joined #openstack-infra | 14:23 | |
| pabelanger | clarkb: I'd like to update nodepool dsvm plugin.sh to use pip_install from devstack, since it already has logic for python3 support. However, we currently don't want to use upper-constraints.txt. How can I best deal with: http://git.openstack.org/cgit/openstack-dev/devstack/tree/inc/python#n309 | 14:25 |
| *** prateek has quit IRC | 14:25 | |
| pabelanger | clarkb: should we just create an empty $REQUIREMENTS_DIR/upper-constraints.txt file? | 14:25 |
| *** sree has quit IRC | 14:25 | |
| clarkb | no that would break the rest of the devstack pip installs | 14:26 |
| pabelanger | that is what I figured, I see you disabled the ability to toggle upper-constraints too. So, not sure how to get pip_install going | 14:27 |
| *** thorst_afk has quit IRC | 14:27 | |
| clarkb | also nodepool uses a virtualenv so the devstack python stuff wont work as is anyways | 14:28 |
| clarkb | I think what you want is to create the venv using the python version set by devstack | 14:28 |
| clarkb | then everything should mostly just work as is | 14:28 |
| pabelanger | clarkb: pip vs pip3 shouldn't be an issue? | 14:29 |
| clarkb | no | 14:29 |
| clarkb | pip3 isnt really a thing | 14:29 |
| pabelanger | k, I thought that was the main issue. But ya, I'll create virtual env with python_version from devstack | 14:29 |
| clarkb | pip installs by default to the install of python it runs under. a python3 venv is that install | 14:30 |
| clarkb | pip3 is a debuntu convenience name I think (I dont have one on suse for example). So ya should work ok in python3 venv to just use pip there | 14:31 |
| *** gordc has joined #openstack-infra | 14:32 | |
| *** hongbin has joined #openstack-infra | 14:34 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Experimental py35 dsvm for ceph devstack plugin https://review.openstack.org/454890 | 14:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 14:37 |
| *** chlong has joined #openstack-infra | 14:38 | |
| *** yamahata has quit IRC | 14:39 | |
| pabelanger | I'm going to start working on nb01.o.o again | 14:40 |
| *** rbrndt_ has joined #openstack-infra | 14:42 | |
| *** rbrndt has quit IRC | 14:42 | |
| *** ccamacho is now known as ccamacho|brb | 14:43 | |
| *** markus_z has quit IRC | 14:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Use catalog endpoint on any errors in image version discovery https://review.openstack.org/467644 | 14:44 |
| *** ykarel has joined #openstack-infra | 14:45 | |
| *** jamesmca_ has quit IRC | 14:46 | |
| *** jamesmca_ has joined #openstack-infra | 14:46 | |
| sc` | still running into near showstopper rubygems timeouts. has anyone explored artifactory as a solution? | 14:46 |
| *** rbrndt_ has quit IRC | 14:46 | |
| *** mdrabe has quit IRC | 14:47 | |
| *** reduxio has joined #openstack-infra | 14:47 | |
| pabelanger | I've been working on setting up a proxy for rubygems, but haven't finished it up yet | 14:48 |
| pabelanger | I should be able to do that this week | 14:48 |
| *** rbrndt__ has joined #openstack-infra | 14:48 | |
| sc` | word. i'm patient, though it is impacting our ability to still appear alive | 14:49 |
| *** ram5391 has joined #openstack-infra | 14:49 | |
| *** annegentle has joined #openstack-infra | 14:49 | |
| pabelanger | https://review.openstack.org/#/c/452253/ | 14:49 |
| pabelanger | but we need to still test it | 14:50 |
| pabelanger | Oh, actually | 14:50 |
| pabelanger | that is the old way | 14:50 |
| pabelanger | when we wanted to run a gem mirror | 14:50 |
| *** cshastri has quit IRC | 14:50 | |
| *** dprince has quit IRC | 14:51 | |
| *** Julien-zte has quit IRC | 14:51 | |
| pabelanger | #status log both nb01.o.o and nb02.o.o are stopped. This is to allow nb03.o.o to build todays images | 14:52 |
| *** Julien-zte has joined #openstack-infra | 14:52 | |
| *** nicolasbock has quit IRC | 14:52 | |
| *** pabelanger has quit IRC | 14:52 | |
| *** pabelanger has joined #openstack-infra | 14:52 | |
| pabelanger | #status log both nb01.o.o and nb02.o.o are stopped. This is to allow nb03.o.o to build todays images | 14:52 |
| openstackstatus | pabelanger: finished logging | 14:52 |
| sc` | so long as gem / bundler don't blow up, however it works is however it works :) | 14:53 |
| *** mdrabe has joined #openstack-infra | 14:53 | |
| *** rossella__ has quit IRC | 14:54 | |
| *** psachin has quit IRC | 14:55 | |
| pabelanger | okay, I am going to start work on nb04.o.o now | 14:56 |
| *** marst_ has joined #openstack-infra | 14:57 | |
| *** jpena|brb is now known as jpena | 14:58 | |
| *** andreas_s has quit IRC | 15:00 | |
| *** ykarel has quit IRC | 15:01 | |
| *** ykarel has joined #openstack-infra | 15:01 | |
| *** bobh has joined #openstack-infra | 15:01 | |
| * jeblair deletes the /akick line he was preparing for -meeting .... | 15:03 | |
| *** esikachev has quit IRC | 15:03 | |
| *** dprince has joined #openstack-infra | 15:05 | |
| *** jamesmca_ has quit IRC | 15:07 | |
| *** Guest88303 has quit IRC | 15:08 | |
| clarkb | mordred: so shade 1.21 is on pypi. We want to restart nodepool.o.o's nodepool daemons ya? | 15:09 |
| *** sree has joined #openstack-infra | 15:09 | |
| clarkb | nodepool.o.o still has 1.20.0 installed | 15:10 |
| *** rossella__ has joined #openstack-infra | 15:10 | |
| *** ram5391 has quit IRC | 15:12 | |
| clarkb | looking at our puppet we do pip install -U nodepool but only on nodepool code changes (its a refreshonly) | 15:14 |
| *** Julien-zte has quit IRC | 15:15 | |
| *** gyee has joined #openstack-infra | 15:15 | |
| mordred | clarkb: yah- I think we want to force an upgrade of shade | 15:16 |
| *** Julien-zte has joined #openstack-infra | 15:16 | |
| mordred | clarkb: should we perhaps add an explicit shade install in our nodepool? or just do this as a one-off? | 15:16 |
| *** jamesmca_ has joined #openstack-infra | 15:16 | |
| clarkb | pabelanger: I just double checked my venv assumptions locally and py3-env/bin/pip does run under python3 if it is a python3 venv so your change should work | 15:17 |
| clarkb | mordred: I think we can do it as a one off here | 15:17 |
| pabelanger | clarkb: k, that a patch up, but looks like python3_enabled function didn't return true for some reason | 15:18 |
| pabelanger | digging into that | 15:18 |
| clarkb | pabelanger: oh except there is another weird interaction | 15:18 |
| *** rcernin has quit IRC | 15:18 | |
| mordred | clarkb: ok - I will go update shade on nodepool.o.o | 15:18 |
| clarkb | pabelanger: python2 -m virtualenv and python3 -m virtualenv say run the virtualenv module under those versions but not necessarily to intsall those versions? | 15:18 |
| mordred | clarkb: that did not install 1.21.0 | 15:19 |
| pabelanger | clarkb: oh, what should I be using | 15:19 |
| clarkb | mordred: pip install -U shade didn't work? | 15:19 |
| clarkb | pabelanger: virtualenv -p python$X is what I always use | 15:20 |
| mordred | clarkb: oh - sorry - I read the output wrong | 15:20 |
| *** cshen has quit IRC | 15:20 | |
| mordred | ton of text about uninstalling 1.20 - small text about installing 1.21 | 15:20 |
| fungi | anybody aware of any changes recently which would have broken http://stackalytics.openstack.org/ ? seems to completely 404 in apache now | 15:20 |
| *** reduxio has quit IRC | 15:20 | |
| pabelanger | clarkb: k | 15:21 |
| *** docaedo has left #openstack-infra | 15:21 | |
| clarkb | pabelanger: I left an inline comment, I think its likely ok as is because virtualenv will default to what it runs under as what it should install | 15:21 |
| clarkb | pabelanger: as for the python3_enabled check, how are you setting USE_PYTHON3? | 15:21 |
| pabelanger | clarkb: http://logs.openstack.org/40/467640/1/experimental/gate-dsvm-nodepool-py35-src-nv/b853d30/logs/devstack-early.txt.gz#_2017-05-24_14_51_30_834 | 15:21 |
| mordred | fungi: I am not, no | 15:21 |
| pabelanger | true == True | 15:21 |
| pabelanger | I think case issue? | 15:21 |
| pabelanger | clarkb: let me check JJB again | 15:22 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/nodepool.yaml#n16 | 15:22 |
| fungi | looks like the vhost just proxies a wsgi interface from 3031/tcp on localhost so maybe something has gone sour (it's definitely listening according to netstat) | 15:22 |
| clarkb | pabelanger: the case doesn't match | 15:22 |
| pabelanger | I am setting DEVSTACK_GATE_USE_PYTHON3 | 15:22 |
| clarkb | pabelanger: t != T | 15:22 |
| pabelanger | clarkb: yup | 15:22 |
| clarkb | pabelanger: so possibly a bug in d-g or devstack | 15:23 |
| pabelanger | will fix JJB | 15:23 |
| clarkb | ah or that | 15:23 |
| clarkb | fungi: I am unaware of changes to stackalytics | 15:23 |
| *** shardy has quit IRC | 15:23 | |
| clarkb | mordred: so are we ready to restart services now? | 15:23 |
| mordred | clarkb: yes - one sec | 15:23 |
| fungi | i'm restarting apache2 on stackalytics.o.o to see if that helps at all | 15:23 |
| clarkb | mordred: just remember to stop nodepool-deleter last and start it first | 15:24 |
| pabelanger | fungi: is memcache still running? | 15:24 |
| *** salv-orlando has joined #openstack-infra | 15:24 | |
| clarkb | nodepool and nodepool-launcher can be stopped/started in any order | 15:24 |
| fungi | pabelanger: yep, running since april 13 according to the process table | 15:24 |
| clarkb | mordred: I can do the service restarts if you want, I just need a minute | 15:25 |
| *** jamesmca_ has quit IRC | 15:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix incorrect case for DEVSTACK_GATE_USE_PYTHON3 https://review.openstack.org/467652 | 15:26 |
| fungi | pabelanger: restarting memcached also doesn't seem to have helped. seeing if uwsgi needs a restart maybe | 15:26 |
| pabelanger | clarkb: AJaeger: ^if you don't mind | 15:27 |
| *** jamesmca_ has joined #openstack-infra | 15:27 | |
| clarkb | pabelanger: ya virtualenv seems to inherit its calling env by default like that | 15:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Bring nb04.o.o online https://review.openstack.org/467653 | 15:29 |
| clarkb | so your code should work once the jjb update is applied | 15:29 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Create priority order for configuration files https://review.openstack.org/465632 | 15:29 |
| *** felipemonteiro_ has quit IRC | 15:29 | |
| pabelanger | clarkb: fungi:^ and nb04.o.o online. I left this one in rackspace so we can split them over clouds | 15:29 |
| fungi | pabelanger: restarting uwsgi at least seems to have switched the problem from a 404 to an internal server error, so there may have been something stuck there i guess | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 15:30 |
| *** salv-orlando has quit IRC | 15:30 | |
| mordred | clarkb: ok. I'm ready (wanted to clear state from head) | 15:30 |
| clarkb | mordred: so I should go ahead and restart services now? | 15:30 |
| pabelanger | fungi: if you look at stackalytics-processor log, it should say what is going on | 15:30 |
| *** nicolasbock has joined #openstack-infra | 15:30 | |
| mordred | clarkb: sure! I'll watch logs | 15:30 |
| clarkb | ok starting now | 15:30 |
| pabelanger | clarkb: okay, I'll update patch once JJB change lands | 15:31 |
| fungi | pabelanger: ouch, "ERROR stackalytics RcsException: Failed to connect to gerrit: No existing session" | 15:31 |
| *** sbadia has left #openstack-infra | 15:31 | |
| clarkb | mordred: all three daemons are restarted now, we should be running on new shade except for nb03 | 15:31 |
| clarkb | nb04 I'm guessing came up with new shade due to when it was built | 15:32 |
| mordred | clarkb: stop - need to roll back | 15:32 |
| clarkb | mordred: ok stopping | 15:32 |
| pabelanger | fungi: Hmm, has paramiko recently changed? | 15:32 |
| pabelanger | Or maybe we have used up all connection slots on gerrit again | 15:32 |
| clarkb | mordred: services are stopped if you want ot downgrade shade now | 15:32 |
| mordred | clarkb: ok. I have downgraded | 15:32 |
| clarkb | ok starting back up again | 15:32 |
| mordred | clarkb: I'm going to double cecka few things- but we hit a bunch of 400 errors on looking for image ... which should not happen | 15:33 |
| *** Julien-zte has quit IRC | 15:33 | |
| clarkb | ok services should be back on old shade | 15:33 |
| fungi | pabelanger: using paramiko 1.16.0 from november 2015 | 15:33 |
| pabelanger | fungi: okay, maybe gerrit side then | 15:33 |
| clarkb | pabelanger: fungi stackalytics is the one that leaks connections to gerrit right? | 15:34 |
| mordred | clarkb: actually - this may not be shade | 15:34 |
| clarkb | was that every fixed? maybe it leaked too many connections? | 15:34 |
| fungi | clarkb: i thought that got patched | 15:34 |
| *** DangerousDaren has quit IRC | 15:34 | |
| *** rbrndt__ has quit IRC | 15:34 | |
| fungi | clarkb: or maybe it was something else we patched that problem in? | 15:34 |
| mordred | clarkb: it's a novaclient error we're getting - looks like nova itself can't find the image we're wanting it to use | 15:34 |
| pabelanger | clarkb: fungi: I wonder if paramiko is too old for the stackalytics patch. I thought it was 2.x we had to close on delete | 15:34 |
| pabelanger | clarkb: fungi: maybe we should upgrade it? | 15:34 |
| clarkb | mordred: in which cloud? we've seen that happen in osic I think | 15:35 |
| *** esikachev has joined #openstack-infra | 15:35 | |
| *** yamahata has joined #openstack-infra | 15:35 | |
| *** aarefiev is now known as aarefiev_afk | 15:35 | |
| clarkb | pabelanger: fungi ya that could be it, I want to say older code just leaked them and newer code still leaks them but gives you the ability to not leak them? | 15:35 |
| mordred | clarkb: osic | 15:35 |
| clarkb | mordred: I want to say thats our new "normal" maybe | 15:36 |
| fungi | pabelanger: ouch, stackalytics reqs list has paramiko>=2.0 so wonder why we're running 1.6.0 | 15:36 |
| mordred | clarkb: ok - I think we can retry the upgrade- sorry, I was too quick to pull the trigger :) | 15:36 |
| clarkb | mordred: np :) let me know when you have shade reinstalled at new version | 15:36 |
| pabelanger | fungi: not sure, guess we don't pip -U install in puppet? | 15:36 |
| *** rbrndt has joined #openstack-infra | 15:36 | |
| mordred | clarkb: I also want to turn on shade debug logging at least for a little bit this time | 15:37 |
| fungi | pabelanger: yeah, i guess that's it (but would think that pip installing would at least choke on unmet reqs minimums there) | 15:37 |
| mordred | clarkb: ready | 15:38 |
| clarkb | mordred: ok restarting services now | 15:38 |
| clarkb | mordred: they should be doing things now | 15:39 |
| fungi | pabelanger: aha, doing pip install -U is failing because it wants to build cffi | 15:39 |
| pabelanger | fungi: ah | 15:39 |
| fungi | because of pyca/cryptography | 15:39 |
| pabelanger | Ya, from paramiko | 15:40 |
| pabelanger | that sounds right | 15:40 |
| fungi | needs libffi-dev, libssl-dev... iterating to find out what else | 15:40 |
| *** spzala has quit IRC | 15:40 | |
| clarkb | mordred: it would be nice if we logged the requested image's name/id | 15:40 |
| fungi | pabelanger: i'll get a patch up for the puppet module here shortly | 15:40 |
| clarkb | I'm going to poke at the osic things now | 15:40 |
| pabelanger | fungi: great | 15:40 |
| *** spzala has joined #openstack-infra | 15:40 | |
| pabelanger | clarkb: issues with images and osic? | 15:41 |
| mordred | clarkb: agree. I think we can add that when we restify that call (the rest calls return structured data in their error messages) | 15:41 |
| clarkb | pabelanger: yes nova can't find images to boot so boots are failing | 15:41 |
| mordred | clarkb: ok - nothing else seems to be breaking - and I confirm the same error on osic on either side of the upgrade | 15:41 |
| pabelanger | clarkb: I noticed we had problems all last night uploading our images after the domain change | 15:41 |
| pabelanger | clarkb: wonder if related | 15:41 |
| *** esikachev has quit IRC | 15:42 | |
| clarkb | why does image-list show a large list of failed images? shouldn't the builders delete those? | 15:42 |
| clarkb | Shrews: ^ fyi | 15:42 |
| pabelanger | clarkb: no, they stay until we upload a new image | 15:43 |
| pabelanger | so, if you have a lot of failures, you get the massive list we have now | 15:43 |
| pabelanger | maybe we should cap it to last failure | 15:43 |
| Shrews | or last X? | 15:43 |
| pabelanger | wfm | 15:43 |
| *** esikachev has joined #openstack-infra | 15:44 | |
| *** mikal_ has joined #openstack-infra | 15:44 | |
| fungi | pabelanger: as for gerrit connections, looks like there are 64 open connections in the gerrit api for the infra-stackalytics account. i'll try to close them all | 15:44 |
| clarkb | pabelanger: Shrews that isn't the behavior I am seeing. | 0000000312 | 0000000478 | osic-cloud1-s3700 | ubuntu-xenial | ubuntu-xenial-1495626849 | c42690e2-4b7f-4147-9ee9-2a6852974d38 | ready | 00:03:49:35 | is three hours old, there are many failed uplaods there that are older than that | 15:44 |
| pabelanger | fungi: Ya, if history repeats, I don't think you'll be able to. But, I hope you prove me wrong :) | 15:44 |
| clarkb | so I think the keep until successful logic must be buggy, I like the idea of last X too though | 15:45 |
| pabelanger | clarkb: only a new image build, will replace failure, not a successful upload of of the image that failed | 15:45 |
| *** spzala has quit IRC | 15:45 | |
| clarkb | that seems like a bug then | 15:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Turn on shade debug logging for nodepool https://review.openstack.org/467660 | 15:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix incorrect case for DEVSTACK_GATE_USE_PYTHON3 https://review.openstack.org/467652 | 15:46 |
| Shrews | clarkb: paste me a few of the fails? | 15:46 |
| pabelanger | clarkb: I think the idea was, an operator wanted to see previous failures. However, I haven't found it too useful and tends just to spam the CLI | 15:46 |
| *** mikal has quit IRC | 15:47 | |
| clarkb | Shrews: http://paste.openstack.org/show/610571/ | 15:47 |
| fungi | pabelanger: confirmed, all of the open connections remained open even after a close-connection | 15:47 |
| fungi | maddening | 15:47 |
| clarkb | pabelanger: I've just found it to be the opposite of useful, makes it hard to see what image is actually failing to boot in osic | 15:47 |
| pabelanger | fungi: ya, it that case we need to wait for gerrit restart :( | 15:47 |
| clarkb | I'm fine with keeping the failures around until success, but we've succeeded I don't care too much about those failures in the cli. We have logs for that | 15:47 |
| *** jaypipes has quit IRC | 15:47 | |
| pabelanger | clarkb: Ya, I open grep -v failed | 15:48 |
| pabelanger | often* | 15:48 |
| clarkb | as for the osic issue c42690e2-4b7f-4147-9ee9-2a6852974d38 does not exist according to the osic api | 15:48 |
| *** ram5391 has joined #openstack-infra | 15:48 | |
| clarkb | I'm going to see if the older images do and if so delete these newer ones | 15:49 |
| Shrews | pabelanger: clarkb: one thing it could be... the zk node is deleted only if the local build on disk is deleted. any chance those removed manually? | 15:49 |
| *** esikachev has quit IRC | 15:49 | |
| *** sree has quit IRC | 15:49 | |
| clarkb | Shrews: thats for the build node though not the upload node right? | 15:49 |
| *** ram5391 has quit IRC | 15:49 | |
| *** Sukhdev_ has joined #openstack-infra | 15:50 | |
| clarkb | I deleted c42690e2-4b7f-4147-9ee9-2a6852974d38 from nodepool now to see if there are other images with this problem in osic | 15:51 |
| mordred | jeblair: happen to know off the top of your head if there is a good way to throw an exception in createServer that will not log a traceback but will cause the create to fail and be retried? | 15:51 |
| Shrews | clarkb: right. oh, these are 'upload' fails not 'build' fails? | 15:51 |
| *** salv-orlando has joined #openstack-infra | 15:52 | |
| clarkb | Shrews: correct, the cloud had about a day long vacation where its api wasn't reachable :) | 15:52 |
| clarkb | Shrews: the dib runs were fine, but when we went to push into the cloud we found the cloud was not available | 15:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-stackalytics master: Add new system deps for libffi-dev and libssl-dev https://review.openstack.org/467663 | 15:52 |
| fungi | pabelanger: ^ | 15:53 |
| clarkb | mordred: nodepool logs look a lot happier now after clearing out that image | 15:53 |
| tushar | Pabelanger : the DIB image creation is failing with error - | 15:54 |
| tushar | + _pub_key=/tmp/in_target.d/zuul-user-ssh-public-key + '[' '!' -f /tmp/in_target.d/zuul-user-ssh-public-key ']' + die 'Can not find Zuul public key!' | 15:54 |
| tushar | basically it is not getting the public key for zuul user | 15:55 |
| *** panda is now known as panda|bbl | 15:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't spam logs with tracebacks on missing image https://review.openstack.org/467664 | 15:55 |
| mordred | clarkb: ^^ there is an attempt at making that less chatty next time- and including info on what image is failing | 15:56 |
| tushar | I think we need to add private and public key for zuul user in common.yaml...right/ | 15:56 |
| tushar | ? | 15:56 |
| clarkb | tushar: no, if you look at the code that is failing there its just the public key and by default it takes the public key for the user building the image | 15:56 |
| clarkb | tushar: you can also set a variable for it | 15:57 |
| *** notmyname has quit IRC | 15:57 | |
| tushar | where I need to setup the variable? | 15:57 |
| *** salv-orlando has quit IRC | 15:57 | |
| tushar | clarkb : where I need to setup the variable? | 15:58 |
| clarkb | mordred: one small thing noted on change | 15:58 |
| *** Apoorva has joined #openstack-infra | 15:59 | |
| *** esikachev has joined #openstack-infra | 15:59 | |
| clarkb | tushar: we do it with the env-vars section in nodepool.yaml https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n1093 | 15:59 |
| *** yamahata has quit IRC | 16:00 | |
| clarkb | tushar: but its whereever you are invoking dib from | 16:00 |
| mordred | clarkb: cool. also - I abandoned the change to add shade debug logging - it's actually not needed | 16:00 |
| *** notmyname has joined #openstack-infra | 16:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't spam logs with tracebacks on missing image https://review.openstack.org/467664 | 16:01 |
| mordred | clarkb: agreed | 16:01 |
| *** rbrndt has quit IRC | 16:01 | |
| *** jaypipes has joined #openstack-infra | 16:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "Remove citycloud from multinode labels" https://review.openstack.org/467667 | 16:03 |
| clarkb | mordred: pabelanger fungi ^ that should be safe now that we are using latest shade with my workaround in place | 16:03 |
| pabelanger | +2 | 16:03 |
| *** spzala has joined #openstack-infra | 16:04 | |
| *** felipemonteiro has joined #openstack-infra | 16:04 | |
| *** esikachev has quit IRC | 16:04 | |
| *** rbrndt has joined #openstack-infra | 16:04 | |
| pabelanger | okay, centos-7 DIB on nb03.o.o is just finishing up, uploads should start shortly | 16:04 |
| fungi | pabelanger: is http://stackalytics.openstack.org/ just going to keep returning an internal server error indefinitely until it can successfully establish an ssh connection to the gerrit api again? | 16:05 |
| clarkb | pabelanger: when you are ready (maybe after these uplaods) we will want ot update shade and restart the builder on nb03 too | 16:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 16:05 |
| tushar | clarkb : okay..but there is no variable to zuul public key in - https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n1093 | 16:05 |
| clarkb | tushar: correct, we rely on the use the current user's key behavior | 16:05 |
| clarkb | I think | 16:05 |
| tushar | clarkb : okay | 16:05 |
| mordred | clarkb: yes. I agree | 16:05 |
| pabelanger | fungi: it is likely missing data to serve up. You could try looking at root crontab to see how to restore data from a back up | 16:06 |
| pabelanger | fungi: that should at least get some data into stackalytics | 16:06 |
| fungi | pabelanger: cool, will try that. thanks! | 16:06 |
| *** r-daneel has joined #openstack-infra | 16:06 | |
| *** ykarel has quit IRC | 16:06 | |
| tushar | clarkb : there is 1 similar case for jenkins public key there is one parameter in common.yaml | 16:07 |
| tushar | clarkb : I was comparing these two scenarios | 16:07 |
| clarkb | tushar: common.yaml? | 16:07 |
| clarkb | oh is the the hiera file? yes, we are no longer using puppet to build our images | 16:08 |
| *** esikachev has joined #openstack-infra | 16:08 | |
| *** spzala has quit IRC | 16:08 | |
| tushar | clarkb : /etc/puppet/environments/common.yaml | 16:09 |
| *** thorst has joined #openstack-infra | 16:09 | |
| fungi | tushar: odds are you need an updated project-config repo with the diskimage-builder element that creates that account | 16:09 |
| *** spzala has joined #openstack-infra | 16:10 | |
| tushar | clarkb: oh ..then what you are using currently to build images..instead of puppet? | 16:10 |
| fungi | tushar: does it here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/install.d/60-zuul-worker#n22 | 16:10 |
| tushar | fungi : actually i am using the latest project config | 16:11 |
| fungi | tushar: the small amount of configuration which was being performed by puppet just got refactored out into diskimage-builder elements directly | 16:11 |
| jeblair | mordred: how bad are the tracebacks you're removing in 467664? | 16:11 |
| jeblair | mordred: tbh, i'm skittish about removing them since i feel like we spent a long time making sure we *had* tracebacks from weird openstack errors. | 16:12 |
| tushar | fungi : yeh I saw that 60-zuul-worker | 16:12 |
| fungi | tushar: maybe you're missing setting the envvar used here? http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user | 16:12 |
| *** salv-orlando has joined #openstack-infra | 16:12 | |
| pabelanger | tushar: clarkb: the issue is single_node_ci.pp does not plum through the nodepool_ssh_public_key setting for nodepool_builder.pp | 16:12 |
| pabelanger | tushar: clarkb: this is the reason the key is not setup properly | 16:13 |
| clarkb | jeblair: the problem with the traceback is it doesn't have actioanble info liek what image is not found | 16:13 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/nodepool_builder.pp#n22 needs to be setup for zuul-worker element | 16:13 |
| clarkb | jeblair: it just says nova couldn't find the image in glance along with ~40 lines of code traceback | 16:13 |
| clarkb | jeblair: mordred I'm fine keeping the traceback if we also want to log the image more better | 16:14 |
| *** spzala has quit IRC | 16:14 | |
| tushar | Pabelanger : http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/nodepool_builder.pp#n22 where to set the env var for this? | 16:15 |
| clarkb | jeblair: I think mordred doesn't like the traceback because it implies nodepool code broke, when reality is its something in the cloud | 16:15 |
| tushar | Pablenager : 1 similar scenarios is here for public of jenkins user - http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/nodepool_builder.pp#n22 | 16:15 |
| fungi | pabelanger: i can't find anywhere stackalytics creates a dump of its data as a backup, so i'm just going to assume this is broken until the next gerrit restart (we're probably coming due for one anyway, last restart was 7 days ago) | 16:15 |
| pabelanger | tushar: if you are using http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp you cannot today, somebody will have to submit a patch | 16:15 |
| jeblair | clarkb: it's true that traceback doesn't have the image info, however, it does have the node id, and all nodepool log entries can be cross-referenced by that. so it's easy to get this: http://paste.openstack.org/show/610578/ | 16:15 |
| tushar | Pabelnager : this is for jenkins user http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/jenkins-slave/extra-data.d/20-jenkins-user | 16:16 |
| pabelanger | tushar: another solution is not to use zuul-worker element for now, since we haven't made the cut over to it in nodepool.o.o | 16:16 |
| tushar | Pablenger : for this I have set the public key in common.yaml | 16:16 |
| pabelanger | tushar: yes, that is the legacy way, which we are still using | 16:17 |
| clarkb | pabelanger: so we just need single_node_ci to set the var? that seems straightfowrad | 16:17 |
| pabelanger | clarkb: no, because it is not using nodepool_builder.pp, but nodepool.pp | 16:17 |
| pabelanger | clarkb: it would be a little work to get it | 16:17 |
| clarkb | jeblair: right hte problem is you don't know what image id was requested | 16:17 |
| clarkb | jeblair: so hard to know if its a bogus id, or if cloud is broken or what | 16:17 |
| clarkb | pabelanger: gotcha | 16:18 |
| pabelanger | actually, nodepool.pp does support it, but ya, maybe that could work | 16:18 |
| clarkb | jeblair: so I think we should log the image id that the cloud says it couldn't find at the very least | 16:18 |
| jeblair | clarkb, mordred: i don't think tracebacks are only for when "code" breaks. we spent years trying to figure out what underlying cloud problem was causing nodepool issues because of exception masking, so that's a dangerous idea to proceed from. i'm actually more okay with this particular patch (which seems like a pretty decent log line) than i am with the idea that we should not log tracebacks unless "code" is broken. | 16:18 |
| jeblair | clarkb: yes, perhaps we should always log the image id we request even on success | 16:19 |
| pabelanger | fungi: let me look quickly | 16:19 |
| *** spzala has joined #openstack-infra | 16:19 | |
| *** esikachev has quit IRC | 16:19 | |
| pabelanger | fungi: http://git.openstack.org/cgit/openstack-infra/puppet-stackalytics/tree/manifests/init.pp#n127 | 16:20 |
| pabelanger | fungi: that command should work | 16:20 |
| pabelanger | oh, it was stackalytics user | 16:20 |
| fungi | pabelanger: oh, i missed that in the manifest | 16:21 |
| fungi | i'll give it a shot | 16:21 |
| jeblair | clarkb, mordred: to riff on mordred's commit message -- sometimes we have needed tracebacks to determine the root cause of data errors. if we're going to start masking them, i just want us to think very carefully about it and make sure we're not hobbling ourselves. | 16:21 |
| mordred | jeblair: oh - totally - I was more trying to say that in _this_ case there is no additional useful info in the traceback | 16:21 |
| mordred | how about I update the commit message | 16:21 |
| jeblair | mordred: (also, i have a request that we bump it to error level -- inline comment explains) | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| *** thorst has quit IRC | 16:23 | |
| tushar | Pabelanger : How can I skip the zuul-worker element ? ..beacause I havnt mentioned it in nodepool.yaml - https://pastebin.com/Przy3YTu | 16:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Remove citycloud from multinode labels" https://review.openstack.org/467667 | 16:23 |
| *** ltomasbo is now known as ltomasbo|away | 16:23 | |
| *** spzala has quit IRC | 16:23 | |
| *** pcaruana has quit IRC | 16:24 | |
| jeblair | mordred, clarkb: if we're pretty sure there's no chance of a traceback adding extra information here, okay. i'm not sure we should mask the actual error we get from nova though. if we wanted to report this to a cloud operator, we might want to give the request id, etc, which is in the error string. | 16:24 |
| pabelanger | tushar: it looks to be pulled in by nodepool-base/elements-dep. You can comment it out for now | 16:24 |
| clarkb | jeblair: thats a good point, being able to track the request down on the cloud side is likely useful in debugging why an image is not available | 16:25 |
| openstackgerrit | Roland Hochmuth proposed openstack-infra/irc-meetings master: Change meeting and channel for Monasca https://review.openstack.org/467673 | 16:25 |
| jeblair | clarkb, mordred: so at least a "str(e)" in the message we do log | 16:25 |
| *** thorst has joined #openstack-infra | 16:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 16:26 |
| *** annegentle has quit IRC | 16:26 | |
| mordred | jeblair: kk | 16:26 |
| tushar | Pabelanger : okay...It is not mandatory element ...I mean I can use the DIB image created without nodepool-base to run the CI? | 16:26 |
| *** hashar has quit IRC | 16:26 | |
| pabelanger | tushar: well, we do use it today, for zuulv3-dev.o.o. But, it is not needed for 3rdparty CI right now | 16:27 |
| pabelanger | it will be shortly, but we are not ready to switch to it | 16:27 |
| *** annegentle has joined #openstack-infra | 16:27 | |
| tushar | Pabelanger : Okay..I am skipping it for now | 16:28 |
| clarkb | Shrews: pabelanger thinking about the failed uploads thing a bit more, I think keeping the last failure only should be sufficient? THat notifies the user that there is or was a failure and then they have to look at logs regardless which will show the true extent of the problem | 16:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't log tracebacks for server-side missing image https://review.openstack.org/467664 | 16:28 |
| jeblair | clarkb, Shrews, pabelanger: i don't think we should store failure records in zk at all | 16:28 |
| clarkb | jeblair: that also works for me | 16:28 |
| mordred | jeblair: ^^ updated commit message, and log message- and also added a TODO to make sure we follow up on that piece when we move create_server from novaclient to REST | 16:28 |
| openstackgerrit | Merged openstack-infra/puppet-stackalytics master: Add new system deps for libffi-dev and libssl-dev https://review.openstack.org/467663 | 16:29 |
| clarkb | basically logs are the primary debugging and information source | 16:29 |
| clarkb | what zk shows is at most a convenience | 16:29 |
| jeblair | yes, we should store error states in zk only as long as it takes to resolve (ie, delete) them | 16:29 |
| jeblair | it should just be application state, not history | 16:29 |
| *** spzala has joined #openstack-infra | 16:30 | |
| pabelanger | I am okay with that too | 16:30 |
| clarkb | I've got to take kids to dentist in an hour and a half but I can work on that patch this afternoon around making a snapshot for bkero | 16:31 |
| clarkb | unless someone else wants to beat me to | 16:31 |
| clarkb | er beat me to it | 16:31 |
| jeblair | such violence | 16:31 |
| pabelanger | but, when I do see a single failed in list, I tend to look at debug logs, to see why it failed | 16:31 |
| *** Apoorva_ has joined #openstack-infra | 16:32 | |
| jeblair | pabelanger: yeah. it's true that it is *a* way to discover failures, it's just that it's not a *great* way to discover failures, since it's so transient. | 16:32 |
| *** Rockyg has joined #openstack-infra | 16:33 | |
| jeblair | pabelanger: absence of a failure record doesn't mean success :) | 16:33 |
| *** efoley_ has joined #openstack-infra | 16:33 | |
| bkero | clarkb: thanks | 16:33 |
| mordred | I was also just thinking about logs while looking at this - it seems like at some point we should audit info and debug - I think we all tend to use the debug log to track down issues - but that's so chatty 'normal' errors like the image missing in osic are easy to miss | 16:33 |
| pabelanger | k, nb03.o.o is upload images now | 16:33 |
| pabelanger | Hmm | 16:34 |
| pabelanger | 2017-05-24 16:34:01,227 DEBUG shade: Connection failure for ImagePutImagesFile, retrying | 16:34 |
| mordred | which is to say - I'm wondering if a log with warning+error only would provide enough info to notify that there is an issue and that the bigger log should be looked at | 16:34 |
| jeblair | failures happen all the time, most of which aren't interesting. if you expect success, then looking through logs is a better way to assure yourself. or, in our case, generally looking at graphs with error rates is a better way to find out when it's worth looking into issues | 16:34 |
| *** Apoorva has quit IRC | 16:34 | |
| *** sambetts is now known as sambetts|afk | 16:34 | |
| *** spzala has quit IRC | 16:34 | |
| mordred | pabelanger: hrm. that doesn't seem pleasing | 16:34 |
| jeblair | mordred: absolutely. i'm certain we have drifted from the ideal. and also probably are missing some cross-referencing ability (lines missing node ids, etc). | 16:34 |
| mordred | jeblair: yah | 16:35 |
| *** florianf has quit IRC | 16:35 | |
| pabelanger | I think this is osic | 16:35 |
| pabelanger | getting more info | 16:35 |
| mordred | jeblair: I don't think we need to do that today - but as we get closer and closer to not-us running nodepool via v3, it would probably be a good thing to put on the list | 16:35 |
| jeblair | ++ | 16:36 |
| *** spzala has joined #openstack-infra | 16:36 | |
| mordred | pabelanger: nb03 right? | 16:36 |
| *** efoley has quit IRC | 16:37 | |
| *** nicolasbock has quit IRC | 16:37 | |
| pabelanger | mordred: yes | 16:37 |
| pabelanger | trying to get a debug log from manual upload | 16:37 |
| mordred | pabelanger: ok - so - we have a connection retry in there to handle normal reconnects | 16:37 |
| mordred | but we should definitley track/doublecheck that ... | 16:38 |
| mordred | also - I think shade should log the cloud name when it does that :) | 16:38 |
| pabelanger | Oh, | 16:38 |
| pabelanger | I have traceback now :D | 16:38 |
| pabelanger | 1 sec | 16:38 |
| pabelanger | http://paste.openstack.org/show/610581/ | 16:38 |
| pabelanger | exception is no delete, but upload failed for some reason | 16:39 |
| pabelanger | I wonder if this is a blip because domain name change | 16:39 |
| pabelanger | mordred: ^ | 16:40 |
| *** spzala has quit IRC | 16:40 | |
| *** gardlt has joined #openstack-infra | 16:41 | |
| *** annegentle has quit IRC | 16:42 | |
| Shrews | jeblair: pabelanger: i *think* we could just not store the failed upload record. i don't think anything in the code depends on it being there, but i'd have to refresh my memory on it | 16:44 |
| mordred | pabelanger: so - the bottom 409 is the original error for the post call that caused us to want ot delete | 16:44 |
| mordred | pabelanger: "Expecting value: line 1 column 1 (char 0)" is not a great error there however | 16:45 |
| pabelanger | mordred: let me see if I can get requests logging | 16:45 |
| mordred | pabelanger: could you try that with http_debug=True instead od debug=True? | 16:45 |
| *** ccamacho|brb is now known as ccamacho|out | 16:45 | |
| pabelanger | mordred: sure | 16:45 |
| mordred | pabelanger: (it'll be rather chatty - but oh-well) | 16:46 |
| *** annegentle has joined #openstack-infra | 16:46 | |
| pabelanger | mordred: http://paste.openstack.org/show/610582/ | 16:47 |
| *** spzala has joined #openstack-infra | 16:48 | |
| pabelanger | http://paste.openstack.org/show/610583/ | 16:48 |
| pabelanger | is truncated bits | 16:48 |
| *** esikachev has joined #openstack-infra | 16:49 | |
| pabelanger | 409 seem to imply an issue on osic side | 16:49 |
| *** jpich has quit IRC | 16:51 | |
| clarkb | pabelanger: perhaps related to how nodepool can think an image uploaded properly but it isn't there in the cloud? | 16:51 |
| *** trown is now known as trown|lunch | 16:51 | |
| *** efoley__ has joined #openstack-infra | 16:52 | |
| pabelanger | clarkb: ya, I am wondering if something in glance doesn't like the new domain name? | 16:52 |
| *** spzala has quit IRC | 16:52 | |
| *** salv-orlando has quit IRC | 16:53 | |
| *** esikachev has quit IRC | 16:53 | |
| *** spzala has joined #openstack-infra | 16:53 | |
| pabelanger | speaking of glance, looks like py35 isn't happy on nodepool dsvm job: http://logs.openstack.org/40/467640/2/experimental/gate-dsvm-nodepool-py35-src-nv/b5fc1c1/logs/screen-g-api.txt.gz#_May_24_16_45_43_506757 | 16:54 |
| mordred | pabelanger: thanks. | 16:54 |
| clarkb | pabelanger: we need to disable swift in those jobs | 16:54 |
| clarkb | pabelanger: swift can't py3 yet and glance uses it as backend if its there | 16:54 |
| pabelanger | clarkb: k | 16:54 |
| *** efoley_ has quit IRC | 16:55 | |
| mordred | pabelanger: that doesn't seem to have any extra http calls in the log :( | 16:55 |
| pabelanger | mordred: ya | 16:55 |
| *** ramishra has quit IRC | 16:55 | |
| *** salv-orlando has joined #openstack-infra | 16:57 | |
| pabelanger | clarkb: I assume glance is smart enough to fall back to filesystem? | 16:57 |
| *** derekh has quit IRC | 16:57 | |
| *** Sukhdev_ has quit IRC | 16:57 | |
| *** spzala has quit IRC | 16:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable swift for py35 nodepool dsvm job https://review.openstack.org/467685 | 16:58 |
| *** e0ne has quit IRC | 16:59 | |
| *** e0ne has joined #openstack-infra | 16:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Log cloud name on Connection retry issues https://review.openstack.org/467686 | 16:59 |
| *** spzala has joined #openstack-infra | 16:59 | |
| *** e0ne has quit IRC | 16:59 | |
| mordred | pabelanger: is that script on nb03 or somewhere else? | 17:00 |
| *** salv-orlando has quit IRC | 17:02 | |
| pabelanger | mordred: I am just running it from python on nodepool.o.o | 17:02 |
| mordred | ah - gotcha. mind pasting me your command? | 17:02 |
| pabelanger | sure | 17:02 |
| pabelanger | mordred: http://paste.openstack.org/show/610584/ | 17:03 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate master: Add etcd3 to the list of base services https://review.openstack.org/467687 | 17:03 |
| *** spzala has quit IRC | 17:04 | |
| *** SumitNaiksatam has quit IRC | 17:05 | |
| *** spzala has joined #openstack-infra | 17:05 | |
| mordred | pabelanger: thanks! | 17:07 |
| *** spzala has quit IRC | 17:08 | |
| *** spzala has joined #openstack-infra | 17:08 | |
| *** esikachev has joined #openstack-infra | 17:09 | |
| *** thorst has quit IRC | 17:09 | |
| *** rcarrillocruz has quit IRC | 17:12 | |
| dims | mordred : sdague : let's switch on etcd3 in features.yaml - https://review.openstack.org/#/c/467687/ | 17:12 |
| *** esikachev has quit IRC | 17:14 | |
| *** jamesmca_ has quit IRC | 17:14 | |
| *** esikachev has joined #openstack-infra | 17:15 | |
| *** ram5391 has joined #openstack-infra | 17:15 | |
| *** sbezverk_ has quit IRC | 17:15 | |
| *** links has quit IRC | 17:16 | |
| pabelanger | mordred: clarkb: fungi: mind a review on https://review.openstack.org/#/c/467653/ bring nb04.o.o into rotation | 17:18 |
| *** esikachev has quit IRC | 17:19 | |
| *** efoley__ has quit IRC | 17:22 | |
| *** jaosorior has quit IRC | 17:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add time reporting to Connection Retry message https://review.openstack.org/467689 | 17:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't log tracebacks for server-side missing image https://review.openstack.org/467664 | 17:25 |
| mordred | jeblair: fixed yourcomment on the nodepool patch ^^ also, two shade patches to make the connection retry message better | 17:26 |
| mordred | pabelanger: you may like those two as well | 17:26 |
| *** thorst has joined #openstack-infra | 17:29 | |
| jeblair | mordred: thx +2 | 17:29 |
| *** esikachev has joined #openstack-infra | 17:30 | |
| *** ijw_ has quit IRC | 17:30 | |
| *** dhajare has quit IRC | 17:33 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:33 | |
| *** thorst has quit IRC | 17:33 | |
| *** thorst has joined #openstack-infra | 17:33 | |
| *** logan- has quit IRC | 17:34 | |
| openstackgerrit | Merged openstack-infra/system-config master: Bring nb04.o.o online https://review.openstack.org/467653 | 17:34 |
| pabelanger | clarkb: if you don't mind confirming: https://review.openstack.org/#/c/467685/ disables swift for py35 | 17:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix python3 issues in functional tests https://review.openstack.org/467693 | 17:35 |
| openstackgerrit | Merged openstack-infra/system-config master: Add python3-yaml to signing node https://review.openstack.org/463055 | 17:35 |
| *** logan- has joined #openstack-infra | 17:35 | |
| *** dhinesh has joined #openstack-infra | 17:37 | |
| *** Apoorva_ has quit IRC | 17:38 | |
| *** esikachev has quit IRC | 17:38 | |
| *** jamesmca_ has joined #openstack-infra | 17:38 | |
| *** Apoorva has joined #openstack-infra | 17:38 | |
| *** jamesmca_ has quit IRC | 17:43 | |
| pabelanger | #status log nb03.o.o and nb04.o.o are online (upgraded to xenial). Will be waiting a day or 2 before deleting nb01.o.o and nb02.o.o. | 17:46 |
| openstackstatus | pabelanger: finished logging | 17:46 |
| *** Sukhdev_ has joined #openstack-infra | 17:46 | |
| *** esikachev has joined #openstack-infra | 17:47 | |
| *** rcarrillocruz has joined #openstack-infra | 17:47 | |
| *** ralonsoh has quit IRC | 17:48 | |
| *** salv-orlando has joined #openstack-infra | 17:49 | |
| *** esikachev has quit IRC | 17:51 | |
| *** esikachev has joined #openstack-infra | 17:52 | |
| *** trown|lunch is now known as trown | 17:54 | |
| *** salv-orlando has quit IRC | 17:56 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-diskimage_builder master: Add zypper package but not for trusty https://review.openstack.org/467697 | 17:58 |
| dirk | pabelanger: cool! | 17:58 |
| dirk | pabelanger: is it normal that the build logs are 0 bytes? | 17:58 |
| dirk | pabelanger: I'm looking at http://nb03.openstack.org/ | 17:58 |
| *** sshnaidm is now known as sshnaidm|afk | 17:58 | |
| pabelanger | dirk: that is because we haven't build any of those images yet | 17:59 |
| dirk | aren't they rebuilt daily? | 17:59 |
| pabelanger | yes, but we have 4 builders right now | 17:59 |
| *** esikachev has quit IRC | 17:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-vpnaas: Move tempest job from experimental to non-voting https://review.openstack.org/453015 | 17:59 |
| pabelanger | until we take nb01 and nb02 out of rotation, they are still holding some builds | 17:59 |
| *** thorst has quit IRC | 18:00 | |
| pabelanger | but they are currently stopped, so nb03.o.o and nb04.o.o are slowly rotating out image builds | 18:00 |
| pabelanger | we also only do 1 build at a time | 18:00 |
| pabelanger | per server | 18:00 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Python3: RuntimeError: dictionary changed size during iteration https://review.openstack.org/466049 | 18:01 |
| dirk | pabelanger: ok, as expected I'd be super excited to see an opensuse image to build.. do you think we can sneak that in? | 18:01 |
| dirk | I'd actually like to propose an experimental job somewhere to see if we can make it do something | 18:01 |
| dirk | actually that's the part I'm unclear about, how do we go on that | 18:02 |
| pabelanger | dirk: if we get all builds over to nb03 / nb04 today, we likely can start a opensuse build. Otherwise, likely tomorror | 18:02 |
| fungi | mtreinish: i haven't forgotten about the health api frequency analysis, but these are the top 100 most requested paths according to the current log retention. i haven't had time to work on response size analysis yet since i've determined that i'll need to switch from a shell script to python to do that bit (for the sake of my own sanity)... http://paste.openstack.org/show/610590 | 18:04 |
| *** jpena is now known as jpena|off | 18:04 | |
| fungi | (the numbers on the left are the number of times they were requested) | 18:05 |
| dirk | pabelanger: ok, sounds good | 18:05 |
| pabelanger | dirk: we'll bring bindep tests online first, that should help show how to do an experimental job | 18:05 |
| fungi | mtreinish: that's ~1 year of logs analyzed btw. if you want it to cover a shorter period, lmk | 18:06 |
| openstackgerrit | Merged openstack-infra/system-config master: Refresh nodepool-builder.logging.conf file https://review.openstack.org/454793 | 18:07 |
| inc0 | hey, I need some help debugging issue with gate guys:( | 18:07 |
| inc0 | http://logs.openstack.org/07/466007/68/experimental/gate-kolla-ansible-dsvm-deploy-multinode-oraclelinux-source-centos-7-2-node-nv/4d718a5/ I | 18:07 |
| inc0 | I'm pretty sure we have bunch of stuff in /tmp/logs | 18:08 |
| inc0 | but it doesn't show there as dir | 18:08 |
| *** esikachev has joined #openstack-infra | 18:10 | |
| *** salv-orlando has joined #openstack-infra | 18:10 | |
| pabelanger | inc0: you'll need to make sure the publisher in JJB is setup to collect logs from there | 18:10 |
| *** e0ne has joined #openstack-infra | 18:11 | |
| inc0 | pabelanger: I'm pretty sure it is, it's how it was working befor | 18:11 |
| inc0 | e | 18:11 |
| *** ramishra has joined #openstack-infra | 18:11 | |
| *** rbrndt has quit IRC | 18:11 | |
| pabelanger | if you copy them to WORKSPACE/logs devstack-logs will pick them up | 18:12 |
| *** Rockyg has quit IRC | 18:12 | |
| pabelanger | http://logs.openstack.org/07/466007/68/experimental/gate-kolla-ansible-dsvm-deploy-multinode-oraclelinux-source-centos-7-2-node-nv/4d718a5/_zuul_ansible/ansible_log.txt | 18:12 |
| pabelanger | shows permission denied | 18:12 |
| pabelanger | you need to make sure your log files are jenkins | 18:12 |
| *** ijw has joined #openstack-infra | 18:13 | |
| pabelanger | jenkins:jenkins permissions | 18:13 |
| pabelanger | and 0644 | 18:13 |
| inc0 | hmm | 18:13 |
| dhinesh | i was following the following instructions https://docs.openstack.org/contributor-guide/additional-git-workflow/rebase.html to resolve merge conflicts | 18:13 |
| dhinesh | git pull --ff-only origin master | 18:13 |
| inc0 | cool, now I can debug it, thanks | 18:13 |
| dhinesh | From github.com:openstack/cinder | 18:14 |
| dhinesh | * branch master -> FETCH_HEAD | 18:14 |
| dhinesh | fatal: Not possible to fast-forward, aborting. | 18:14 |
| *** ram5391 has quit IRC | 18:14 | |
| dhinesh | throws error like above | 18:14 |
| dhinesh | any way to resolve this | 18:14 |
| mordred | fungi: in https://review.openstack.org/#/c/462815 we describe a well-known location for publication of the service-type data. the current proposal is https://specs.openstack.org/service-types.json ... whatever we use will need to be a URL that sticks around fairly much indefinitely ... do you think that url is ok? or should we come up with a different one? | 18:14 |
| *** esikachev has quit IRC | 18:16 | |
| pabelanger | dhinesh: have you committed code to master branch locally? | 18:16 |
| pabelanger | dhinesh: you likely need to git rebase origin/master | 18:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 18:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Wrap map() in list() for python3 https://review.openstack.org/466069 | 18:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Python3: encode / decode data as utf8 https://review.openstack.org/466065 | 18:17 |
| dhinesh | pabelanger: i don't think so, i have been commiting changes to my branch | 18:18 |
| dhinesh | pabelanger: do i do it after switching to master branch | 18:19 |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Allow nova servers to reach heat services on port 80 https://review.openstack.org/467703 | 18:19 |
| *** thorst has joined #openstack-infra | 18:19 | |
| pabelanger | dhinesh: what branch are you on now? | 18:20 |
| *** bobh has quit IRC | 18:21 | |
| pabelanger | dhinesh: if --ff-only fails, you should be able to rebase next | 18:21 |
| dhinesh | pabelanger: on branch 'bp/hedvig-cinder' | 18:21 |
| pabelanger | dhinesh: try git rebase origin/master | 18:21 |
| fungi | mordred: i'd rather not encumber the specs.o.o site with expectations that random applications will rely on it being stable/present long-term. i think we should create a dedicated domain name we can use for this, which gives us a lot of flexibility about where and how we serve it in the future | 18:21 |
| pabelanger | dhinesh: step 5 | 18:22 |
| fungi | mordred: i'll leave a comment on the proposal | 18:22 |
| *** salv-orlando has quit IRC | 18:22 | |
| *** electrofelix has quit IRC | 18:22 | |
| jeblair | bikeshed.openstack.org | 18:23 |
| fungi | yup | 18:23 |
| jeblair | i threw that out there as a joke. it's kind of growing on me though. | 18:24 |
| fungi | jeblair: that's what the debian community decided to call their ~equivalent of a ppa, after a particularly long bout of bikeshedding on the name | 18:25 |
| *** esikachev has joined #openstack-infra | 18:26 | |
| *** bobh has joined #openstack-infra | 18:26 | |
| *** ramishra has quit IRC | 18:26 | |
| jeblair | fungi: imagine "bikeshed.ietf.org/rfc2606" | 18:26 |
| *** rcarrillocruz has quit IRC | 18:27 | |
| *** ram5391 has joined #openstack-infra | 18:27 | |
| *** Swami has joined #openstack-infra | 18:28 | |
| fungi | i credited you with that fallback recommendation | 18:29 |
| jeblair | haha :) | 18:30 |
| jeblair | i feel like i owe the conversation a serious offering, so: registry.openstack.org | 18:31 |
| jeblair | having done that... /me votes for bikeshed | 18:32 |
| *** spzala has quit IRC | 18:32 | |
| fungi | yeah, i figure something sensible like registry.o.o or sta.o.o (the "service types authority" is what the proposal calls it) will end up being chosen | 18:33 |
| fungi | but it's possible we'll want to host similar client-facing global metadata there, in which case making the name service-types specific could be a poor choice i guess | 18:33 |
| openstackgerrit | Matthew Thode proposed openstack-infra/irc-meetings master: update meeting time https://review.openstack.org/467711 | 18:35 |
| clarkb | pabelanger: there are other localrc rhings further down that will undo your disable of swift | 18:35 |
| clarkb | pabelanger: I would comment inline but on phone right now | 18:36 |
| *** rcarrillocruz has joined #openstack-infra | 18:36 | |
| pabelanger | clarkb: k, let me check | 18:36 |
| pabelanger | clarkb: I see, let me move the code block | 18:36 |
| openstackgerrit | Matthew Thode proposed openstack-infra/irc-meetings master: update meeting time for requirements team https://review.openstack.org/467711 | 18:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable swift for py35 nodepool dsvm job https://review.openstack.org/467685 | 18:39 |
| pabelanger | clarkb: thanks, updated | 18:40 |
| *** ram5391 has quit IRC | 18:41 | |
| *** jascott1 has joined #openstack-infra | 18:42 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/irc-meetings master: update meeting time for requirements team https://review.openstack.org/467711 | 18:43 |
| *** rlandy is now known as rlandy|afk | 18:43 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:43 | |
| mtreinish | fungi: great, thanks. I had forgotten about it too :) | 18:44 |
| mtreinish | fungi: ooh interesting, the rss feeds are popular | 18:45 |
| mtreinish | I wasn't convinced anyone was using them | 18:45 |
| *** jascott1 has quit IRC | 18:46 | |
| mtreinish | I find it interesting that /status was called 94 times. That's just a health check for the api service | 18:46 |
| fungi | well, 94x is roughly one hit every four days on average | 18:47 |
| *** e0ne has quit IRC | 18:47 | |
| mtreinish | even that is pretty frequent | 18:47 |
| mtreinish | fungi: I mean: http://health.openstack.org/status all it says is whether it can talk to the db and es | 18:48 |
| mtreinish | I only ever need to look at it if things aren't working | 18:48 |
| pabelanger | ianw: where did we end up with stackviz package? I'd like to finishing the removal of npm from our base images | 18:49 |
| mtreinish | fungi: also / was called 4852 times? I think more people are playing around with the api then I expected | 18:50 |
| mtreinish | because that's just an index of all the endpoints, and not advertised everywhere | 18:50 |
| mtreinish | err that or just crawlers I guess | 18:50 |
| *** rbrndt has joined #openstack-infra | 18:53 | |
| *** csomerville has joined #openstack-infra | 18:53 | |
| jeblair | i love than in firefox the json display shows the word "green" in red type. | 18:53 |
| openstackgerrit | Matthew Thode proposed openstack-infra/irc-meetings master: update meeting time for requirements team https://review.openstack.org/467711 | 18:53 |
| jeblair | it's like one of those psych experiments | 18:53 |
| mordred | fungi, jeblair: awesome. thanks! | 18:54 |
| mtreinish | jeblair: hah, I didn't even notice that | 18:55 |
| * mtreinish misses the days when firefox didn't render the json for him | 18:56 | |
| pabelanger | Hmm, there seems to be an issue with the proposal node | 18:57 |
| *** Guest41638 is now known as zeus | 18:57 | |
| pabelanger | checking logs | 18:57 |
| *** zeus has quit IRC | 18:57 | |
| *** zeus has joined #openstack-infra | 18:57 | |
| *** cody-somerville has joined #openstack-infra | 18:57 | |
| pabelanger | OFFLINE_NODE_WHEN_COMPLETE": "1" | 18:58 |
| pabelanger | :( | 18:58 |
| pabelanger | guess we added a new job | 18:58 |
| pabelanger | ansible it looks like | 18:58 |
| *** rbrndt has quit IRC | 19:00 | |
| fungi | guess we never reached an agreement on just making ziil-launcher so that offlining can be disabled for all of zlstatic01's workers | 19:01 |
| fungi | s/ziil/zuul/ | 19:01 |
| *** csomerville has quit IRC | 19:01 | |
| AJaeger | mmh, that ansible change updated zuul/openstack_functions.py - so, what's broken with the regex? | 19:01 |
| pabelanger | ya, bad regex | 19:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix python3 issues in functional tests https://review.openstack.org/467693 | 19:01 |
| pabelanger | incoming patch | 19:01 |
| *** thorst has quit IRC | 19:02 | |
| dhinesh | it worked pabelanger: thanks | 19:02 |
| AJaeger | pabelanger: I see it as well now ... Thanks | 19:02 |
| *** rbrndt has joined #openstack-infra | 19:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix regext for propose-openstack-ansible-update-osa-test-scripts https://review.openstack.org/467720 | 19:03 |
| pabelanger | AJaeger: fungi: ^ | 19:03 |
| pabelanger | fungi: ya, I had a patch up for that. Let me see what every happened to it | 19:03 |
| AJaeger | pabelanger: thanks, +2 | 19:03 |
| *** eharney has quit IRC | 19:04 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Python3: encode / decode data as utf8 https://review.openstack.org/466065 | 19:06 |
| pabelanger | okay, zlstatic01 restarted | 19:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix regex for propose-openstack-ansible-update-osa-test-scripts https://review.openstack.org/467720 | 19:07 |
| AJaeger | fungi, pabelanger proposed two new repos and this needs PTL review by you: https://review.openstack.org/#/c/467300/ | 19:07 |
| fungi | thanks | 19:07 |
| AJaeger | pabelanger: if there are more propose-openstack-ansible-update-osa-test-scripts jobs currently queued, you need to restart again. I don't see a job in the queue, so we should be good - but please check again later. | 19:09 |
| pabelanger | AJaeger: agree | 19:10 |
| *** hashar has joined #openstack-infra | 19:16 | |
| *** eharney has joined #openstack-infra | 19:17 | |
| pabelanger | clarkb: mordred: Our nodepool-dsvm jobs have been pretty stable over the last month or so, do you think it is time to make them voting? | 19:20 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Reduce citycloud-sto2 to 40 max servers https://review.openstack.org/467722 | 19:20 |
| clarkb | pabelanger: I've always been a fan of having them voting, but I know jeblair had reservations | 19:20 |
| clarkb | pabelanger: fungi mordred ^ this change to citycloud should clear out the errors in the grafana graphs, hitting quota limits due to having at least one test node out there for the network thing | 19:21 |
| pabelanger | +3 | 19:21 |
| pabelanger | err | 19:21 |
| pabelanger | +2 | 19:21 |
| mordred | +3 | 19:21 |
| *** hashar has quit IRC | 19:21 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Wrap map() in list() for python3 https://review.openstack.org/466069 | 19:24 |
| *** hashar has joined #openstack-infra | 19:25 | |
| AJaeger | fungi, on the two new repos: You only gave +2 instead of +2A - any special reasons? Otherwise I'll +2A now... | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 19:26 |
| pabelanger | mordred: clarkb: mind a review on https://review.openstack.org/#/c/467697/ adds zypper to puppet-diskimage-builder | 19:26 |
| pabelanger | going to give an opensuse-minimal build a try | 19:26 |
| *** thorst has joined #openstack-infra | 19:26 | |
| fungi | AJaeger: i didn't approve because you had held off reviewing so i was giving you a chance to record your vote | 19:26 |
| fungi | AJaeger: approve at will! | 19:27 |
| *** bulat has joined #openstack-infra | 19:27 | |
| mordred | pabelanger: wfm | 19:27 |
| bulat | Hi, Folks please review https://review.openstack.org/#/c/463554/ | 19:27 |
| pabelanger | oh | 19:27 |
| pabelanger | I might have typo | 19:27 |
| pabelanger | zypper | 19:27 |
| AJaeger | fungi: done | 19:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-diskimage_builder master: Add zypper package but not for trusty https://review.openstack.org/467697 | 19:28 |
| clarkb | pabelanger: too many whys I mean 'y's | 19:28 |
| pabelanger | mordred: clarkb: sorry^ | 19:28 |
| clarkb | pabelanger: np +2 | 19:29 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix regex for propose-openstack-ansible-update-osa-test-scripts https://review.openstack.org/467720 | 19:29 |
| *** thorst has quit IRC | 19:31 | |
| *** cody-somerville has quit IRC | 19:31 | |
| *** cody-somerville has joined #openstack-infra | 19:32 | |
| clarkb | jeblair: I'm looking at the etherpad for lists.o.o snapshotting and it doesn't list the services to disable (but not stop). Looking at ps output it appears that exim and mailman are the two I want ot disable, are there others? | 19:32 |
| *** bulat has quit IRC | 19:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable swift for py35 nodepool dsvm job https://review.openstack.org/467685 | 19:34 |
| clarkb | also I love that ubunut has like 5 different ways to manage this in trusty | 19:34 |
| clarkb | I can't wait to be systemd everywhere if only to avoid this confusion | 19:35 |
| cmurphy | pabelanger: clarkb want to review 452558 466689 and 466690 for me? last few things i need to cross this project off my list | 19:35 |
| *** felipemonteiro_ has joined #openstack-infra | 19:36 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: DNM: testing global_request_id https://review.openstack.org/467417 | 19:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Create zuul-base-jobs and zuul-jobs https://review.openstack.org/467300 | 19:37 |
| *** masayukig has quit IRC | 19:38 | |
| *** e0ne has joined #openstack-infra | 19:38 | |
| *** e0ne has quit IRC | 19:39 | |
| *** felipemonteiro has quit IRC | 19:39 | |
| clarkb | cmurphy: yup looking | 19:39 |
| *** darvon has quit IRC | 19:39 | |
| cmurphy | ty | 19:39 |
| *** darvon has joined #openstack-infra | 19:39 | |
| clarkb | oh looks like pabelanger and mordred are beating me to it | 19:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Reduce citycloud-sto2 to 40 max servers https://review.openstack.org/467722 | 19:40 |
| clarkb | re https://review.openstack.org/#/c/452558/ do we need the certname for reports generation? | 19:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove references to osic-cloud8 https://review.openstack.org/467732 | 19:42 |
| pabelanger | clarkb: don't think so, we are in the process of shutting down puppetboard | 19:42 |
| cmurphy | clarkb: well report generation doesn't work, so i wasn't worried about it, but also i think http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/lib/puppet/reports/puppetdb_file.rb just grabs the hostname and doesn't need any of the puppet parameters | 19:42 |
| pabelanger | we decided not to upgrade it to trusty | 19:43 |
| mtreinish | pabelanger: I still miss puppetboard, it made my life a lot easier | 19:43 |
| pabelanger | https://review.openstack.org/#/c/449167/ | 19:43 |
| pabelanger | I need to rebase it | 19:44 |
| pabelanger | mtreinish: I'd like to get a spec up for ARA server | 19:44 |
| clarkb | I seem to recall weirdness if certname wasn't set, but maybe that was always within the context of the master and agent | 19:44 |
| pabelanger | mtreinish: but requires from changes to ARA for database | 19:44 |
| cmurphy | clarkb: there are no templates using that parameter so this wouldn't change any cert names on disk | 19:45 |
| mtreinish | pabelanger: honestly it doesn't really help much. I can get the same data ara gets from firehose.o.o, but we turn off the stdout for puppet so it just says puppet ran | 19:48 |
| *** dhajare has joined #openstack-infra | 19:48 | |
| clarkb | cmurphy: oh yup grep confirms | 19:48 |
| *** openstackgerrit has quit IRC | 19:48 | |
| clarkb | so should be safe then | 19:48 |
| mtreinish | pabelanger: the value of puppetboard for me was I could see where I screwed up things. So if there was a failure I could identify what went wrong | 19:49 |
| mtreinish | since I don't have access to any of the logs | 19:49 |
| *** salv-orlando has joined #openstack-infra | 19:49 | |
| pabelanger | mtreinish: why did we disable puppet stdout on firehose? | 19:49 |
| sc` | yay. more timeouts. more rechecks. how can i help move this forward so i'm not spending months spinning my wheels on a single change? | 19:50 |
| fungi | pabelanger: facter was echoing megabytes of useless data for every host into the stdout apparently | 19:50 |
| mtreinish | pabelanger: I assumed that was how we setup the ansible role. At least last time I looked stdout was blank on run puppet tasks | 19:50 |
| mtreinish | fungi: heh, well it still does that :) | 19:51 |
| cmurphy | megabytes? | 19:51 |
| cmurphy | that's a lot of facts | 19:51 |
| mtreinish | cmurphy: all the system facts for every node in infracloud | 19:51 |
| mtreinish | I'll start listening to the firehose and see if I can catch a run_puppet task go through | 19:51 |
| fungi | cmurphy: multiplied across all the hosts it's megabytes per run at least | 19:51 |
| sc` | good thing you're not using chef where you're using puppet. ohai would pound the shit out of things | 19:52 |
| fungi | somewhere within an order of magnitude at least | 19:52 |
| *** openstackgerrit has joined #openstack-infra | 19:52 | |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder master: Add zypper package but not for trusty https://review.openstack.org/467697 | 19:52 |
| fungi | sc`: what project/job? is the job running too long and hitting a timeout, or are you talking about some manner of network timeout reaching something during the job? | 19:52 |
| pabelanger | sc`: if you can come up with the proxypass configuration for rubygems.org, and test gems are cached, that will speed things up: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n95 | 19:53 |
| mtreinish | pabelanger: http://paste.openstack.org/show/610603/ that's what gets passed to the callback plugin (the mqtt callback just dumps it all) | 19:53 |
| sc` | fungi: chef openstack - we've been unable to reliably use our gates due to rubygems timeouts. the only thing that has reliably worked is some ugly ghetto wget/gem mess in a rakefile | 19:53 |
| pabelanger | mtreinish: ya, I think we disable by default now. I thought we did talk about setting up a syslog server | 19:53 |
| sc` | mirroring all the gems we need and stuffing them in a cookbook is a big anti-pattern that i'm not willing to enforce | 19:54 |
| *** jamesmca_ has joined #openstack-infra | 19:54 | |
| sc` | integration gates have been failing consistently for roughly a year, give or take a couple months, due to the rubygems.org dependency. it has resulted in people thinking chef openstack is 100% dead | 19:55 |
| fungi | sc`: have we tried setting up the rubygems site on our caching proxies yet? | 19:55 |
| sc` | fungi: don't know. i pop in every few months when i have a lull | 19:55 |
| pabelanger | no, on my list to do | 19:55 |
| pabelanger | should be 3 lines of apache, but need to test | 19:56 |
| *** trown is now known as trown|brb | 19:56 | |
| pabelanger | if they are using CND, will be harder | 19:56 |
| *** Apoorva has quit IRC | 19:56 | |
| cmurphy | ty clarkb | 19:56 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix issues in OVB preparement servers scripts https://review.openstack.org/467620 | 19:57 |
| sc` | like i said, i'm patient. downstream seems to be patient while they're rocking eol releases of openstack, but that luck cannot hold | 19:58 |
| *** esikachev has quit IRC | 19:58 | |
| sc` | at some point, i'll have no users to support and fewer reasons to keep maintaining what i maintain | 19:59 |
| pabelanger | http://paste.openstack.org/show/610604/ | 19:59 |
| pabelanger | that should be it but somebody needs to validate it with apache | 19:59 |
| *** ym200 has joined #openstack-infra | 19:59 | |
| pabelanger | sc`: if you want to test and report back, you should see gems getting cached | 19:59 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n95 | 20:00 |
| pabelanger | if our apache configuration | 20:00 |
| sc` | i'll try when i'm back home. my available hardware is limited to a somewhat senior citizen macbook pro | 20:00 |
| pabelanger | but I suspect they use a CDN, which means we need to do a little more work | 20:00 |
| sc` | afaik rubygems is all hosted on aws | 20:02 |
| sc` | eg. https://github.com/rubygems/rubygems-aws/blob/7a027d6ccf2ce2834eace772e72a98e8e6243743/chef/site-cookbooks/rubygems/templates/default/nginx_balancer.conf.erb#L185-L203 | 20:02 |
| openstackgerrit | Merged openstack-infra/system-config master: Clean up puppetmaster cruft https://review.openstack.org/452558 | 20:02 |
| sc` | (yay, they use chef!) | 20:02 |
| *** Apoorva has joined #openstack-infra | 20:03 | |
| pabelanger | then we'll need to do the same things we did for docker registry | 20:03 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 20:03 |
| pabelanger | likely add cloudfront to proxypassreverse | 20:03 |
| jeblair | pabelanger, clarkb: i'd like to defer revisiting the nodepool-devstack voting decision until after v3 is released, if that's okay. | 20:03 |
| jeblair | clarkb: exim+mailman should do it. | 20:03 |
| *** trown|brb is now known as trown | 20:03 | |
| pabelanger | wfm! | 20:03 |
| mordred | ++ | 20:04 |
| *** cody-somerville has quit IRC | 20:04 | |
| sc` | ooh. no. pabelanger - dns shows rubygems.org is all on fastly | 20:04 |
| *** cody-somerville has joined #openstack-infra | 20:04 | |
| sc` | tricksy hobbitses | 20:04 |
| clarkb | jeblair: so you just update-rc.d exim4/mailmain disable, take snapshot, then update-rc.d exim4/mailman enable ? | 20:07 |
| jeblair | clarkb: yes | 20:09 |
| *** e0ne has joined #openstack-infra | 20:09 | |
| clarkb | ok doing that now, and will then boot snapshot and get bkero's key on the server | 20:09 |
| clarkb | bkero: ^ can you point me at a key at some point? | 20:09 |
| fungi | actually, bkero should propose his key to the system-config repo and agree to our guidelines | 20:10 |
| bkero | Okay, can do that too. | 20:10 |
| *** bobh has quit IRC | 20:10 | |
| * bkero gets his system-config repo up-to-date. | 20:10 | |
| fungi | this snapshot will have some sensitive data on it (ml moderator/admin passwords, archives for some private-recipient lists, et cetera) | 20:10 |
| * bkero nods | 20:10 | |
| *** ym200 has quit IRC | 20:11 | |
| fungi | so just making sure we have it on record this way | 20:11 |
| clarkb | sounds good | 20:11 |
| *** eharney has quit IRC | 20:11 | |
| fungi | bkero: so an addition at the end of https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/users.pp | 20:12 |
| mmedvede | hi sdague, would you reconsider https://review.openstack.org/#/c/467437/ (etcd download url override). There is no need to add sha256sum to d-g to make it work. | 20:12 |
| sdague | mmedvede: ok, +2 | 20:13 |
| fungi | bkero: and in the commit message mention that you've read and agree to follow https://docs.openstack.org/infra/system-config/sysadmin.html#ssh-access | 20:13 |
| mmedvede | sdague: thanks | 20:13 |
| clarkb | `sudo update-rc.d mailman disable` and `sudo update-rc.d exim4 disable` has been run. I am grabbing a snapshot now | 20:14 |
| *** jamesmca_ has quit IRC | 20:15 | |
| *** rlandy|afk is now known as rlandy | 20:17 | |
| openstackgerrit | Ben Kero proposed openstack-infra/system-config master: Add bkero to users.pp https://review.openstack.org/467753 | 20:18 |
| bkero | fungi, clarkb: ^ | 20:18 |
| mordred | bkero: you whitespace weird!!! | 20:19 |
| bkero | I do? | 20:19 |
| * bkero looks | 20:19 | |
| bkero | Oh bother. It didn't look that way in my terminal. | 20:19 |
| bkero | Gah why are those tabs. | 20:19 |
| * bkero fixes | 20:19 | |
| openstackgerrit | Ben Kero proposed openstack-infra/system-config master: Add bkero to users.pp https://review.openstack.org/467753 | 20:20 |
| openstackgerrit | Ben Kero proposed openstack-infra/system-config master: Add bkero to users.pp https://review.openstack.org/467753 | 20:21 |
| fungi | lgtm | 20:24 |
| fungi | thanks bkero! | 20:24 |
| bkero | :) | 20:25 |
| clarkb | bkero: we are mostly just waiting for the cloud to cloud faster now. image snapshot is "queued" | 20:25 |
| bkero | heh | 20:26 |
| fungi | yeah, these ancient rackspace flavors with the giant rootfs do take a while to snapshot | 20:26 |
| bkero | Go disks go | 20:26 |
| *** rcarrillocruz has quit IRC | 20:27 | |
| fungi | but if we want to keep the same ip address with its established sender history, that's the price we pay (at least until we can't keep doing it anyway) | 20:28 |
| bkero | I'm sure everyone's greylisting will thank you. | 20:28 |
| *** rossella__ has quit IRC | 20:32 | |
| *** mat128 has quit IRC | 20:34 | |
| *** rcarrillocruz has joined #openstack-infra | 20:35 | |
| *** dprince has quit IRC | 20:39 | |
| *** rossella__ has joined #openstack-infra | 20:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Start building opensuse based nodepool images https://review.openstack.org/412065 | 20:40 |
| *** dimtruck is now known as zz_dimtruck | 20:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove stable compat jobs from oslo libs https://review.openstack.org/467760 | 20:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Sync latest nodepool-builder log file https://review.openstack.org/467762 | 20:42 |
| pabelanger | clarkb: fungi: ^do you mind +3. Latest changes for nodepool-builder logging | 20:43 |
| dirk | pabelanger: are you sure only zypper is needed? I remember i had to install more stuff, like for example gnupg2 | 20:43 |
| pabelanger | dirk: I am not, but we'll find out shortly | 20:43 |
| pabelanger | dirk: is there an ubuntu bug to fix those dependencies? | 20:44 |
| dirk | Sorry didn't have time to comment on this in time for the previous review | 20:44 |
| dirk | pabelanger: yes | 20:44 |
| pabelanger | dirk: k, if you link it I can add it for puppet-diskimage_builder | 20:44 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/462712 | 20:44 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 20:44 |
| pabelanger | dirk: gnupg is currently installed | 20:45 |
| clarkb | bkero: now it is "saving" | 20:46 |
| *** coreyob has quit IRC | 20:47 | |
| fungi | pabelanger: 467762 is just the output of running the script, right? | 20:47 |
| *** esikachev has joined #openstack-infra | 20:48 | |
| pabelanger | fungi: yup | 20:48 |
| pabelanger | tools/nodepool_log_config.py | 20:48 |
| fungi | cool, then i'll just +3. no need to burn more core reviewers on something like that | 20:48 |
| pabelanger | great | 20:48 |
| dirk | pabelanger: zypper uses the 2.x version explicitly, which isn't the default on ubuntu | 20:48 |
| *** pvaneck has joined #openstack-infra | 20:49 | |
| *** salv-orlando has quit IRC | 20:49 | |
| *** trown is now known as trown|outtypewww | 20:49 | |
| pabelanger | dirk: k, so if you want to patch puppet-diskimage_builder and leave link to upstream bug that helps | 20:49 |
| dirk | pabelanger: i remember i did a pretty nasty hack for that somewhere with a reference to the bug report, just need to find it | 20:49 |
| *** e0ne has quit IRC | 20:49 | |
| dirk | pabelanger: https://bugs.launchpad.net/bugs/1639428 | 20:50 |
| openstack | Launchpad bug 1639428 in zypper (Ubuntu) "zypper doesn't work with official opensuse repositories" [Medium,Confirmed] | 20:50 |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack master: Fix puppet lint errors https://review.openstack.org/460787 | 20:50 |
| *** dhajare has quit IRC | 20:50 | |
| *** esikachev has quit IRC | 20:52 | |
| *** bobh has joined #openstack-infra | 20:52 | |
| *** jamesmca_ has joined #openstack-infra | 20:54 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/462712 | 20:57 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 20:57 |
| *** bobh has quit IRC | 20:57 | |
| openstackgerrit | Merged openstack-infra/shade master: Fix python3 issues in functional tests https://review.openstack.org/467693 | 20:58 |
| *** krtaylor has quit IRC | 20:59 | |
| *** jamesmca_ has quit IRC | 20:59 | |
| *** esikachev has joined #openstack-infra | 20:59 | |
| openstackgerrit | Merged openstack-infra/system-config master: Sync latest nodepool-builder log file https://review.openstack.org/467762 | 21:01 |
| *** Goneri has quit IRC | 21:02 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/puppet-diskimage_builder master: Add gnupg2 install for zypper as well https://review.openstack.org/467768 | 21:03 |
| dirk | pabelanger: clarkb : cmurphy : ^^ | 21:03 |
| *** annegentle has quit IRC | 21:05 | |
| mordred | dirk: that seems like broken packaging in ubuntu for zypper - but I suppose that's life :) | 21:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove puppet / bindep-cache from opensuse DIB https://review.openstack.org/467770 | 21:06 |
| pabelanger | dirk: thanks | 21:07 |
| pabelanger | +3 | 21:07 |
| pabelanger | mordred: clarkb: ^467770 was missed when we did our depuppet for opensuse | 21:08 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Don't break root ssh with authorized_keys copy https://review.openstack.org/467773 | 21:10 |
| clarkb | pabelanger: mordred ^ noticed that when looking more into citycloud private ip stuff | 21:10 |
| *** Julien-zte has joined #openstack-infra | 21:10 | |
| clarkb | would appreciate your review as it involves ansible. But tldr is now that we rely on root authorized keys to ssh multinode testing breaks that once d-g runs | 21:10 |
| pabelanger | just stepping away for some food, will pick it up if still open when I return | 21:11 |
| clarkb | pabelanger: +2 | 21:11 |
| dirk | mordred: the zypper package in ubuntu is fairly broken indeed. I had to spend quite some hoops to get diskimage-builder install a opensuse env quickly so that we can do the actual image build with a working zypper | 21:11 |
| *** fguillot has quit IRC | 21:11 | |
| clarkb | bkero: snapshot is still saving | 21:12 |
| bkero | clarkb: How big is that thing? | 21:12 |
| *** ldnunes has quit IRC | 21:12 | |
| clarkb | bkero: 26GB used | 21:12 |
| bkero | oh joy | 21:13 |
| fungi | "used" probably makes little impact here | 21:13 |
| fungi | it's a 160gb rootfs | 21:13 |
| dirk | pabelanger: argh, the comment was backwards :-/ | 21:13 |
| *** esikachev has quit IRC | 21:13 | |
| dirk | anyway | 21:13 |
| fungi | odds are the backing store lost track of "unused" blocks for that instance long ago | 21:13 |
| fungi | in its lifetime, it's probably written to most blocks at least once | 21:14 |
| fungi | unless something is running in the instance to actively zero unused blocks | 21:14 |
| fungi | which seems unlikely (does rackspace's nova-agent do that maybe?) | 21:14 |
| *** esikachev has joined #openstack-infra | 21:15 | |
| clarkb | I doubt it, but using fstrim is likely something we could test | 21:16 |
| fungi | or `cp /dev/zero /filler;rm /filler;sync` before snapshotting | 21:16 |
| fungi | assuming they even do any dedupe or zero-block identification | 21:17 |
| *** zz_dimtruck is now known as dimtruck | 21:17 | |
| *** salv-orlando has joined #openstack-infra | 21:19 | |
| *** srobert has quit IRC | 21:19 | |
| *** dave-mccowan has quit IRC | 21:20 | |
| *** esikachev has quit IRC | 21:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Don't break root ssh with authorized_keys copy https://review.openstack.org/467773 | 21:22 |
| *** cody-somerville has quit IRC | 21:23 | |
| *** gardlt has quit IRC | 21:23 | |
| *** priteau has quit IRC | 21:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Wait for merger to complete before item is ready https://review.openstack.org/463054 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add default-branch property to projects https://review.openstack.org/467334 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger https://review.openstack.org/461176 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor https://review.openstack.org/461177 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to jobs https://review.openstack.org/467775 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Return resulting commits from merger https://review.openstack.org/467776 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use the executor cached repos more often https://review.openstack.org/467777 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Check out the appropriate branch in executor https://review.openstack.org/467778 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Enable test_one_branch cloner test https://review.openstack.org/467779 | 21:25 |
| *** bobh has joined #openstack-infra | 21:27 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Delete HPCloud instructions https://review.openstack.org/466689 | 21:27 |
| *** esikachev has joined #openstack-infra | 21:28 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Update README for DIB-based images https://review.openstack.org/466690 | 21:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Disable leaked repo check https://review.openstack.org/466810 | 21:29 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Stop suggesting people use the nova command line client https://review.openstack.org/466702 | 21:29 |
| *** pradk has quit IRC | 21:29 | |
| *** salv-orlando has quit IRC | 21:30 | |
| *** pradk has joined #openstack-infra | 21:31 | |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder master: Add gnupg2 install for zypper as well https://review.openstack.org/467768 | 21:31 |
| *** esikachev has quit IRC | 21:32 | |
| clarkb | bkero: I haven't forgotten you, still saving :/ | 21:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bring opensuse-42.2 DIB online https://review.openstack.org/467782 | 21:35 |
| pabelanger | clarkb: mordred: fungi: ^okay, that should allow for single node opensuse-42.2 | 21:37 |
| pabelanger | we'll also need https://review.openstack.org/#/c/467770 | 21:37 |
| *** esikachev has joined #openstack-infra | 21:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Cleanup failed upload records https://review.openstack.org/467783 | 21:39 |
| mordred | pabelanger: \o/ | 21:39 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: First Pass: De-client-ify list_users https://review.openstack.org/467784 | 21:40 |
| clarkb | pabelanger: can we drop the .2? we don't use it for centos and avoiding .'s in hostnames is gneerally a good idea | 21:41 |
| *** esikachev has quit IRC | 21:42 | |
| clarkb | bkero: image is active now. Going to get it booted and manually add your key | 21:42 |
| pabelanger | clarkb: would we ever have 42.2 and 42.3 at the same time? | 21:45 |
| clarkb | pabelanger: I'm hoping no if thsoe updates happen like centos' do | 21:46 |
| clarkb | dirk: ^ | 21:46 |
| pabelanger | I think 42.2 is the actually release: http://mirror.dfw.rax.openstack.org/opensuse/distribution/leap/42.2/ | 21:46 |
| pabelanger | but, not sure how opensuse works | 21:47 |
| *** thorst has joined #openstack-infra | 21:47 | |
| clarkb | I just tumbleweed so releases are foreign to me in suse | 21:47 |
| dirk | pabelanger: clarkb : it think we'll only handle ine release long term | 21:47 |
| clarkb | pabelanger: if we need to keep the .2 I would make it -2 | 21:47 |
| *** bobh has quit IRC | 21:47 | |
| dirk | But therr will be a time where we need 42.2 and 42.3 at the same time | 21:48 |
| clarkb | dirk: so its not like how rhel/centos do their releases where they just sort of blend into one another over time? | 21:48 |
| fungi | yeah, i didn't even think about the fact that we encode image names into the hostnames | 21:49 |
| clarkb | dirk: they are distinct and maintained separately? | 21:49 |
| dirk | clarkb: yes, correct | 21:49 |
| fungi | so i agree avoiding the . is good | 21:49 |
| *** sbezverk_ has joined #openstack-infra | 21:49 | |
| pabelanger | k, let me change | 21:49 |
| fungi | dirk: does suse maintain series names for them we should use instead of version numbers? | 21:49 |
| dirk | fungi: not really | 21:49 |
| fungi | so far we do series names for ubuntu and debian, integers only for centos and fedora... | 21:49 |
| pabelanger | opensuse-42_2 or opensuse-42-2? | 21:49 |
| pabelanger | -2 might get consusing for multinode | 21:50 |
| fungi | - not _ if it's going in a hostname | 21:50 |
| clarkb | ya _ can't be in hostname | 21:50 |
| pabelanger | k | 21:50 |
| fungi | _ is technically invalid in the hostname part of a dns name | 21:50 |
| clarkb | or even do 422 | 21:50 |
| dirk | The next version is going to be -15 btw | 21:50 |
| *** esberglu has quit IRC | 21:50 | |
| clarkb | and drop the character entirely | 21:50 |
| * fungi resists the temptation to trot out another ietf rfc url | 21:50 | |
| dirk | Just to make it more confusing. | 21:50 |
| dirk | Going forward it will be fedora like sequentially numbered starting with 15 | 21:51 |
| *** esberglu has joined #openstack-infra | 21:51 | |
| clarkb | dirk: not 42-15 though just 15? | 21:51 |
| dirk | Just 15 | 21:51 |
| dirk | So its a downgrade | 21:51 |
| *** annegentle has joined #openstack-infra | 21:51 | |
| clarkb | will 15 then have .1 .2 .3 etc? | 21:52 |
| dirk | I think it won't bit that hasn't been decided yet | 21:52 |
| dirk | But | 21:52 |
| pabelanger | ya 422 seems better | 21:52 |
| dirk | I'm good with 422 | 21:52 |
| clarkb | gotcha then ya I would do 422 | 21:52 |
| dirk | I originally had it that way but something didn't like that | 21:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bring opensuse-422 DIB online https://review.openstack.org/467782 | 21:53 |
| pabelanger | Ya, I think I said use . last time | 21:53 |
| pabelanger | my mistake | 21:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Rename to opensuse-422 for DIB logging https://review.openstack.org/467786 | 21:54 |
| pabelanger | fixed logging too^ | 21:54 |
| *** kzaitsev_mb has quit IRC | 21:55 | |
| *** esberglu has quit IRC | 21:55 | |
| clarkb | bkero: fungi and now booting is slow I'm guessing becuase all that data must be copied :/ | 21:56 |
| fungi | yep! | 21:56 |
| pabelanger | fungi: if you mind the dependency too: 467770 | 21:57 |
| dirk | pabelanger: cool. So with that in place we can add a experimental job somewhere using that image? Any good suggestion? | 21:57 |
| pabelanger | dirk: we'll add bindep job first, to that is the only thing infra needs | 21:58 |
| clarkb | dirk: devstack job like the centos7 and fedora jobs I think | 21:58 |
| pabelanger | then if you want to maintain devstack job, ya centos7 / fedora are good examples | 21:58 |
| fungi | pabelanger: yep, sorry i missed spotting it earlier | 21:58 |
| dirk | pabelanger: most definitely | 21:58 |
| dirk | Bit i agree bindep is a good first start for debugging | 21:59 |
| clarkb | fwiw I think it would be awesome to have a tumbleweed image that we run daily tempest runs on | 21:59 |
| dirk | I did test that the image boots but not more | 21:59 |
| clarkb | as that will get all the new shiny things from everywherea nd we can see where the universe is broken in the future | 21:59 |
| fungi | right, we need to add the bindep integration jobs mainly to ensure that changes we make to the bindep-fallback.txt file and also to bindep itself won't break jobs running on that image | 21:59 |
| dirk | clarkb: i like that idea | 21:59 |
| dirk | clarkb: it's going to be an absolute nightmare | 22:00 |
| clarkb | dirk: yup :) | 22:00 |
| fungi | clarkb: i was pushing for the same for debian-sid, fwiw ;) | 22:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add opensuse-422 non-voting job for bindep https://review.openstack.org/467788 | 22:00 |
| clarkb | ya I think mtreinish has mentioned similar with arch too | 22:00 |
| fungi | (for the same reasons, it's what i run on my desktop so i'd like to know whether our tools continue to work there) | 22:00 |
| clarkb | basically the fact that it would be bropken a lot means it never happens | 22:00 |
| dirk | We used to run tempest against tumbleweed and it turned out to be more than a full time job just to understand what blew up every day | 22:00 |
| pabelanger | yes that is why fedora is experimental | 22:01 |
| pabelanger | you need full time people to watch it for sure | 22:01 |
| clarkb | but on the other hand there is a lot of desire for new kernel, new ovs, new libvirt, new new new | 22:01 |
| pabelanger | clarkb: sure, but nobody seems to want to maintain the jobs | 22:01 |
| *** salv-orlando has joined #openstack-infra | 22:01 | |
| clarkb | and if we can point them at $thing and say "ya its all broken and been on fire for a month" then maybe reality will sink in | 22:01 |
| fungi | the bleeding edge is always going to be a bloody mess | 22:01 |
| pabelanger | ++ | 22:01 |
| clarkb | pabelanger: ya I think you don't even maintain it, just every time libvirt says "you are testing on the wrong thing" you point them at that and say no we aren't :) | 22:02 |
| fungi | "we're closing your bug report. please reopen if you see the same issues when running with a custom-compiled kernel from trunk" | 22:02 |
| dirk | clarkb: 42.3 is a mix of tumbleweed, so it might be a good start | 22:03 |
| dirk | I would actually like to have that because it will be the future sle | 22:04 |
| pabelanger | okay, if we approve https://review.openstack.org/#/c/467782 I can keep an eye on it for a bit tonight | 22:04 |
| pabelanger | otherwise, pick it up in the morning | 22:04 |
| clarkb | pabelanger: I'll review it now | 22:05 |
| *** jkilpatr has quit IRC | 22:05 | |
| * mordred thinks we should just have a tumbleweed images | 22:06 | |
| clarkb | pabelanger: its approved and so is its parent | 22:06 |
| *** baoli has quit IRC | 22:06 | |
| *** vhosakot has quit IRC | 22:06 | |
| mordred | since we build new images every night anyway | 22:06 |
| pabelanger | clarkb: danke | 22:06 |
| mordred | it would be _fascinating_ to see how it did | 22:06 |
| *** salv-orl_ has joined #openstack-infra | 22:07 | |
| dirk | mordred: hardlink cirros to /bin/false will be similar ;-) | 22:07 |
| *** mdrabe has quit IRC | 22:07 | |
| dirk | Well, anyway, Happy to take a.look at that once we have devstack for opensuse-42.2. that's gonna take some weeks | 22:08 |
| *** felipemonteiro_ has quit IRC | 22:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove puppet / bindep-cache from opensuse DIB https://review.openstack.org/467770 | 22:09 |
| *** esikachev has joined #openstack-infra | 22:09 | |
| dirk | pabelanger: can you add the bindep job with the opensue image? | 22:09 |
| pabelanger | https://review.openstack.org/#/c/467788/ | 22:10 |
| *** jamesdenton has quit IRC | 22:11 | |
| *** jbernard has quit IRC | 22:11 | |
| pabelanger | once we boot an image, I'll remove WIP | 22:11 |
| pabelanger | okay, afk for a bit | 22:11 |
| *** salv-orlando has quit IRC | 22:11 | |
| pabelanger | will check back on builds | 22:11 |
| dirk | pabelanger: thanks for all the help so far | 22:12 |
| *** hashar has quit IRC | 22:13 | |
| *** esikachev has quit IRC | 22:13 | |
| clarkb | dirk: definitely I don't think its a rush to do something like that. And likely significantly more value in using the stable leap releases | 22:15 |
| *** rwsu has quit IRC | 22:15 | |
| *** jbernard has joined #openstack-infra | 22:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Rename to opensuse-422 for DIB logging https://review.openstack.org/467786 | 22:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Bring opensuse-422 DIB online https://review.openstack.org/467782 | 22:18 |
| dirk | clarkb: yeah, current goal is voting devstack, but it's good that I understand the whole setup steps so that we can do it again for the next distro. 42.3 is just around the corner and I would want to have that as well | 22:18 |
| *** rhallisey has quit IRC | 22:18 | |
| *** salv-orl_ has quit IRC | 22:19 | |
| *** adriant has joined #openstack-infra | 22:20 | |
| *** esikachev has joined #openstack-infra | 22:22 | |
| *** thorst has quit IRC | 22:23 | |
| *** Swami has quit IRC | 22:23 | |
| *** marst_ has quit IRC | 22:23 | |
| *** marst_ has joined #openstack-infra | 22:24 | |
| *** annegentle has quit IRC | 22:24 | |
| *** jkilpatr has joined #openstack-infra | 22:26 | |
| *** esikachev has quit IRC | 22:26 | |
| *** ijw has quit IRC | 22:28 | |
| *** marst_ has quit IRC | 22:29 | |
| clarkb | so thats interesting adduser gave bkero uid 1000 | 22:29 |
| clarkb | oh well for this server it shouldn't matter | 22:30 |
| bkero | O_o | 22:30 |
| bkero | That sounds broken, I should be 2027 | 22:30 |
| clarkb | bkero: bkero@166.78.27.133 | 22:30 |
| bkero | clarkb: does it have a floating iP? | 22:30 |
| bkero | w00t | 22:30 |
| clarkb | bkero: well I didn't create you with puppet | 22:30 |
| clarkb | I just ran adduser | 22:30 |
| bkero | ah | 22:30 |
| clarkb | which I expected to pick the next uid in the user range by default but I guess not? | 22:30 |
| clarkb | its fine for this | 22:31 |
| clarkb | I am going to reenable exim4 and mailman on the actual lists server as it appears that the snapshot went well and is now booted | 22:31 |
| *** gouthamr has quit IRC | 22:32 | |
| clarkb | and done so I think we are good now | 22:32 |
| *** squid has joined #openstack-infra | 22:32 | |
| bkero | There's really not a lot running on this host -- sshd, ntpd, unbound, snmpd, nova-agent, apache, xe-daemon | 22:32 |
| *** squid is now known as calebb | 22:32 | |
| clarkb | ya normally it has exim4 and mailman running too ,but disabled on that snapshot | 22:33 |
| bkero | okay | 22:33 |
| mtreinish | clarkb: what about arch? | 22:33 |
| *** bobh has joined #openstack-infra | 22:33 | |
| clarkb | mtreinish: one of the many options for testing the future universe and seeing all the fires | 22:34 |
| mtreinish | heh, yeah I can only imagine how many fires there would be. BUt I would help out with that if it something someone else wanted to drive | 22:34 |
| *** sbezverk_ has quit IRC | 22:35 | |
| clarkb | bkero: also I booted the test server on the same flavor in the same region as the production server so everything should be as like for like as possible | 22:36 |
| bkero | That's good. We shouldn't OOM if we shouldn't otherwise. :) | 22:36 |
| *** esberglu has joined #openstack-infra | 22:36 | |
| *** adam_g has joined #openstack-infra | 22:39 | |
| *** esberglu has quit IRC | 22:40 | |
| * bkero should go back to the etherpad to figure out what steps need to be performed next. | 22:43 | |
| bkero | clarkb: Did this coincide with a mailman 3 upgrade? | 22:43 |
| clarkb | bkero: this being the trusty to xenial upgrade? I didn't think it would | 22:44 |
| clarkb | ya xenail mailman is 2.1.20 | 22:44 |
| bkero | okay cool | 22:44 |
| bkero | Also mailman3 doesn't have topics | 22:45 |
| bkero | They decided to implement and use dlists instead AIUI | 22:45 |
| clarkb | https://packages.ubuntu.com/search?suite=xenial&searchon=names&keywords=mailman | 22:45 |
| clarkb | so its largely just doing an in place OS upgrade and making sure we survive that | 22:45 |
| clarkb | bkero: https://etherpad.openstack.org/p/lists.o.o-trusty-upgrade is from when we did precise to trusty | 22:45 |
| clarkb | and is likely a good template | 22:45 |
| bkero | okay | 22:46 |
| * bkero makes a similar plan before doing anything | 22:46 | |
| clarkb | the filesystem upgrade from ext3 to ext4 is no longer something we need to worry about so this upgrade shopuld be much simpler | 22:47 |
| bkero | Plenty of inodes, plenty of space | 22:48 |
| bkero | I guess there'd probably be a perf improvement with the extents, although that might not be necessary. | 22:48 |
| *** krtaylor has joined #openstack-infra | 22:49 | |
| clarkb | bkero: we already did that last time around | 22:50 |
| clarkb | we should be proper ext4, it was a fun update that required remounting / ro and updating things then hoping it all worked :) | 22:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add opensuse-422 non-voting job for bindep https://review.openstack.org/467788 | 22:50 |
| *** ijw has joined #openstack-infra | 22:51 | |
| clarkb | mordred: can I get a second review on https://review.openstack.org/#/c/467773/2 its an ansible related change in d-g to setup ssh keys more properly | 22:52 |
| *** gordc has quit IRC | 22:52 | |
| clarkb | mordred: zuul hasn't reported back but we passed the base multinode job so it should be happy | 22:52 |
| *** hamzy has quit IRC | 22:53 | |
| *** ijw has quit IRC | 22:56 | |
| *** pradk has quit IRC | 22:57 | |
| mordred | clarkb: looks great | 22:59 |
| pabelanger | looks like we are just compressing opensuse DIB now | 22:59 |
| *** jamesmca_ has joined #openstack-infra | 22:59 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:00 | |
| *** Julien-zte has quit IRC | 23:03 | |
| *** Julien-zte has joined #openstack-infra | 23:03 | |
| *** gouthamr has joined #openstack-infra | 23:04 | |
| *** jamesmca_ has quit IRC | 23:04 | |
| *** esikachev has joined #openstack-infra | 23:04 | |
| *** ijw has joined #openstack-infra | 23:05 | |
| *** aeng has quit IRC | 23:05 | |
| clarkb | mordred: I was very cofnused for a bit when I couldn ssh to multinode test nodes to check private IPs | 23:05 |
| *** kzaitsev_mb has quit IRC | 23:05 | |
| *** ijw has quit IRC | 23:06 | |
| *** gildub has joined #openstack-infra | 23:06 | |
| mordred | clarkb: yah. I'll bet you were! | 23:07 |
| mordred | clarkb, pabelanger: btw - I STILL do not know what's going on with osic uploads | 23:08 |
| pabelanger | starting upload of opensuse now | 23:08 |
| *** ijw has joined #openstack-infra | 23:08 | |
| mordred | clarkb, pabelanger: I ran a thing in http trace mode and I'm not wiser than I was before - same outcome as pabelanger's run | 23:09 |
| *** esikachev has quit IRC | 23:09 | |
| *** ijw has joined #openstack-infra | 23:09 | |
| pabelanger | mordred: I think we'll need help from cloudnull tomorrow, see what is happening on openstack side | 23:09 |
| *** rbrndt has quit IRC | 23:09 | |
| pabelanger | it seem to start about the time we switched domains for OSIC | 23:09 |
| mordred | we get a connection timeout - so we retry - the retry gets 409 then we do a delete | 23:11 |
| mordred | NOW - we do seem to do two deletes, which I cannot explain and will look in to | 23:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/467428 | 23:11 |
| *** bobh has quit IRC | 23:11 | |
| mordred | or it mgith just be logging it twice | 23:11 |
| mordred | there is a some text/html content from the 409 - if I try again, i'll add code to print that | 23:12 |
| pabelanger | 2017-05-24 23:12:59,248 INFO nodepool.builder.UploadWorker.6: Image build opensuse-422-0000000001 in infracloud-chocolate is ready | 23:13 |
| pabelanger | :) | 23:13 |
| mordred | cloudnull: if you happen to see this: http://paste.openstack.org/show/610606/ is the http log from the client side | 23:14 |
| *** sbezverk has joined #openstack-infra | 23:16 | |
| pabelanger | Yay | 23:17 |
| pabelanger | | 8977043 | infracloud-chocolate | None | opensuse-422 | zuul | zl06 | opensuse-422-infracloud-chocolate-8977043 | opensuse-422-infracloud-chocolate-8977043 | 71fa4e6c-440c-40ed-a530-9ee3dccc0119 | 15.184.69.108 | ready | 00:00:00:10 | None | | 23:17 |
| *** sdague has quit IRC | 23:17 | |
| *** Julien-zte has quit IRC | 23:17 | |
| *** Julien-zte has joined #openstack-infra | 23:18 | |
| pabelanger | mordred: mind a +3 so we can bring running a job for opensuse-422? https://review.openstack.org/#/c/467788/ | 23:19 |
| pabelanger | #status success opensuse-422-infracloud-chocolate-8977043 launched by nodepool | 23:20 |
| openstackstatus | pabelanger: unknown command | 23:20 |
| pabelanger | #success opensuse-422-infracloud-chocolate-8977043 launched by nodepool | 23:20 |
| openstackstatus | pabelanger: Added success to Success page | 23:20 |
| *** aeng has joined #openstack-infra | 23:22 | |
| mordred | pabelanger: done | 23:23 |
| pabelanger | mordred: thanks | 23:23 |
| *** yuanying has quit IRC | 23:25 | |
| dims | clarkb : fungi : one of you please bless this d-g change? i'll be around all evening to support if anything breaks | 23:27 |
| dims | clarkb : fungi : has a +2 from monty - https://review.openstack.org/#/c/467687/ | 23:27 |
| * fungi checks that his vial of holy water is topped up | 23:27 | |
| *** aeng has quit IRC | 23:27 | |
| dims | :) | 23:27 |
| pabelanger | btw, I love that we are about to roll over 10 million VMs launched by nodepool | 23:28 |
| pabelanger | we likely are there, if we account subnodes | 23:28 |
| fungi | i can't even begin to guess what's going to blow up if we ask d-g to start etc3d everywhere, but yolo! | 23:29 |
| fungi | it passes the self-testing at least | 23:29 |
| openstackgerrit | Merged openstack-infra/project-config master: Add opensuse-422 non-voting job for bindep https://review.openstack.org/467788 | 23:30 |
| fungi | dims: mitaka jobs failed on that... any idea if we even care at this point? it's supposed to be eol we just lost our stable team, right? | 23:30 |
| *** jaypipes has quit IRC | 23:31 | |
| * dims braces for impact :) | 23:33 | |
| fungi | all hands to battle stations | 23:34 |
| clarkb | fungi: right that | 23:34 |
| clarkb | oh we should success adding citycloud too | 23:34 |
| clarkb | #success infra added citycloud to the pool of test nodes | 23:34 |
| openstackstatus | clarkb: Added success to Success page | 23:35 |
| *** Apoorva_ has joined #openstack-infra | 23:36 | |
| *** Julien-zte has quit IRC | 23:36 | |
| *** Julien-zte has joined #openstack-infra | 23:38 | |
| *** aeng has joined #openstack-infra | 23:39 | |
| *** Apoorva has quit IRC | 23:39 | |
| *** Apoorva_ has quit IRC | 23:40 | |
| *** Julien-zte has quit IRC | 23:41 | |
| *** salv-orlando has joined #openstack-infra | 23:41 | |
| *** Julien-zte has joined #openstack-infra | 23:41 | |
| *** mriedem1 has joined #openstack-infra | 23:44 | |
| *** mriedem1 has quit IRC | 23:45 | |
| *** mriedem has quit IRC | 23:45 | |
| *** mriedem has joined #openstack-infra | 23:45 | |
| *** sbezverk has quit IRC | 23:46 | |
| *** Julien-zte has quit IRC | 23:47 | |
| *** yamahata has joined #openstack-infra | 23:51 | |
| *** salv-orlando has quit IRC | 23:54 | |
| *** ijw has quit IRC | 23:56 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: First Pass: De-client-ify list_users https://review.openstack.org/467784 | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!