| clarkb | launch node is running, doing the yum update now (which appears to have found the right kernel but I'll double check pti is enabled before updating dns records | 00:04 |
|---|---|---|
| *** esberglu has quit IRC | 00:04 | |
| *** xarses_ has quit IRC | 00:11 | |
| *** jistr has quit IRC | 00:11 | |
| *** caphrim007_ has joined #openstack-infra | 00:12 | |
| clarkb | ok think I've noticed is that our install_puppet that install the rdo stuff that we want to remove from the dib builds is also there for when we install puppet on real centos servers | 00:13 |
| *** caphrim007_ has quit IRC | 00:13 | |
| clarkb | I'm just going to roll with that for now, BUT we should consider rebuilding these servers again in the future if we can excise that | 00:14 |
| *** jaypipes has quit IRC | 00:14 | |
| *** caphrim007 has quit IRC | 00:15 | |
| *** zhurong has joined #openstack-infra | 00:16 | |
| clarkb | (it was present on the old servers too) | 00:16 |
| *** jistr has joined #openstack-infra | 00:16 | |
| mordred | clarkb: ++ | 00:18 |
| clarkb | pti is enabled and test clones work | 00:19 |
| clarkb | I will update dns now please keep eyeballs open for any weirdness | 00:19 |
| *** xarses_ has joined #openstack-infra | 00:20 | |
| *** slaweq has joined #openstack-infra | 00:20 | |
| *** sdague has quit IRC | 00:22 | |
| clarkb | ok DNS is updated 2001:4800:7819:103:be76:4eff:fe04:77e6 and 104.130.246.85 is the new server | 00:22 |
| clarkb | I'll wait about an hour then check connections to old server before deleting old server | 00:23 |
| clarkb | then will bump the dns ttl back up to an hour | 00:23 |
| *** esberglu has joined #openstack-infra | 00:28 | |
| *** esberglu has quit IRC | 00:29 | |
| *** EmilienM has quit IRC | 00:30 | |
| *** markmcclain has quit IRC | 00:30 | |
| *** markmcclain has joined #openstack-infra | 00:32 | |
| clarkb | mwhahaha: hey if our centos7 image builds haven't updated yet with pti enabled they should be updating soonish. Will be curious to know if you notice significantly longer test runs as a result. (you are biggest consumer of centos7 images I think) | 00:33 |
| mwhahaha | clarkb: ok. We'll keep an eye out | 00:34 |
| *** xarses_ has quit IRC | 00:36 | |
| *** edmondsw has joined #openstack-infra | 00:36 | |
| *** edmondsw has quit IRC | 00:41 | |
| dmsimard | clarkb: TTL is largely a suggestion, you might break people by killing off the old machine right away | 00:46 |
| *** dave-mccowan has joined #openstack-infra | 00:48 | |
| fungi | dmsimard: easy enough to figure out by watching logs or tcpdump though | 00:48 |
| clarkb | dmsimard: yaI know I am tailing logs too | 00:48 |
| clarkb | there has definitely been a transition from one to the other | 00:48 |
| dmsimard | Indeed | 00:48 |
| clarkb | in the past we have had to just break some people though because things like jvm resolve a record once and never reresolve it | 00:49 |
| *** gyee has quit IRC | 00:49 | |
| clarkb | there is about another half hour on the actual ttl at this point | 00:50 |
| dmsimard | Sure | 00:50 |
| *** slaweq has quit IRC | 00:53 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard master: Use correct tox command for upgrading the database https://review.openstack.org/531272 | 00:54 |
| *** bobh has joined #openstack-infra | 00:55 | |
| *** slaweq has joined #openstack-infra | 00:56 | |
| *** cuongnv has joined #openstack-infra | 00:58 | |
| *** slaweq has quit IRC | 01:00 | |
| *** markmcclain has quit IRC | 01:05 | |
| *** markmcclain has joined #openstack-infra | 01:13 | |
| *** hoangcx has joined #openstack-infra | 01:15 | |
| *** nicolasbock has joined #openstack-infra | 01:17 | |
| *** kiennt26 has joined #openstack-infra | 01:18 | |
| hoangcx | Hi Team, I have a small question regarding to this patch: https://review.openstack.org/#/c/522695/. It has just been merged recently. Then I tried to open the docs link: https://docs.openstack.org/neutron-vpnaas/latest/ with unlucky | 01:20 |
| hoangcx | Is the the link the access is correct? Or do i need to do more work to have it? Thanks a lot | 01:21 |
| clarkb | hoangcx: have you merged any docs changes to neutron-vpnaas? those builds run in response to docs changes merging | 01:22 |
| *** caphrim007 has joined #openstack-infra | 01:26 | |
| clarkb | dmsimard: there is a redhat ip still hitting the old git.o.o host. Is your concern thta you know red hat breaks dns? :) | 01:27 |
| dmsimard | (╯°□°）╯︵ ┻━┻ | 01:28 |
| dmsimard | My plan is foiled | 01:28 |
| clarkb | there are a couple other IPs that I can't identify as well | 01:28 |
| clarkb | and some web crawlers | 01:29 |
| *** caphrim007 has quit IRC | 01:31 | |
| *** slaweq has joined #openstack-infra | 01:33 | |
| *** Apoorva_ has joined #openstack-infra | 01:33 | |
| clarkb | at this rate I may just leave the old server up until morning. Production is shifted over to new server but there are a few stragglers and I am running out of daylight. I also expect tomorrow to be a long day assuming we get kernels so don't want to work late tonight | 01:36 |
| *** Apoorva has quit IRC | 01:37 | |
| clarkb | #status log git0*.openstack.org patched and kernels running with PTI enabled. git.openstack.org has been replaced with a new server running with PTI enabled. The old server is still in places for straggler clients. Will need to be deleted and DNS record TTLs set back to one hour | 01:37 |
| openstackstatus | clarkb: finished logging | 01:37 |
| *** slaweq has quit IRC | 01:37 | |
| *** Apoorva_ has quit IRC | 01:38 | |
| *** hemna_ has quit IRC | 01:39 | |
| fungi | thanks clarkb! | 01:45 |
| *** jroll has quit IRC | 01:46 | |
| *** kgiusti has left #openstack-infra | 01:49 | |
| *** gagehugo has joined #openstack-infra | 01:50 | |
| *** jroll has joined #openstack-infra | 01:51 | |
| *** hoangcx has quit IRC | 01:55 | |
| *** hoangcx has joined #openstack-infra | 01:55 | |
| hoangcx | clarkb: Ah I see. I will try to merged one doc patch to see the reflection. Thanks for your info/guidance. | 01:56 |
| *** cuongnv has quit IRC | 01:57 | |
| *** cuongnv has joined #openstack-infra | 01:57 | |
| *** caphrim007 has joined #openstack-infra | 01:58 | |
| *** liujiong has joined #openstack-infra | 02:02 | |
| *** caphrim007 has quit IRC | 02:03 | |
| *** slaweq has joined #openstack-infra | 02:05 | |
| hoangcx | clarkb: BTW, Could you please help to check and land this one? https://review.openstack.org/#/c/527872/ Thank you | 02:12 |
| *** threestrands_ has joined #openstack-infra | 02:14 | |
| *** eharney has quit IRC | 02:16 | |
| *** threestrands has quit IRC | 02:16 | |
| *** zzzeek has quit IRC | 02:19 | |
| *** zzzeek has joined #openstack-infra | 02:20 | |
| *** namnh has joined #openstack-infra | 02:20 | |
| *** zzzeek has quit IRC | 02:20 | |
| *** zzzeek has joined #openstack-infra | 02:21 | |
| *** smatzek has joined #openstack-infra | 02:24 | |
| *** edmondsw has joined #openstack-infra | 02:24 | |
| *** edmondsw has quit IRC | 02:28 | |
| *** markmcclain has quit IRC | 02:29 | |
| *** eharney has joined #openstack-infra | 02:30 | |
| *** mriedem has quit IRC | 02:31 | |
| *** markmcclain has joined #openstack-infra | 02:34 | |
| *** harlowja has quit IRC | 02:36 | |
| *** smatzek has quit IRC | 02:38 | |
| *** slaweq has quit IRC | 02:39 | |
| *** markmcclain has quit IRC | 02:43 | |
| *** markmcclain has joined #openstack-infra | 02:45 | |
| *** slaweq has joined #openstack-infra | 02:46 | |
| *** spzala has quit IRC | 02:47 | |
| *** EmilienM has joined #openstack-infra | 02:51 | |
| *** slaweq has quit IRC | 02:51 | |
| *** xinliang has quit IRC | 02:52 | |
| *** markmcclain has quit IRC | 02:53 | |
| *** markmcclain has joined #openstack-infra | 02:55 | |
| *** caphrim007 has joined #openstack-infra | 02:59 | |
| *** caphrim007 has quit IRC | 02:59 | |
| *** ijw has quit IRC | 03:00 | |
| *** caphrim007 has joined #openstack-infra | 03:01 | |
| *** xinliang has joined #openstack-infra | 03:04 | |
| *** xinliang has quit IRC | 03:04 | |
| *** xinliang has joined #openstack-infra | 03:04 | |
| *** swest has quit IRC | 03:05 | |
| *** swest has joined #openstack-infra | 03:05 | |
| *** spzala has joined #openstack-infra | 03:06 | |
| *** spzala has quit IRC | 03:12 | |
| *** ramishra has joined #openstack-infra | 03:14 | |
| *** nicolasbock has quit IRC | 03:14 | |
| *** slaweq has joined #openstack-infra | 03:21 | |
| *** greghaynes has quit IRC | 03:23 | |
| *** slaweq has quit IRC | 03:25 | |
| *** greghaynes has joined #openstack-infra | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:28 | |
| *** links has joined #openstack-infra | 03:29 | |
| *** Apoorva has joined #openstack-infra | 03:30 | |
| *** links has quit IRC | 03:34 | |
| *** zhurong has quit IRC | 03:34 | |
| *** eharney has quit IRC | 03:38 | |
| *** eharney has joined #openstack-infra | 03:39 | |
| *** s-shiono has joined #openstack-infra | 03:42 | |
| *** markmcclain has quit IRC | 03:45 | |
| dmsimard | Just stumbled on this: "Add a welcome bot to Gerrit for first time contributors" https://phabricator.wikimedia.org/T73357 | 03:45 |
| dmsimard | Sounds like some people do appreciate our attempts at being welcoming to new contributors :D | 03:46 |
| *** markmcclain has joined #openstack-infra | 03:47 | |
| *** liujiong has quit IRC | 03:49 | |
| *** liujiong has joined #openstack-infra | 03:49 | |
| *** spzala has joined #openstack-infra | 03:51 | |
| *** spzala has quit IRC | 03:55 | |
| *** annp has joined #openstack-infra | 03:56 | |
| *** links has joined #openstack-infra | 03:56 | |
| *** bobh has quit IRC | 03:57 | |
| *** slaweq has joined #openstack-infra | 04:00 | |
| *** s-shiono_ has joined #openstack-infra | 04:06 | |
| *** udesale has joined #openstack-infra | 04:08 | |
| *** s-shiono has quit IRC | 04:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules https://review.openstack.org/522945 | 04:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul https://review.openstack.org/509715 | 04:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager https://review.openstack.org/524773 | 04:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers https://review.openstack.org/524620 | 04:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image https://review.openstack.org/516920 | 04:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 04:13 |
| *** edmondsw has joined #openstack-infra | 04:13 | |
| *** edmondsw has quit IRC | 04:17 | |
| *** sree has joined #openstack-infra | 04:27 | |
| *** psachin has joined #openstack-infra | 04:33 | |
| *** eharney has quit IRC | 04:34 | |
| *** slaweq has quit IRC | 04:34 | |
| *** slaweq has joined #openstack-infra | 04:36 | |
| *** slaweq has quit IRC | 04:41 | |
| *** ijw has joined #openstack-infra | 04:41 | |
| *** ijw has quit IRC | 04:42 | |
| *** ijw has joined #openstack-infra | 04:42 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules https://review.openstack.org/522945 | 04:50 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul https://review.openstack.org/509715 | 04:50 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Disambiguate with Netflix and Javascript zuul https://review.openstack.org/531292 | 04:58 |
| *** sree_ has joined #openstack-infra | 04:59 | |
| *** sree_ is now known as Guest45790 | 04:59 | |
| *** sree has quit IRC | 05:02 | |
| *** dave-mccowan has quit IRC | 05:06 | |
| *** hoangcx has left #openstack-infra | 05:06 | |
| *** slaweq has joined #openstack-infra | 05:08 | |
| *** dmsimard has quit IRC | 05:09 | |
| *** markmcclain has quit IRC | 05:12 | |
| *** slaweq has quit IRC | 05:13 | |
| *** markmcclain has joined #openstack-infra | 05:15 | |
| *** iyamahat has joined #openstack-infra | 05:20 | |
| *** janki has joined #openstack-infra | 05:24 | |
| *** dmsimard has joined #openstack-infra | 05:26 | |
| *** dmsimard has quit IRC | 05:26 | |
| *** dmsimard has joined #openstack-infra | 05:26 | |
| *** dmsimard has quit IRC | 05:32 | |
| *** dmsimard has joined #openstack-infra | 05:34 | |
| *** dmsimard has quit IRC | 05:34 | |
| *** dmsimard has joined #openstack-infra | 05:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test job https://review.openstack.org/512653 | 05:36 |
| *** hongbin has joined #openstack-infra | 05:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs https://review.openstack.org/512653 | 05:44 |
| AJaeger | stephenfin, mordred, here's the updated tox-with-sudo jobs including a functional one ^ | 05:44 |
| *** iyamahat has quit IRC | 05:45 | |
| AJaeger | stephenfin: I'll test it with openvif | 05:45 |
| AJaeger | I mean: os-vif | 05:45 |
| *** slaweq has joined #openstack-infra | 05:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs https://review.openstack.org/512653 | 05:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs https://review.openstack.org/512653 | 05:49 |
| *** ianychoi has quit IRC | 05:52 | |
| *** caphrim007 has quit IRC | 05:53 | |
| *** Apoorva has quit IRC | 05:55 | |
| *** caphrim007 has joined #openstack-infra | 05:56 | |
| *** iyamahat has joined #openstack-infra | 05:59 | |
| *** aeng has quit IRC | 05:59 | |
| *** jtomasek has joined #openstack-infra | 06:11 | |
| *** threestrands_ has quit IRC | 06:13 | |
| *** slaweq has quit IRC | 06:19 | |
| openstackgerrit | megan guiney proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 06:19 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/openstack-zuul-jobs master: Restore branch conditional on tempest-full https://review.openstack.org/531058 | 06:27 |
| *** slaweq has joined #openstack-infra | 06:28 | |
| *** bhavik1 has joined #openstack-infra | 06:29 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Added openstack-tempest-plugins job template https://review.openstack.org/531307 | 06:31 |
| chandankumar | AJaeger: ^^ | 06:31 |
| chandankumar | AJaeger: i am not sure how to use template jobs with in a job template need help on that | 06:32 |
| *** armaan_ has joined #openstack-infra | 06:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs https://review.openstack.org/512653 | 06:56 |
| AJaeger | chandankumar: commented - you cannot. Just remove them | 06:58 |
| *** jtomasek has quit IRC | 07:04 | |
| *** pgadiya has joined #openstack-infra | 07:06 | |
| *** slaweq has quit IRC | 07:07 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Added openstack-tempest-plugins job template https://review.openstack.org/531307 | 07:11 |
| chandankumar | AJaeger: we need one more change in project config to update the jobs for all tempest plugins na? | 07:12 |
| *** claudiub has joined #openstack-infra | 07:14 | |
| *** armax has quit IRC | 07:29 | |
| *** iyamahat has quit IRC | 07:35 | |
| *** slaweq has joined #openstack-infra | 07:35 | |
| chandankumar | AJaeger: https://review.openstack.org/#/c/530024/ please have a look gate jobs are not working | 07:38 |
| *** slaweq has quit IRC | 07:40 | |
| *** slaweq has joined #openstack-infra | 07:41 | |
| chandankumar | AJaeger: regarding this review https://review.openstack.org/#/c/530023 you have given this example https://review.openstack.org/#/c/530099/ so we donot need to move any playbooks here? | 07:45 |
| *** zhurong has joined #openstack-infra | 07:48 | |
| *** ijw has quit IRC | 07:48 | |
| *** edmondsw has joined #openstack-infra | 07:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager https://review.openstack.org/524773 | 07:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers https://review.openstack.org/524620 | 07:52 |
| *** edmondsw has quit IRC | 07:53 | |
| *** shardy has joined #openstack-infra | 07:57 | |
| *** shardy has quit IRC | 08:01 | |
| *** shardy has joined #openstack-infra | 08:03 | |
| *** spzala has joined #openstack-infra | 08:07 | |
| *** markvoelker has joined #openstack-infra | 08:09 | |
| *** migi has quit IRC | 08:11 | |
| *** spzala has quit IRC | 08:11 | |
| *** eumel8 has joined #openstack-infra | 08:12 | |
| *** tesseract has joined #openstack-infra | 08:12 | |
| *** gouthamr has quit IRC | 08:13 | |
| AJaeger | chandankumar: yes, you need to update project-config as well | 08:14 |
| AJaeger | chandankumar: what do you mean: Gates not working on 530024? | 08:15 |
| *** florianf has joined #openstack-infra | 08:15 | |
| *** armaan_ has quit IRC | 08:15 | |
| *** armaan has joined #openstack-infra | 08:16 | |
| AJaeger | chandankumar: regarding 530023: you don't need to move the playbooks for the two *tox* jobs. just follow that example and use native tox jobs. | 08:16 |
| *** zhurong has quit IRC | 08:16 | |
| *** armaan has quit IRC | 08:16 | |
| AJaeger | chandankumar: checked 530024 - all looks good to me. | 08:18 |
| *** aviau has quit IRC | 08:19 | |
| *** aviau has joined #openstack-infra | 08:20 | |
| *** aviau has joined #openstack-infra | 08:20 | |
| AJaeger | stephenfin: so, testing with 530961 looks good from my side for the tox-with-sudo job. Problem is that the helper is not found but sudo works - so, might need some more work from your side on that one... If you think the job is wrong, please tell us. | 08:24 |
| *** zhurong has joined #openstack-infra | 08:27 | |
| *** rcernin has quit IRC | 08:28 | |
| *** armaan has joined #openstack-infra | 08:31 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs project template for all tempest plugins https://review.openstack.org/531339 | 08:42 |
| *** markvoelker has quit IRC | 08:44 | |
| *** ijw has joined #openstack-infra | 08:49 | |
| *** migi has joined #openstack-infra | 08:51 | |
| *** armax has joined #openstack-infra | 08:55 | |
| *** ijw has quit IRC | 08:55 | |
| *** s-shiono_ has quit IRC | 08:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers https://review.openstack.org/524620 | 09:00 |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** liujiong has quit IRC | 09:03 | |
| *** stakeda has quit IRC | 09:09 | |
| *** armax has quit IRC | 09:14 | |
| *** johnthetubaguy has quit IRC | 09:17 | |
| *** johnthetubaguy has joined #openstack-infra | 09:18 | |
| stephenfin | AJaeger: Quite possibly. Going to look into it today and report back what I find | 09:18 |
| stephenfin | AJaeger: And thank you for following up on that. I appreciate it :) | 09:19 |
| AJaeger | stephenfin: you're welcome. | 09:21 |
| AJaeger | config-core, please review https://review.openstack.org/#/c/512653/ to add tox-with-sudo jobs and https://review.openstack.org/#/c/530104/ for bindep v3 native testing | 09:22 |
| *** derekh has joined #openstack-infra | 09:23 | |
| AJaeger | stephenfin: you might also need a full devstack setup for the functional tests. Might be that testing with tox is not possible. so, check first whether you get it working on your system. | 09:23 |
| *** lucas-afk is now known as lucasagomes | 09:33 | |
| *** udesale__ has joined #openstack-infra | 09:33 | |
| *** mnencia has quit IRC | 09:34 | |
| *** udesale has quit IRC | 09:35 | |
| *** mnencia has joined #openstack-infra | 09:36 | |
| *** edmondsw has joined #openstack-infra | 09:37 | |
| *** mnencia has quit IRC | 09:40 | |
| *** markvoelker has joined #openstack-infra | 09:41 | |
| *** edmondsw has quit IRC | 09:42 | |
| *** mnencia has joined #openstack-infra | 09:43 | |
| *** jascott1 has quit IRC | 09:43 | |
| *** jascott1 has joined #openstack-infra | 09:43 | |
| *** bhavik1 has quit IRC | 09:44 | |
| *** jascott1 has quit IRC | 09:48 | |
| *** claudiub|2 has joined #openstack-infra | 09:51 | |
| *** ijw has joined #openstack-infra | 09:51 | |
| *** hongbin has quit IRC | 09:52 | |
| *** claudiub has quit IRC | 09:54 | |
| *** tosky has joined #openstack-infra | 09:54 | |
| *** ijw has quit IRC | 09:56 | |
| *** jtomasek has joined #openstack-infra | 09:58 | |
| aspiers | I'd like to register a storyboard task to oslo-specs, but storyboard seems to be missing a whole bunch of projects including that one. Any ideas? | 10:01 |
| aspiers | Do I have to wait for the Oslo team to migrate to storyboard? | 10:02 |
| *** Guest45790 has quit IRC | 10:06 | |
| *** sree has joined #openstack-infra | 10:07 | |
| *** sree has quit IRC | 10:09 | |
| *** sree has joined #openstack-infra | 10:09 | |
| *** pbourke has joined #openstack-infra | 10:11 | |
| *** e0ne has joined #openstack-infra | 10:12 | |
| *** cuongnv has quit IRC | 10:13 | |
| *** markvoelker has quit IRC | 10:15 | |
| frickler | aspiers: seems oslo issues are still being tracked at https://bugs.launchpad.net/oslo | 10:21 |
| *** kiennt26 has quit IRC | 10:22 | |
| aspiers | frickler: yeah, I figured | 10:23 |
| frickler | aspiers: is there a specific reason you want to use storyboard? most projects are still on launchpad | 10:25 |
| aspiers | frickler: yes there is: http://lists.openstack.org/pipermail/openstack-sigs/2017-November/000176.html | 10:27 |
| *** udesale has joined #openstack-infra | 10:28 | |
| aspiers | frickler: but also because it seems to be the Right Thing to Do; IIUC, long-term the community is planning a wholesale migration from launchpad to SB, hence presentations like https://www.youtube.com/watch?v=n_PaKuN4Skk | 10:29 |
| *** gcb has quit IRC | 10:29 | |
| *** udesale__ has quit IRC | 10:29 | |
| *** namnh_ has joined #openstack-infra | 10:33 | |
| *** namnh_ has quit IRC | 10:34 | |
| *** shardy has quit IRC | 10:35 | |
| eumel8 | Storyboards are almost pushed by the Foundation. You can contact Kendall for any questions about that. | 10:35 |
| aspiers | eumel8: thanks for the info | 10:35 |
| *** sree has quit IRC | 10:35 | |
| *** sree has joined #openstack-infra | 10:36 | |
| *** namnh has quit IRC | 10:37 | |
| *** sambetts|afk is now known as sambetts | 10:37 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] Refactor status functions, add more status endpoints https://review.openstack.org/531354 | 10:39 |
| chandankumar | AJaeger: https://review.openstack.org/531339 i have added the template here | 10:39 |
| aspiers | hrm diablo_rojo is not in this channel | 10:39 |
| aspiers | I guess I'll ask her in #storyboard | 10:40 |
| frickler | aspiers: eumel8: well, IMHO storyboard isn't ready for general use yet, ymmv | 10:41 |
| aspiers | frickler: sure | 10:42 |
| *** dtantsur|afk is now known as dtantsur | 10:42 | |
| *** shardy has joined #openstack-infra | 10:42 | |
| *** sree has quit IRC | 10:42 | |
| *** sdague has joined #openstack-infra | 10:43 | |
| AJaeger | chandankumar: seen it - but we first need to figure out some problems with pypi, I'm not approving any changes with new publish-to-pypi templates. | 10:43 |
| *** sree has joined #openstack-infra | 10:43 | |
| chandankumar | AJaeger: ack! | 10:43 |
| *** sree has quit IRC | 10:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard master: Create a StoryBoard gui manual https://review.openstack.org/325474 | 10:50 |
| AJaeger | aspiers: ^ | 10:51 |
| *** sree has joined #openstack-infra | 10:52 | |
| *** ijw has joined #openstack-infra | 10:52 | |
| *** szaher has joined #openstack-infra | 10:54 | |
| frickler | aspiers: ah, found your story about storyboard ;) added to the list of issues I consider blockers, but I'm about to discussing that, too https://storyboard.openstack.org/#!/search?tags=blocking-storyboard-migration | 10:54 |
| *** electrofelix has joined #openstack-infra | 10:55 | |
| *** zhurong has quit IRC | 10:57 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] Refactor status functions, add more status endpoints https://review.openstack.org/531354 | 11:01 |
| *** pcaruana has joined #openstack-infra | 11:01 | |
| *** udesale__ has joined #openstack-infra | 11:02 | |
| *** udesale has quit IRC | 11:05 | |
| AJaeger | frickler, could you review https://review.openstack.org/#/c/512653/ to add tox-with-sudo jobs and https://review.openstack.org/#/c/530104/ for bindep v3 native testing , please? | 11:06 |
| *** Qiming has quit IRC | 11:10 | |
| *** sree has quit IRC | 11:10 | |
| *** markvoelker has joined #openstack-infra | 11:11 | |
| *** Qiming has joined #openstack-infra | 11:13 | |
| *** sree has joined #openstack-infra | 11:14 | |
| *** sree has quit IRC | 11:18 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Run pep8 jobs for cinder tempest plugin https://review.openstack.org/531361 | 11:19 |
| *** sree has joined #openstack-infra | 11:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard master: Create a StoryBoard gui manual https://review.openstack.org/325474 | 11:25 |
| AJaeger | frickler: check https://review.openstack.org/#/c/529662/ for why 512653 should work... | 11:26 |
| *** udesale has joined #openstack-infra | 11:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add native Zuul v3 bindep fallback tests https://review.openstack.org/530104 | 11:28 |
| frickler | AJaeger: oops, doing a git pull, thanks | 11:28 |
| AJaeger | frickler: we discussed that yesterday in chat and that's why patchset 1 and 2 are completely different ;) | 11:29 |
| *** udesale__ has quit IRC | 11:29 | |
| *** khappone has quit IRC | 11:29 | |
| AJaeger | frickler: see also the needed-by, it seems to work just fine | 11:29 |
| AJaeger | (the error there is unrelated, sudo works) | 11:30 |
| *** iyamahat has joined #openstack-infra | 11:32 | |
| AJaeger | frickler: could you review https://review.openstack.org/531307 as well, please? https://review.openstack.org/531339 and https://review.openstack.org/531361 show why I want it ;) | 11:32 |
| AJaeger | thanks! | 11:32 |
| *** iyamahat_ has joined #openstack-infra | 11:32 | |
| *** Qiming has quit IRC | 11:33 | |
| *** Qiming has joined #openstack-infra | 11:34 | |
| *** nicolasbock has joined #openstack-infra | 11:35 | |
| frickler | AJaeger: ack | 11:35 |
| *** sree has quit IRC | 11:36 | |
| *** iyamahat has quit IRC | 11:36 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs template for cinder-tempest-plugin https://review.openstack.org/531361 | 11:36 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Added openstack-tempest-plugins job template https://review.openstack.org/531307 | 11:40 |
| *** sree has joined #openstack-infra | 11:40 | |
| *** udesale has quit IRC | 11:44 | |
| *** iyamahat_ has quit IRC | 11:45 | |
| *** markvoelker has quit IRC | 11:45 | |
| *** katkapilatova1 has joined #openstack-infra | 11:53 | |
| *** sree has quit IRC | 11:53 | |
| *** katkapilatova1 has left #openstack-infra | 11:54 | |
| *** sree has joined #openstack-infra | 11:56 | |
| *** Qiming has quit IRC | 11:57 | |
| *** Qiming has joined #openstack-infra | 12:00 | |
| *** annp has quit IRC | 12:00 | |
| *** jcoufal has joined #openstack-infra | 12:04 | |
| *** spzala has joined #openstack-infra | 12:07 | |
| *** spzala has quit IRC | 12:12 | |
| *** sree has quit IRC | 12:16 | |
| *** katkapilatova1 has joined #openstack-infra | 12:20 | |
| *** sree has joined #openstack-infra | 12:22 | |
| *** kong has quit IRC | 12:23 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs template for cinder-tempest-plugin https://review.openstack.org/531361 | 12:23 |
| *** smatzek has joined #openstack-infra | 12:23 | |
| *** numans has quit IRC | 12:24 | |
| *** numans has joined #openstack-infra | 12:24 | |
| *** spzala has joined #openstack-infra | 12:25 | |
| *** jamesdenton has quit IRC | 12:25 | |
| *** rhallisey has joined #openstack-infra | 12:25 | |
| *** numans has quit IRC | 12:25 | |
| *** jcoufal_ has joined #openstack-infra | 12:27 | |
| *** spzala has quit IRC | 12:29 | |
| *** jcoufal has quit IRC | 12:29 | |
| *** numans has joined #openstack-infra | 12:30 | |
| *** sree has quit IRC | 12:30 | |
| *** sree has joined #openstack-infra | 12:34 | |
| *** sree has quit IRC | 12:39 | |
| *** markvoelker has joined #openstack-infra | 12:42 | |
| *** sree has joined #openstack-infra | 12:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add #openstack-kayobe to accessbot channel list https://review.openstack.org/530778 | 12:44 |
| *** lucasagomes is now known as lucas-hungry | 12:46 | |
| *** sree has quit IRC | 12:47 | |
| *** rkukura has quit IRC | 12:47 | |
| *** rkukura has joined #openstack-infra | 12:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix bindep_file for bindep fallback https://review.openstack.org/531369 | 12:48 |
| *** janki has quit IRC | 12:50 | |
| AJaeger | frickler: the check experimental on the bindep showed that it did not work, see http://logs.openstack.org/25/531225/1/experimental/project-config-bindep-fallback-centos-7/bb3a0a0/job-output.txt.gz . What do you think of fix above? ^ | 12:51 |
| *** sree has joined #openstack-infra | 12:56 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Fix bindep_file for bindep fallback https://review.openstack.org/531369 | 12:57 |
| AJaeger | frickler: oops - thanks for fixing | 12:58 |
| *** sree has quit IRC | 13:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix bindep_file for bindep fallback https://review.openstack.org/531369 | 13:06 |
| *** psachin has quit IRC | 13:09 | |
| *** jkilpatr has joined #openstack-infra | 13:09 | |
| *** erlon has joined #openstack-infra | 13:11 | |
| *** sree has joined #openstack-infra | 13:11 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Add neutron-tempest-plugin-api job https://review.openstack.org/529000 | 13:12 |
| frickler | AJaeger: as you may have noticed, I started hijacking your etherpad, do you mind if I add ^^ too? | 13:13 |
| *** edmondsw has joined #openstack-infra | 13:13 | |
| *** markvoelker has quit IRC | 13:15 | |
| AJaeger | frickler: go ahead! | 13:17 |
| *** edmondsw has quit IRC | 13:17 | |
| AJaeger | frickler: I was planning to retire it but we can keep using it ... | 13:17 |
| *** kiennt26 has joined #openstack-infra | 13:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: DNM: Test new bindep tests https://review.openstack.org/531225 | 13:19 |
| AJaeger | frickler: bindep check experimental worked on 53125, now let's do negative test ;) | 13:20 |
| *** sree has quit IRC | 13:20 | |
| AJaeger | mordred: could you review https://etherpad.openstack.org/p/zbUQLqDR6H , please? | 13:21 |
| *** dave-mccowan has joined #openstack-infra | 13:22 | |
| *** sree has joined #openstack-infra | 13:23 | |
| AJaeger | mordred: eumel8 and myself wonder why the variable is set at all - it breaks openstack-manuals publishing | 13:23 |
| *** eharney has joined #openstack-infra | 13:24 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: Add /node-list to the webapp https://review.openstack.org/499969 | 13:26 |
| *** ijw has quit IRC | 13:26 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: Add /label-list to the webapp https://review.openstack.org/500254 | 13:26 |
| *** sree has quit IRC | 13:27 | |
| *** jcoufal_ has quit IRC | 13:28 | |
| *** sree has joined #openstack-infra | 13:28 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add relase notes to n-g-s https://review.openstack.org/531375 | 13:29 |
| *** dbecker has joined #openstack-infra | 13:30 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531161 | 13:30 |
| fungi | dmsimard: we should keep an eye on mw's efforts to reimplement the welcome message feature as a gerrit plugin, since it sounds like we'll need it for our next gerrit upgrade anyway | 13:31 |
| fungi | great find--thanks for the link! | 13:32 |
| *** pgadiya has quit IRC | 13:36 | |
| *** markvoelker has joined #openstack-infra | 13:36 | |
| AJaeger | config-core, frickler and myself cleaned up the backlog list - and added new ones. Please take a few changes from https://etherpad.openstack.org/p/Nvt3ovbn5x for review. | 13:36 |
| *** fultonj has quit IRC | 13:37 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add tox-nodejs-npm base job for python and javascript https://review.openstack.org/529663 | 13:40 |
| *** rlandy has joined #openstack-infra | 13:41 | |
| *** fultonj has joined #openstack-infra | 13:45 | |
| *** links has quit IRC | 13:47 | |
| *** caphrim007_ has joined #openstack-infra | 13:47 | |
| *** pcichy_ has joined #openstack-infra | 13:48 | |
| *** hughhalf has quit IRC | 13:48 | |
| *** tonyb has quit IRC | 13:49 | |
| *** caphrim007 has quit IRC | 13:50 | |
| *** tonyb has joined #openstack-infra | 13:50 | |
| *** swest has quit IRC | 13:50 | |
| *** pahuang has quit IRC | 13:50 | |
| *** jistr is now known as jistr|biab | 13:51 | |
| *** pgadiya has joined #openstack-infra | 13:53 | |
| *** edmondsw has joined #openstack-infra | 13:53 | |
| *** pgadiya has quit IRC | 13:53 | |
| *** pahuang has joined #openstack-infra | 13:54 | |
| *** hughhalf has joined #openstack-infra | 13:54 | |
| *** spzala has joined #openstack-infra | 13:56 | |
| *** swest has joined #openstack-infra | 14:00 | |
| *** sree has quit IRC | 14:00 | |
| *** sree has joined #openstack-infra | 14:01 | |
| *** spzala has quit IRC | 14:03 | |
| *** sree has quit IRC | 14:05 | |
| *** claudiub|2 is now known as claudiub | 14:08 | |
| *** mriedem has joined #openstack-infra | 14:12 | |
| *** bobh has joined #openstack-infra | 14:14 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531161 | 14:19 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/infra-manual master: zuul v3: Expand section on what *to* convert https://review.openstack.org/524566 | 14:21 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/infra-manual master: Remove version and release https://review.openstack.org/524567 | 14:21 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/infra-manual master: trivial: Remove cruft from 'conf.py' https://review.openstack.org/524568 | 14:21 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Adjust gerrit ACLs for networking-generic-switch https://review.openstack.org/531382 | 14:26 |
| *** ijw has joined #openstack-infra | 14:27 | |
| *** bobh has quit IRC | 14:29 | |
| dmsimard | fungi: yeah I didn't read the whole thread but it seemed like they wanted to leverage jeepyb | 14:29 |
| *** spzala has joined #openstack-infra | 14:30 | |
| *** smatzek has quit IRC | 14:31 | |
| *** smatzek has joined #openstack-infra | 14:32 | |
| *** spzala has quit IRC | 14:35 | |
| *** katkapilatova1 has quit IRC | 14:35 | |
| *** smatzek has quit IRC | 14:37 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Adjust gerrit ACLs for networking-generic-switch https://review.openstack.org/531382 | 14:37 |
| *** ramishra has quit IRC | 14:37 | |
| *** gridinv has quit IRC | 14:39 | |
| *** esberglu has joined #openstack-infra | 14:39 | |
| *** gridinv has joined #openstack-infra | 14:40 | |
| *** esberglu_ has joined #openstack-infra | 14:42 | |
| *** esberglu has quit IRC | 14:45 | |
| *** esberglu_ is now known as esberglu | 14:45 | |
| *** hongbin has joined #openstack-infra | 14:48 | |
| *** jaypipes has joined #openstack-infra | 14:48 | |
| *** jaypipes is now known as leakypipes | 14:51 | |
| *** nicolasbock has quit IRC | 14:51 | |
| *** lucas-hungry is now known as lucasagomes | 14:52 | |
| fungi | dmsimard: first they did, but then they didn't like relying on the ssh api (understandable, there was no rest api when we wrote that originally) and then they decided that relying on the gerrit database is fragile (which it is) and noted that the design will break completely with gerrit 2.14, so proposed reimplementing as a gerrit plugin, which i think is a grand idea now that the plugin interface | 14:57 |
| fungi | has been ironed out | 14:57 |
| fungi | if i had the slightest bit of familiarity with java, i'd have offered to help since we'll need it anyway | 14:58 |
| *** smatzek has joined #openstack-infra | 15:02 | |
| *** jistr|biab is now known as jistr | 15:04 | |
| *** sree has joined #openstack-infra | 15:07 | |
| *** Goneri has joined #openstack-infra | 15:13 | |
| pabelanger | wow, git05.o.o spam happy yesterday :) | 15:16 |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Move location of devstack-gate unstack to post-tempest https://review.openstack.org/530726 | 15:17 |
| *** sree has quit IRC | 15:21 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params https://review.openstack.org/531390 | 15:21 |
| *** gouthamr has joined #openstack-infra | 15:22 | |
| fungi | pabelanger: we seem to spur the git servers on to send out their backlog of accumulated crash reports when we reboot them. for some reason those aren't getting sent when they happen (perhaps because their local mta is stopped?) | 15:22 |
| pabelanger | fungi: okay, is anybody looking into that? | 15:23 |
| fungi | not that i'm aware of... it's hard to pin down because we don't find out about it until it's fixed | 15:24 |
| pabelanger | also, does look like git servers are updated to 7.4. Going to see how we can turn on automatic updates now, like ubuntu | 15:24 |
| pabelanger | fungi: ack | 15:24 |
| fungi | someone could start, i suppose, by trying to generate e-mail from those servers to root and seeing if we receive them or they end up queued somewhere | 15:25 |
| *** weshay is now known as weshay_interview | 15:28 | |
| *** wolverineav has joined #openstack-infra | 15:31 | |
| *** caphrim007_ has quit IRC | 15:32 | |
| *** caphrim007 has joined #openstack-infra | 15:33 | |
| mriedem | clarkb: is there a known issue with getting cinder-specs docs refreshed? https://specs.openstack.org/openstack/cinder-specs/ | 15:37 |
| mriedem | hasn't been updated since 12/21 | 15:37 |
| *** caphrim007 has quit IRC | 15:37 | |
| AJaeger | mriedem: from 21st to 2nd we didn't publish any docs by accident. Should be fixed now. | 15:38 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params https://review.openstack.org/531390 | 15:39 |
| AJaeger | mriedem: https://wiki.openstack.org/wiki/Infrastructure_Status actually from 23rd to 3rd | 15:39 |
| mriedem | AJaeger: so are the docs updated on a cron? | 15:39 |
| mriedem | or does something have to be forced | 15:39 |
| AJaeger | mriedem: no, as post jobs. So, merge something and it should publish | 15:39 |
| mriedem | oh | 15:39 |
| AJaeger | it was just broken for that time ;( | 15:39 |
| mriedem | is there a way to do a manual publish? | 15:40 |
| fungi | mriedem: a zul admin can manually reenqueue the ref for the branch tip into the post pipeline | 15:41 |
| fungi | er, a zuul admin (sorry zul!) | 15:41 |
| mnaser | mriedem: time to fix the copyright year in cinder-specs) | 15:41 |
| *** kiennt26 has quit IRC | 15:41 | |
| mriedem | mnaser: smcginnis said he'd push a cleanup patch | 15:41 |
| *** spzala has joined #openstack-infra | 15:43 | |
| AJaeger | config-core, the old backlog review list is cleaned up - but frickler and myself added some more items to it. Please take a few changes from https://etherpad.openstack.org/p/Nvt3ovbn5x for review. | 15:46 |
| smcginnis | mnaser: In a doc, or is that the generated timestamp? | 15:48 |
| mnaser | smcginnis: i was making a bad joke about making a bunch of patches that just update dates :> | 15:49 |
| smcginnis | mnaser: Haha, pretty close to what I did. ;) | 15:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install yum-crontab on CentOS servers https://review.openstack.org/531401 | 15:50 |
| *** iyamahat has joined #openstack-infra | 15:50 | |
| *** stephenfin is now known as finucannot | 15:53 | |
| *** slaweq has quit IRC | 15:56 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install yum-crontab on CentOS servers https://review.openstack.org/531401 | 15:56 |
| *** iyamahat has quit IRC | 15:59 | |
| *** iyamahat has joined #openstack-infra | 16:03 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Add roles for static web content build and publication https://review.openstack.org/531406 | 16:05 |
| *** mtreinish has quit IRC | 16:08 | |
| *** mtreinish has joined #openstack-infra | 16:09 | |
| mordred | AJaeger: re: zuul_work_dir - responded a little on that etherpad | 16:09 |
| AJaeger | mordred: thanks. Reading your comments, did I understand you correctly that you will fix this? | 16:11 |
| AJaeger | mordred: btw. 531406 above might be interesting for you to review | 16:11 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Don't treat finger client disconnect as exception https://review.openstack.org/531408 | 16:12 |
| *** ijw has quit IRC | 16:13 | |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Move location of devstack-gate unstack to post-tempest https://review.openstack.org/530726 | 16:14 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params https://review.openstack.org/531390 | 16:18 |
| mordred | AJaeger: yah - on it | 16:23 |
| *** yamamoto has quit IRC | 16:26 | |
| AJaeger | mordred: great, thanks | 16:27 |
| *** weshay_interview is now known as weshay | 16:30 | |
| *** pcichy_ has quit IRC | 16:30 | |
| fungi | infra-root: a useful data point on meltdown patching (while waiting for ubuntu to get their packages built)... my debian/stable systems in rackspace rebooted with the purported patched kernel do have CONFIG_PAGE_TABLE_ISOLATION=y in /boot/config-`uname -r` and "Kernel/User page tables isolation: enabled" shows in dmesg output, but no cpu_insecure in /proc/cpuinfo (even though it's GenuineIntel) and | 16:34 |
| fungi | no /sys/kernel/debug/x86/pti_enabled or /proc/config.gz to check | 16:34 |
| *** e0ne has quit IRC | 16:34 | |
| dmsimard | ugh | 16:34 |
| fungi | i have a feeling ubuntu trusty and xenial will be similar | 16:34 |
| *** benj_ has joined #openstack-infra | 16:35 | |
| corvus | can we just build our own kernels? /me ducks | 16:38 |
| mnaser | or you can use centos /me ducks | 16:38 |
| openstackgerrit | Merged openstack-infra/statusbot master: Fix continuation character under Python 2 https://review.openstack.org/527461 | 16:39 |
| mnaser | also side note does anyone know how can one get an openstack cloak on freenode :> | 16:39 |
| *** caphrim007 has joined #openstack-infra | 16:39 | |
| *** armax has joined #openstack-infra | 16:39 | |
| corvus | mnaser: we rebooted centos yesterday and none of the things that are supposed to tell us kpti is enabled were present. | 16:39 |
| *** caphrim007 has quit IRC | 16:39 | |
| mnaser | corvus: CONFIG_KAISER=y | 16:39 |
| corvus | mnaser: it's the most frakenkernel of them all | 16:39 |
| mnaser | in /boot/config-$(uname -r) | 16:40 |
| mnaser | (i only found this because i was looking through the git repos) | 16:40 |
| *** openstackstatus has quit IRC | 16:40 | |
| corvus | mnaser: wow, didn't they stop calling it kaiser in like october? | 16:40 |
| *** caphrim007 has joined #openstack-infra | 16:40 | |
| dmsimard | corvus: we found a reliable way to tell after you left, actually | 16:40 |
| mnaser | https://git.centos.org/commitdiff/rpms!kernel.git/6202b1a49906b21f0cdce08b4ff853a65c658750 | 16:40 |
| mnaser | thats how i found out it was that anyways | 16:41 |
| dmsimard | corvus: /sys/kernel/debug/x86/pti_enabled will be present and 1 if enabled | 16:41 |
| *** caphrim007 has quit IRC | 16:41 | |
| *** openstackstatus has joined #openstack-infra | 16:41 | |
| *** ChanServ sets mode: +v openstackstatus | 16:41 | |
| *** caphrim007 has joined #openstack-infra | 16:41 | |
| dmsimard | however it turns out that debian's flavor doesn't have that so.. | 16:41 |
| openstackgerrit | Darragh Bailey proposed openstack/python-jenkins master: Pass form data as body instead of in the url https://review.openstack.org/474952 | 16:41 |
| mnaser | also, i think it might have to do with the qemu-kvm which runs the VMs | 16:41 |
| openstackgerrit | Darragh Bailey proposed openstack/python-jenkins master: Migration to using requests https://review.openstack.org/205637 | 16:41 |
| *** caphrim007 has quit IRC | 16:41 | |
| mnaser | because the qemu-kvm patch for libvirt exposes microcode related stuff to the guests | 16:42 |
| * mnaser looks for patch again | 16:42 | |
| mnaser | https://git.centos.org/blob/rpms!libvirt.git/4cd28d3f2984894ad10b69b4cd5880a9c8d0e156/SOURCES!libvirt-cpu-add-CPU-features-and-model-for-indirect-branch-prediction-protection.patch | 16:42 |
| dmsimard | yeah that's if you expose host-model | 16:43 |
| dmsimard | it depends, you could expose a generic qemu processor for example | 16:43 |
| mnaser | but i think by default it picks one of those based on your capabilities | 16:43 |
| *** caphrim007 has joined #openstack-infra | 16:43 | |
| *** mikal has quit IRC | 16:43 | |
| dmsimard | what default? the one from nova? | 16:43 |
| dmsimard | I think the default is host-model | 16:43 |
| dmsimard | which is a subset of all the host processor flags which should have the most compatibility (for things like live migration etc) | 16:44 |
| mnaser | yep | 16:44 |
| mnaser | so with that patch now that processor flag includes the microcode related stuff | 16:44 |
| *** mikal has joined #openstack-infra | 16:45 | |
| *** jaosorior has quit IRC | 16:45 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params https://review.openstack.org/531390 | 16:46 |
| fungi | remember we're mostly talking about hosts in rackspace at the moment, so these are on xen anyway | 16:46 |
| *** katkapilatova1 has joined #openstack-infra | 16:47 | |
| fungi | there are some similar xen patches out as of yesterday, but its hypervisor is also purportedly immune to meltdown getting attacked by guests if you're using hvm for all guests (which all the flavors we've booted are) and they provided a hypervisor shim you can live migrate your pv guests on top of to mitigate attacks from them as well | 16:48 |
| *** yamamoto has joined #openstack-infra | 16:50 | |
| *** Apoorva has joined #openstack-infra | 16:51 | |
| eharney | i'm seeing some odd behavior with irrelevant-files filtering here: https://review.openstack.org/#/c/531197/ | 16:52 |
| eharney | AFAICT tempest-full shouldn't run here, because of https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml?id=0a93ac#n3892 ? | 16:52 |
| *** ijw has joined #openstack-infra | 16:55 | |
| fungi | eharney: i want to say irrelevant-files doesn't work in a job variant unless it's a subset of irrelevant-files in the original job definition due to zuul not knowing how to deselect a job via the variant once it's been selected to run | 16:55 |
| fungi | someone should fact-check me on that one ^ | 16:56 |
| corvus | eharney: what fungi said plus: it's been selected to run because tempest-full is in the integrated-gate template | 16:56 |
| AJaeger | fungi: I run 4.14.11 on openSUSE have cpu_insecure, and the dmesg output of enabled - but no pti_enabled | 16:56 |
| smcginnis | Oh right, it can't be in a child definition, it needs to be at the source, right? | 16:56 |
| eharney | ok, should we just clean out the irrelevant-file filters in project-config there, then? | 16:56 |
| *** ijw_ has joined #openstack-infra | 16:57 | |
| smcginnis | And speaking of tempest-full, any update on that job with stable branches? | 16:57 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Delete IncludeRole object from result object for include_role tasks https://review.openstack.org/504238 | 16:57 |
| *** armaan has quit IRC | 16:57 | |
| fungi | AJaeger: is that on bare metal or a virtual machine? | 16:57 |
| AJaeger | fungi: bare metal | 16:58 |
| *** armaan has joined #openstack-infra | 16:58 | |
| fungi | i'm still waiting for updated kernel packages in debian/unstable, which is all i have on my bare metal systems so far | 16:58 |
| corvus | eharney, smcginnis: yeah, if you want it not to run there, you can add those irrelevant-files to the job itself, or you could drop the use of the integrated-gate template and keep the project stanza you have. | 16:58 |
| AJaeger | smcginnis: there's a patch for tempest that needs to merge | 16:58 |
| dmsimard | mordred: btw mnaser asked earlier if anyone knew how to get an openstack freenode cloak ? | 16:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove explicit bindep parameters https://review.openstack.org/531430 | 16:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add zuul_work_dir defaults to base-test https://review.openstack.org/531431 | 16:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Set defaults for zuul_work_dir and bindep_dir https://review.openstack.org/531432 | 16:58 |
| mnaser | dmsimard: you somehow have one | 16:59 |
| mnaser | :P | 16:59 |
| corvus | eharney, smcginnis: (or add those irrelevant-files to the integrated-gate template) | 16:59 |
| mordred | dmsimard: corvus is the right person to ask about knowing how - I think we have the ability but currently have no defined process for handing them out? | 16:59 |
| dmsimard | mnaser: it's a red hat cloak | 16:59 |
| dmsimard | mnaser: redhat/openstack | 16:59 |
| mnaser | ohhh i see | 16:59 |
| AJaeger | smcginnis: https://review.openstack.org/#/c/531046 | 16:59 |
| AJaeger | sorry, smcginnis, check https://review.openstack.org/#/c/531305/ | 17:00 |
| dmsimard | mordred: it needs to be handed out by ircops regardless but I believe someone in specific needs to do the request | 17:00 |
| smcginnis | AJaeger: Thank you! | 17:00 |
| corvus | mordred: oh, did i inherit that too? i've never worked on getting a cloak. :) | 17:00 |
| mordred | corvus: I just figure you have all the mega freenode powers | 17:00 |
| *** ijw has quit IRC | 17:00 | |
| corvus | mordred: according to the docs, yes... | 17:00 |
| AJaeger | smcginnis: 531046 merged 3h ago and should fix it, rest is cleanup. Sorrry, was a bit confused... | 17:00 |
| mordred | dmsimard, corvus: I got my openstack cloak from rick clark 7 years ago | 17:01 |
| dirk | AJaeger: doesn't loading ucode-intel remove that cpu_insecure marker? | 17:01 |
| dmsimard | that name doesn't ring me a bell, I'm a newbie from grizzly :( | 17:01 |
| mordred | dmsimard: :) | 17:01 |
| AJaeger | dirk: don't think so | 17:01 |
| dirk | AJaeger: fungi: iirc the official kernel updates for opensuse are meanwhile released | 17:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params https://review.openstack.org/531390 | 17:02 |
| corvus | mnaser, mordred, dmsimard: if someone wants to document the criteria for obtaining a cloak, i'm happy to execute the process as the openstack group contact. :) | 17:02 |
| mordred | dmsimard: he was the rackspace engineering manager over the team who initially kicked off the effort that became openstack and was on thefirst several instaces of the PPB | 17:02 |
| mnaser | https://meta.wikimedia.org/wiki/IRC/Cloaks -- i think this is a nice base document | 17:02 |
| fungi | dirk: awesome. and yeah, my debian/stable systems are already patched, i'm waiting for equivalent patches for debian/unstable at this point (which tends to lag since it doesn't get explicit security team support) | 17:03 |
| mnaser | my question is does corvus have access to doing them directly or does he have to reach out to staff, because that might be a lot of work to take on | 17:03 |
| mordred | corvus: I kind of think the criteria should be "any active ATC who requests one" | 17:03 |
| fungi | dirk: but ubuntu unfortunately still seems to have no updated kernel packages | 17:03 |
| dmsimard | mnaser: only ircops can hand out cloaks, but someone authorized to do so must do the request | 17:03 |
| * Shrews wants to trade his invisibility cloak for the elder wand | 17:03 | |
| mordred | corvus: but given I think the mechanism is "corvus requests a cloak" - we might want to make the request one from corvus have a larger barrier to entry | 17:03 |
| fungi | Shrews: i have a +3 lart of smiting if you need to borrow one | 17:04 |
| dmsimard | yeah "criteria to obtain a cloak" is certainly a good question | 17:04 |
| corvus | mnaser: i believe i will have to request access. but i'm happy to do it. if there are a lot of requests (there have been zero over the past 4 years), i will probably batch them. :) | 17:04 |
| mordred | corvus: unless we wanted to make the openstack bot account the openstack group contact :) | 17:04 |
| Shrews | fungi: lol | 17:04 |
| mugsie | so they never finished the cloak / group management tool? | 17:04 |
| dmsimard | there is a group management thing now | 17:04 |
| *** mriedem is now known as mriedem_luncharo | 17:04 | |
| *** Apoorva has quit IRC | 17:04 | |
| dmsimard | they reserve a namespace automatically when you register a "primary namespace" channel | 17:04 |
| dmsimard | as well as a cloak | 17:05 |
| dmsimard | but you still need to ask for the cloak | 17:05 |
| dmsimard | like I have the #ara namespace reserved and I could have an ara cloak | 17:05 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config master: Clean up irrelevant-files for Cinder tempest-full https://review.openstack.org/531434 | 17:05 |
| mnaser | yeah the asking for cloak happens manually | 17:05 |
| dmsimard | but someone could make ##ara in the silly secondary namespace thing | 17:05 |
| dmsimard | but no one could register #ara-is-awesome without my consent | 17:06 |
| mordred | AJaeger: I pushed up three patches - https://review.openstack.org/531430 should fix your original issue | 17:08 |
| corvus | mnaser: yeah, that wiki page looks like a good basis. if something like that lands in infra-manual, i'll make the requests. | 17:08 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Switch to pep8 jobs for manila-specs https://review.openstack.org/521159 | 17:08 |
| *** jpich has quit IRC | 17:08 | |
| mordred | AJaeger, dmsimard, corvus, pabelanger: https://review.openstack.org/531432 is a followup idea that I'd like opinions on before I do any of the work to verify it works | 17:08 |
| clarkb | good morning. Looks like no USNs yet | 17:09 |
| dmsimard | mordred: should it just be included in zuul proper ? | 17:09 |
| clarkb | as far as rax and our control plane goes apparently xen is only vulnerable in a limited set of use cases (64bit PV guests can attack xen but no others and all our instances should be hvm so if instances are segregated by xen mode or if rax stopped doing pv instances we are good) | 17:09 |
| dmsimard | at least the work dir | 17:09 |
| clarkb | I'm going to take a look at old git.o.o now and prep for deleting it | 17:10 |
| dmsimard | mordred: wait, I'm thinking of something else -- nevermind what I said | 17:10 |
| AJaeger | mordred: thanks, reviewing | 17:10 |
| chandankumar | dmsimard: please have a look on this https://review.openstack.org/#/c/531361/ thanks :-) | 17:11 |
| mordred | dmsimard: yah - must be in base if we do it anywhere - otherwise variable precedence will do the wrong thing | 17:11 |
| dmsimard | chandankumar: you're in luck, I'm actually doing a batch of review right now | 17:11 |
| *** dtroyer has quit IRC | 17:11 | |
| corvus | mordred: is that so that we can drop zuul_work_dir from defaults/main.yaml in all roles? anything else? | 17:11 |
| *** jtomasek has quit IRC | 17:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-vpnaas: Move permission to neutron-release/neutron-stable-maint https://review.openstack.org/529542 | 17:12 |
| dmsimard | AJaeger, frickler, chandankumar: I added a comment in https://review.openstack.org/#/c/531307/ -- we should be able to prevent those, indentation mistakes in yaml should not land | 17:12 |
| AJaeger | mordred: readd the bindep_profile: doc to 531430, please | 17:12 |
| dmsimard | chandankumar: can you please submit a follow up fix to address the indentation mistake | 17:13 |
| chandankumar | dmsimard: yup | 17:13 |
| dmsimard | chandankumar: there's actually two indentation mistakes | 17:13 |
| dmsimard | added a second comment | 17:13 |
| AJaeger | dmsimard: that template is not used currently ;( We should test it - but don't ;( | 17:13 |
| AJaeger | dmsimard: good catch, thanks! | 17:14 |
| dmsimard | corvus: does zuul do some amount of yaml linting-ish checks when it validates the configuration or should we add a job to detect things like bad indentation ? | 17:14 |
| dmsimard | corvus: i.e, https://review.openstack.org/#/c/531307/2/zuul.d/project-templates.yaml | 17:15 |
| AJaeger | dmsimard: if you want to review, you could also go over the entries in https://etherpad.openstack.org/p/Nvt3ovbn5x - but whatever works for you... I put in that list a backlog over some of our repos and frickler enhanced | 17:15 |
| *** ijw_ has quit IRC | 17:15 | |
| dmsimard | AJaeger: going through my two dashboards right now, I'll hit the pad after | 17:15 |
| corvus | dmsimard: it does not lint for style. | 17:16 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Fixed indentation typo in tempest plugin job template https://review.openstack.org/531436 | 17:16 |
| dmsimard | chandankumar: thanks | 17:17 |
| dmsimard | corvus: style in yaml can arguably lead to syntax errors (or outright unexpected behavior) though is mostly what I'm worried about | 17:17 |
| AJaeger | dmsimard: then you might have found everything on that one ;) Thanks | 17:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to pep8 jobs for manila-specs https://review.openstack.org/521159 | 17:18 |
| clarkb | ok git.openstack.org is being hit by one web search indexer and a handful of IPs owned by advancedhosters.com | 17:19 |
| clarkb | er old git.o.o | 17:20 |
| AJaeger | chandankumar: thanks! | 17:20 |
| clarkb | infra-root ^ any idea who advancedhosters.com could be? and should I go ahead and delete the old server at this point? | 17:20 |
| *** bobh has joined #openstack-infra | 17:20 | |
| *** markmcclain has quit IRC | 17:20 | |
| *** markmcclain has joined #openstack-infra | 17:21 | |
| *** pcaruana has quit IRC | 17:21 | |
| corvus | dmsimard: yeah, there's one point that i fully agree with that on: always indent after a colon. not doing that frequently leads people to errors (especially between lists and dicts), so we've adopted that as the house style in docs and zuul-jobs. i think it might be okay to try out a linter to catch that and ensure consistent indentation on zuul-jobs and maybe the other repos. i wouldn't want zuul itself to enforce indentation ... | 17:22 |
| corvus | ... though (eg, 531307 should always be accepted by zuul, but we could locally enforce more consistency) | 17:22 |
| *** dtantsur is now known as dtantsur|afk | 17:22 | |
| fungi | clarkb: on the xen meltdown front, my take is that we still need guest kernel updates to secure processes on our guests from other processes on our guests, but that rackspace has probably mostly dodged the bullet from the perspective of guests being able to attack other guests or attack the hypervisor host | 17:23 |
| clarkb | fungi: right we still need to patch to protect within our VMs but other tenants are hopfully locked away because xen | 17:23 |
| clarkb | also firefox is mitigating attack via javascript by reducing timer precision. So updating your firefox to 57 or greater is a good idea (even with a patched kernel) | 17:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Add api-ref job to blazar jobs https://review.openstack.org/530051 | 17:25 |
| fungi | clarkb: www.advancedhosters.com looks to be somewhere around amsterdam according to my traceroutes. dunno if that helps. at this point anyone who has continued hitting a host we took out of dns >12 hours ago (with a 1h ttl) can't exactly complain | 17:26 |
| corvus | i'm happy that we only run code from trusted projects on the executors | 17:26 |
| fungi | corvus: same here! | 17:26 |
| fungi | and the extra layers of effort we went to in securing secrets | 17:26 |
| fungi | so that zuul won't provide them to hosts running untrusted code | 17:27 |
| fungi | (at least insofar as our code review process manages to catch that) | 17:27 |
| *** iyamahat has quit IRC | 17:30 | |
| *** trown is now known as trown|lunch | 17:30 | |
| clarkb | alright I am deleting 39106fe5-2e12-4ca0-b5ba-b79e35abc590 in rax dfw now. This is the old git.openstack.org server | 17:32 |
| clarkb | that is done, increasing the dns ttl back to 1 hour on git.o.o records | 17:35 |
| fungi | huh, so openbsd has had effective meltdown workarounds in place for a decade... https://bsd.network/@phessler/99291274119329309 | 17:35 |
| fungi | makes me happy to continue placing my trust in it | 17:36 |
| *** danpawlik has quit IRC | 17:37 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fixed indentation typo in tempest plugin job template https://review.openstack.org/531436 | 17:37 |
| fungi | always nice when theo's rants create results https://marc.info/?l=openbsd-misc&m=118296441702631&w=2 | 17:38 |
| *** florianf has quit IRC | 17:39 | |
| clarkb | #status log Old git.openstack.org server has been deleted. New server's A and AAAA dns record TTLs bumped to an hour. We are now running with PTI enabled on all CentOS control plane servers. | 17:40 |
| openstackstatus | clarkb: finished logging | 17:40 |
| mordred | clarkb: woot | 17:40 |
| * fungi wonders whether we can rename our project testing interface to "consistent testing interface" for purposes of disambiguation :P | 17:41 | |
| *** niska has quit IRC | 17:42 | |
| dmsimard | either that or we need a branding, logo and our own website for our version of pti | 17:44 |
| dmsimard | and two or three nice marketing slogans | 17:44 |
| *** yamamoto has quit IRC | 17:47 | |
| johnsom | mordred Is there anything more that needs to happen with https://review.openstack.org/#/c/521211/ ? | 17:47 |
| SamYaple | fungi: too many bad experinces with/around theo :( turned me off of openbsd, but i wont disparange openbsd at all | 17:47 |
| *** niska has joined #openstack-infra | 17:47 | |
| fungi | i've found him less vitriolic than linus in recent years ;) | 17:48 |
| *** derekh has quit IRC | 17:48 | |
| SamYaple | well you arent wrong there. but i also dont deal/hear about linus on the daily when using linux. i do hear theo-isms in openbsd non-stop | 17:49 |
| SamYaple | i just want cool software ok! | 17:49 |
| fungi | fair point | 17:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Use tempest-plugin-jobs template for cinder-tempest-plugin https://review.openstack.org/531361 | 17:49 |
| pabelanger | infra-root: https://review.openstack.org/531401/ could use a review please, that should keep our centos servers packages up to date, as we do under ubuntu | 17:50 |
| *** shardy is now known as shardy_afk | 17:53 | |
| *** jascott1 has joined #openstack-infra | 17:53 | |
| *** iyamahat has joined #openstack-infra | 17:54 | |
| *** Apoorva has joined #openstack-infra | 17:54 | |
| *** xarses_ has joined #openstack-infra | 17:54 | |
| *** jascott1 has quit IRC | 17:54 | |
| clarkb | pabelanger: +2'd but left some comments on one behavior I noticed updating things yesterday and another is a minor puppet nit | 17:56 |
| *** sree has joined #openstack-infra | 17:56 | |
| *** dave-mccowan has quit IRC | 17:57 | |
| *** lucasagomes is now known as lucas-afk | 18:02 | |
| *** yamamoto has joined #openstack-infra | 18:03 | |
| *** yamamoto has quit IRC | 18:07 | |
| *** harlowja has joined #openstack-infra | 18:09 | |
| *** sree has quit IRC | 18:09 | |
| AJaeger | config-core, the tox-with-sudo jobs works, could mordred and myself get a second +2 on https://review.openstack.org/512653, please? | 18:13 |
| clarkb | "Ubuntu users of the 64-bit x86 architecture (aka, amd64) can expect updated kernels by the original January 9, 2018 coordinated release date, and sooner if possible." | 18:14 |
| clarkb | AJaeger: I will take a look | 18:14 |
| *** ijw has joined #openstack-infra | 18:15 | |
| dmsimard | yeah the coordinated release date was thrown out the window and they released the vuln info before the planned date | 18:16 |
| dmsimard | so it caught some vendors off guard | 18:16 |
| dmsimard | the vulns were getting way too much exposure even before they were officially announced | 18:16 |
| *** mriedem_luncharo is now known as mriedem | 18:17 | |
| *** sree has joined #openstack-infra | 18:17 | |
| clarkb | AJaeger: mordred why does js require sudo | 18:17 |
| *** jascott1 has joined #openstack-infra | 18:19 | |
| clarkb | oh is it because pre of parent runs before your pre? that must be it | 18:19 |
| AJaeger | clarkb: yes | 18:20 |
| clarkb | they still revoke sudo just later in the lits of things | 18:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Add #openstack-self-healing to accessbot https://review.openstack.org/531135 | 18:20 |
| *** sree has quit IRC | 18:26 | |
| *** chandankumar is now known as {archer} | 18:27 | |
| *** pcichy_ has joined #openstack-infra | 18:28 | |
| *** pcichy_ has quit IRC | 18:29 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs https://review.openstack.org/512653 | 18:30 |
| *** sree has joined #openstack-infra | 18:30 | |
| *** electrofelix has quit IRC | 18:30 | |
| *** pcichy_ has joined #openstack-infra | 18:31 | |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/project-config master: Remove Neutron legacy jobs definition https://review.openstack.org/530500 | 18:33 |
| *** sambetts is now known as sambetts|afk | 18:37 | |
| *** dtroyer has joined #openstack-infra | 18:37 | |
| *** armaan has quit IRC | 18:44 | |
| *** trown|lunch is now known as trown | 18:45 | |
| *** armaan has joined #openstack-infra | 18:47 | |
| *** sree has quit IRC | 18:47 | |
| *** yamamoto has joined #openstack-infra | 18:48 | |
| clarkb | with 512653 merged above I'm going to mark all the stuff on zuulv3-issues related to lack of sudo + tox as fixed (jobs will need to be updated to do the right thing but now there is a clear right thing for them to do) | 18:48 |
| *** katkapilatova1 has quit IRC | 18:48 | |
| *** numans has quit IRC | 18:50 | |
| *** numans has joined #openstack-infra | 18:51 | |
| *** yamamoto has quit IRC | 18:52 | |
| *** pcichy_ has quit IRC | 18:53 | |
| *** rlandy has quit IRC | 18:56 | |
| AJaeger | clarkb: thanks | 18:58 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test https://review.openstack.org/531470 | 18:58 |
| *** ilpianista_ has quit IRC | 19:03 | |
| *** ilpianista_ has joined #openstack-infra | 19:06 | |
| zxiiro | fungi: We'd like to cut a jjb 2.0 b3 release. Are you the one who can do that? | 19:06 |
| fungi | zxiiro: the acl says this is the group which can do it: https://review.openstack.org/#/admin/groups/jenkins-job-builder-release | 19:08 |
| fungi | zxiiro: i notice that zaro is in there explicitly, but we can certainly add more if clarkb says it's okay | 19:09 |
| clarkb | fungi: zxiiro ++ to delegating that to the active devs | 19:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test https://review.openstack.org/531470 | 19:09 |
| fungi | zxiiro: https://docs.openstack.org/infra/manual/drivers.html#tagging-a-release is the process for creating and pushing a tag | 19:10 |
| clarkb | I'm happy for zxiiro to do it if zxiiro is happy to do it | 19:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fake more of the github3 api https://review.openstack.org/531478 | 19:10 |
| *** openstack has joined #openstack-infra | 21:14 | |
| *** ChanServ sets mode: +o openstack | 21:14 | |
| fungi | infra-root: how have we been dealing with these servers which got built with no swap space? were there more during the xenial upgrade sprint which ran into this? | 21:15 |
| fungi | the servers i built during the sprint have swap partitions, so it doesn't seem to have impacted everything | 21:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove explicit bindep parameters https://review.openstack.org/531430 | 21:18 |
| AJaeger | mordred: please check my change ^ | 21:18 |
| * AJaeger calls it a day, would appreciate somebody telling me where i screwed up with https://review.openstack.org/531470 a | 21:19 | |
| fungi | thanks AJaeger! | 21:19 |
| *** smatzek has quit IRC | 21:20 | |
| *** smatzek has joined #openstack-infra | 21:20 | |
| *** hughhalf has quit IRC | 21:20 | |
| *** tonyb has quit IRC | 21:21 | |
| fungi | for some reason, there is no ephemeral disk on eavesdrop01 | 21:23 |
| *** smatzek has quit IRC | 21:25 | |
| *** numans has quit IRC | 21:25 | |
| *** erlon has quit IRC | 21:27 | |
| *** slaweq has joined #openstack-infra | 21:27 | |
| *** numans has joined #openstack-infra | 21:28 | |
| *** slaweq has quit IRC | 21:28 | |
| *** slaweq has joined #openstack-infra | 21:29 | |
| fungi | pabelanger: my irc log says you were the one who did the eavesdrop xenial rebuild during the sprint? do you happen to know if there was anything unusual about booting that which might have caused it to get no ephemeral disk? | 21:31 |
| fungi | in the interim, i'm going to create a swapfile on the rootfs. it has plenty of space | 21:32 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/openstack-zuul-jobs master: Remove Neutron legacy jobs definition https://review.openstack.org/531496 | 21:33 |
| *** ijw has joined #openstack-infra | 21:33 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add regex support to project stanzas https://review.openstack.org/513368 | 21:35 |
| fungi | #status log added 2gb swap file to eavesdrop01 at /swapfile since it has no ephemeral disk | 21:35 |
| openstackstatus | fungi: finished logging | 21:35 |
| dmsimard | fungi: that's a hotfix but if 1GB isn't enough we should consider making it 2GB :) | 21:36 |
| *** trown is now known as trown|outtypewww | 21:37 | |
| openstackgerrit | Gage Hugo proposed openstack-infra/project-config master: Remove legacy jobs from Keystone https://review.openstack.org/524232 | 21:38 |
| fungi | dmsimard: yeah, i'd like to hear back from pabelanger once he's recovered to discuss whether there was a specific reason we went with a 1gb flavor | 21:38 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: grenade: wait for service logs before matching against patterns https://review.openstack.org/493987 | 21:39 |
| dmsimard | fungi, apevec: re - unreachable nodes, the first step was to establish if there was a pattern in which the issues occured and we're thus tracking them with elastic-recheck queries | 21:40 |
| dmsimard | see http://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=76b9afa9d977a8e95cd349df2091f67e2cbbbf80 and http://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=9358e081deaed58f0a8a0e1972558752e2f707ef | 21:41 |
| dmsimard | I admittedly haven't spent time looking through those since the holidays | 21:41 |
| openstackgerrit | Gage Hugo proposed openstack-infra/project-config master: Remove legacy jobs from Keystone https://review.openstack.org/524232 | 21:41 |
| *** hashar has joined #openstack-infra | 21:41 | |
| dmsimard | e-r shows 55 hits in 10 days for privilege escalation timeout and, oh, yikes, 853 fails in 10 days for unreachable tasks | 21:42 |
| dmsimard | there was a definitive spike yesterday, I'll see if we can understand what's happening. | 21:43 |
| * dmsimard puts on good music | 21:43 | |
| fungi | dmsimard: so if you see the discussion we had about this particular incident, it wasn't unreachable, but was no longer responding to ansible task initiation and so timing out (if it had disconnected and attempted to reconnect over ssh, it would have seen a broken pipe after a ~2-minute hang, assuming my experience testing with an ssh client is indicative) | 21:43 |
| dmsimard | fungi: incident == packstack ? | 21:44 |
| *** sree has joined #openstack-infra | 21:44 | |
| fungi | yep | 21:44 |
| dmsimard | is this the issue where ansible basically hangs forever until the executor kills it ? | 21:44 |
| fungi | sounds similar at least | 21:45 |
| dmsimard | I've been meaning to troubleshoot that. I've seen it happen a few times. | 21:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Share a fake pull request database across connections https://review.openstack.org/531506 | 21:47 |
| dmsimard | infra-root: random finding: zuul-executor being oom' killed (on ze02) | 21:47 |
| fungi | anyway, on eavesdrop01 seems it's been hovering around 0.7gb ram used most of the time. something happened today to cause memory utilization to spike suddenly, i think. maybe someone spammed it with privmsg or something | 21:47 |
| *** gouthamr has joined #openstack-infra | 21:47 | |
| dmsimard | http://paste.openstack.org/raw/639284/ | 21:48 |
| *** Apoorva has joined #openstack-infra | 21:48 | |
| dmsimard | that paste got truncated for some reason | 21:48 |
| corvus | dmsimard: that will probably keep happening until we have a ram governor | 21:48 |
| fungi | dmsimard: yeah, lodgeit (what paste.o.o hosts) has a fairly smallish content size limit | 21:49 |
| openstackgerrit | Zara proposed openstack-infra/storyboard master: Pin version of webob to 1.7.4 https://review.openstack.org/531507 | 21:49 |
| dmsimard | corvus: it's thundering herd symptom you think ? should we consider adding a few executors ? looking at grafana.o.o they're pretty busy | 21:49 |
| dmsimard | sometimes running >80 concurrent jobs is probably asking a lot of ssh and throughput in general | 21:50 |
| *** dave-mccowan has quit IRC | 21:50 | |
| Shrews | dmsimard: that's actually probably the finger daemon being killed, not the actual executor | 21:50 |
| Shrews | since it's a child process | 21:50 |
| dmsimard | Shrews: when Ansible runs, the output of the command (non-streamed) is buffered on the control node or on the client node ? I ask in terms of RAM usage | 21:51 |
| Shrews | streaming module just reads from the console.log file | 21:52 |
| dmsimard | yeah I know that, but I'm going to guess we're not actually discarding the original output | 21:52 |
| dmsimard | either something's wrong with ze09 and ze10 or something's up with grafana.. their reported values are static | 21:54 |
| corvus | dmsimard: it's buffered on the remote side, then transmitted back json encoded in the normal way | 21:54 |
| clarkb | fungi: re eavesdrop I think we've typically just created a swap partition and shuffled data around as necessary to make that happen | 21:54 |
| dmsimard | oh sweet | 21:54 |
| dmsimard | dmsimard@ze09:~$ touch me | 21:54 |
| dmsimard | touch: cannot touch 'me': Read-only file system | 21:54 |
| clarkb | fungi: but if not ephemeral disk is prsent that would be a problem. Is that a flavor thing? | 21:54 |
| dmsimard | infra-root: I'm going to reboot ze09 | 21:55 |
| corvus | dmsimard: i don't know off hand if ansbile stores all of that in memory when decoding it, or otherwise. | 21:55 |
| dmsimard | looks like a live migration gone bad http://paste.openstack.org/raw/639285/ | 21:55 |
| *** Apoorva_ has joined #openstack-infra | 21:56 | |
| corvus | dmsimard: if you can clear out /var/lib/zuul/builds before the executor restarts, that would be good | 21:56 |
| fungi | dmsimard: check dmesg for info on why the filesystem was marked read-only? | 21:56 |
| dmsimard | corvus: read only filesystem | 21:56 |
| corvus | dmsimard: i understand :) | 21:56 |
| dmsimard | fungi: dmesg in paste | 21:56 |
| fungi | dmsimard: aha, i was still catching up | 21:57 |
| dmsimard | corvus: I'll try and do it quick after the reboot? | 21:57 |
| fungi | and yes, sounds likely | 21:57 |
| corvus | dmsimard: so you'll need to remount it before rebooting, or configure it not to start on boot | 21:57 |
| *** jtomasek has quit IRC | 21:57 | |
| corvus | dmsimard: do not race you will lose | 21:57 |
| dmsimard | corvus: I can systemctl disable | 21:57 |
| clarkb | dmsimard: evne on a ro fs? | 21:57 |
| dmsimard | ah, guess I can't do that heh | 21:57 |
| dmsimard | hmm | 21:57 |
| clarkb | remounting rw is probably best option | 21:57 |
| *** sree has quit IRC | 21:58 | |
| dmsimard | ok let me try | 21:58 |
| corvus | /var/lib/zuul/builds is not on the / fs, fwiw. | 21:58 |
| dmsimard | yeah I just noticed | 21:58 |
| corvus | (perhaps zuul-executor should clear out its build directory on start?) | 21:58 |
| fungi | yeah, i was just about to point that out | 21:58 |
| fungi | (the fact that /var/lib/zuul is on another device) | 21:59 |
| clarkb | corvus: I like that idea, similar to nodepool clearing out transitional state entries in its db on start | 21:59 |
| corvus | (then 'reboot' would be an okay fix to problems like this) | 21:59 |
| corvus | clarkb: ya | 21:59 |
| *** Apoorva has quit IRC | 21:59 | |
| *** spzala has quit IRC | 22:00 | |
| *** sree has joined #openstack-infra | 22:02 | |
| dmsimard | what's the service name for zuul-executor ? systemctl list-unit-files isn't showing anything resembling zuul/executor | 22:04 |
| dmsimard | or is it upstart ?& | 22:04 |
| *** yamamoto has joined #openstack-infra | 22:04 | |
| clarkb | dmsimard: zuul-executor | 22:04 |
| dmsimard | fwiw ps aux | grep zuul isn't returning anything either | 22:04 |
| clarkb | its a sysv init script | 22:05 |
| clarkb | run through systemd's compat laye | 22:05 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Configure all dvr nodes to dvr_snat agent mode https://review.openstack.org/531509 | 22:05 |
| dmsimard | ah, that's prolly why list-unit-files isn't picking it up | 22:05 |
| *** sree has quit IRC | 22:06 | |
| dmsimard | ok anyway, I was able to remount /var/lib/zuul rw after killing a few hung processed, cleaning /var/lib/zuul/builds and I'll reboot | 22:06 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 22:06 |
| clarkb | note depending on when the fs went ro the fixes to the init script may not have been applied yet | 22:07 |
| clarkb | so zuul-executor won't start on boot in that case, instead we need puppet to run then we can start it | 22:07 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Configure all dvr nodes to dvr_snat agent mode https://review.openstack.org/531509 | 22:07 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 22:07 |
| *** gouthamr has quit IRC | 22:07 | |
| *** bobh has quit IRC | 22:08 | |
| dmsimard | even remounting rw isn't working, dmesg complaining about filesystem errors.. going to run a fsck | 22:08 |
| dmsimard | wow that's a lot of errors :/ | 22:09 |
| *** Goneri has quit IRC | 22:10 | |
| dmsimard | are the executors using volumes or ephemeral storage ? | 22:10 |
| *** sree has joined #openstack-infra | 22:10 | |
| fungi | mgagne: looks like a partition on the ephemeral disk | 22:12 |
| fungi | er, dmsimard ^ | 22:12 |
| fungi | sorry mgagne! | 22:12 |
| *** gouthamr has joined #openstack-infra | 22:12 | |
| fungi | dmsimard: was /var/lib/zuul read-only too? | 22:13 |
| dmsimard | after fsck: 4.7Glost+found/ | 22:13 |
| dmsimard | fungi: yeah, it was in a pretty bad shape | 22:13 |
| fungi | i can write to it on ze09 | 22:13 |
| dmsimard | I just remounted it after finishing the fsck | 22:14 |
| fungi | oh, you already checked and remounted | 22:14 |
| fungi | k | 22:14 |
| dmsimard | maybe we should consider using boot from volume, it helps especially for live migrations | 22:14 |
| *** armaan_ has quit IRC | 22:14 | |
| fungi | does it in all cases? | 22:14 |
| fungi | again, remember, this is xen | 22:14 |
| dmsimard | hmm, I don't know about xen -- but with a volume you're basically syncing the dirty page, disconnecting the volume from somewhere and connecting it elsewhere | 22:15 |
| *** gouthamr has quit IRC | 22:15 | |
| fungi | i don't even know if rax has bfv support | 22:15 |
| *** gouthamr has joined #openstack-infra | 22:15 | |
| dmsimard | for ephemeral (unless it's backed by centralized storage like ceph), you're doing a cold rsync of the backing disk | 22:16 |
| dmsimard | considering the executors have high i/o throughput, they're not very good candidates | 22:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 22:16 |
| clarkb | the executors are also ephemeral instances | 22:16 |
| fungi | makes sense not to use ephemeral for write-heavy use cases in that scenario, agreed | 22:16 |
| clarkb | so sticking them on long lived data safe volumes feels wrong | 22:16 |
| *** gouthamr has quit IRC | 22:16 | |
| *** yamamoto has quit IRC | 22:17 | |
| *** bobh has joined #openstack-infra | 22:17 | |
| dmsimard | it's not so much about the data being important than it is about making it easier to live migrate them | 22:17 |
| clarkb | dmsimard: right but there is no point in live migrating them at all | 22:17 |
| clarkb | (but no good way to communicate that to rax) | 22:17 |
| dmsimard | rax live migrates stuff when they do maintenance on compute nodes, it's routine | 22:17 |
| fungi | starting to wonder if that means putting swap on ephemeral disk might explain some of our migration failures/hangs too | 22:17 |
| clarkb | the other issue with boot from volume is our volume quote is far more limited | 22:18 |
| clarkb | (because the idea is its for data you car eabout) | 22:18 |
| clarkb | we would not be able to boot all our instances from volume with currentquota or any where close iirc | 22:18 |
| fungi | and also the minimum cinder volume size rax allows is 100gb | 22:18 |
| dmsimard | well, anyway, something to keep at the back of our heads | 22:19 |
| *** wolverineav has joined #openstack-infra | 22:19 | |
| dmsimard | builds are clean, there's 4GB worth of stuff in lost+found, I doubt we have anything important to recover from there | 22:20 |
| dmsimard | rebooting | 22:20 |
| clarkb | dmsimard: we shouldn't since its just git repos and build data. The git repos are hosted elsewhere and builds we just rerun | 22:21 |
| dmsimard | hm, actually I suspect we'll need to do fsck in console -- do we have a root password ? | 22:21 |
| clarkb | dmsimard: we don't | 22:21 |
| dmsimard | ok I'll set one | 22:21 |
| dmsimard | no wait I can't :D | 22:21 |
| dmsimard | oh well, here goes | 22:22 |
| *** edmondsw has quit IRC | 22:22 | |
| fungi | worst case we rebuild the instance | 22:22 |
| dmsimard | yeah | 22:22 |
| fungi | there's nothing of importance stored on it | 22:22 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 22:22 |
| *** sree has quit IRC | 22:22 | |
| *** edmondsw has joined #openstack-infra | 22:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Delete stale jobdirs at startup https://review.openstack.org/531510 | 22:23 |
| *** bobh has quit IRC | 22:23 | |
| *** wolverineav has quit IRC | 22:24 | |
| clarkb | ubuntu has updated firefox for meltdown/spectre | 22:24 |
| dmsimard | ew, straight to initramfs for ze09 | 22:24 |
| *** slaweq has quit IRC | 22:25 | |
| *** edmondsw has quit IRC | 22:26 | |
| *** smatzek has joined #openstack-infra | 22:27 | |
| *** sree has joined #openstack-infra | 22:27 | |
| fungi | it's too bad i have to use firefox-esr to be able to continue using xul extensions | 22:28 |
| fungi | ff57 no longer supports them at all | 22:28 |
| fungi | hopefully they'll backport the protections to the esr branch | 22:29 |
| *** bobh has joined #openstack-infra | 22:30 | |
| *** smatzek has quit IRC | 22:32 | |
| dmsimard | ze09's root partition is hosed | 22:34 |
| dmsimard | I'll check ze10 for the time being which is also out of commission | 22:34 |
| *** bobh has quit IRC | 22:34 | |
| clarkb | probably best to just rebuild them then (assuming 10 is in similar state | 22:35 |
| dmsimard | yeah I can't ssh to ze10 so it doesn't look good | 22:35 |
| clarkb | if you can ssh to 10 then just try a reboot | 22:35 |
| clarkb | thats the typical fix for live migration woes | 22:35 |
| *** spzala has joined #openstack-infra | 22:35 | |
| clarkb | (and typically ssh doesn't work) | 22:35 |
| dmsimard | yeah it doesn't but it looks like ze10 might be recoverable, only see oom errors in console .. no filesystem errors | 22:36 |
| dmsimard | so I'll retry a reboot through the api | 22:37 |
| *** slaweq has joined #openstack-infra | 22:39 | |
| dmsimard | clarkb: procedure to rebuild a ze is similar to a logstash-worker ? :D i.e, build new node, change dns and that's pretty much it ? | 22:39 |
| *** slaweq has quit IRC | 22:39 | |
| clarkb | dmsimard: yes, I'm not sure of all the places where the firewall might need to be kicked to pick up the new node but otherwise yup | 22:40 |
| dmsimard | I think corvus fixed dns resolution | 22:40 |
| dmsimard | for the firewall rules | 22:40 |
| clarkb | dmsimard: thats on boot, but places like the zookeeper server will need to be kicked | 22:40 |
| clarkb | I think nodepool.openstack.org and zuulv3.openstack.org are the two places with holes punched for zuul executors (zk and gearman respectively) | 22:41 |
| *** hashar has quit IRC | 22:41 | |
| clarkb | but we can sort that out after the node is booting | 22:41 |
| *** wolverineav has joined #openstack-infra | 22:41 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 22:42 |
| *** e0ne has joined #openstack-infra | 22:42 | |
| *** bobh has joined #openstack-infra | 22:43 | |
| *** wolverin_ has joined #openstack-infra | 22:43 | |
| fungi | well, puppet is auto-resolving the hostnames to ip addresses and updating the rules with them at each pulse | 22:43 |
| fungi | so in theory it'll work as long as you can wait for puppet to do that? | 22:44 |
| clarkb | oh right because its causing a hcange in the config file it will pulse it | 22:45 |
| *** wolverin_ has quit IRC | 22:45 | |
| clarkb | so ya ~1-45 minutes after dns updates it will update | 22:45 |
| dmsimard | ze10 is back but zuul builds could not be cleaned in time :/ | 22:45 |
| *** wolverineav has quit IRC | 22:45 | |
| dmsimard | server otherwise looks healthy | 22:46 |
| *** hongbin has quit IRC | 22:46 | |
| clarkb | dmsimard: if you look at ls -l output you'll see the stale builds are quite stale and stand out based on timestamp | 22:46 |
| dmsimard | yeah that's what I had in mind | 22:47 |
| clarkb | we should be able to delete anything older than today | 22:47 |
| clarkb | oh also zuul isn't running | 22:47 |
| clarkb | so I don't think it raced? | 22:47 |
| *** bobh has quit IRC | 22:47 | |
| *** wolverineav has joined #openstack-infra | 22:47 | |
| dmsimard | I did a stop but not fast enough | 22:48 |
| dmsimard | :D | 22:48 |
| clarkb | ah | 22:48 |
| *** katkapilatova1 has joined #openstack-infra | 22:48 | |
| dmsimard | there's one build going afaict | 22:48 |
| *** katkapilatova1 has left #openstack-infra | 22:48 | |
| dmsimard | or 3, rather | 22:48 |
| *** slaweq_ has quit IRC | 22:48 | |
| dmsimard | there's actually quite a few stale builds, going back as far as december 4th | 22:48 |
| clarkb | ya | 22:48 |
| dmsimard | should there be something to prune those ? | 22:48 |
| *** bobh has joined #openstack-infra | 22:49 | |
| clarkb | dmsimard: thats jeblairs change above | 22:49 |
| clarkb | https://review.openstack.org/531510 | 22:49 |
| *** sree has quit IRC | 22:49 | |
| dmsimard | oh, neat | 22:49 |
| *** wolverin_ has joined #openstack-infra | 22:50 | |
| *** jtomasek has joined #openstack-infra | 22:50 | |
| *** wolverineav has quit IRC | 22:52 | |
| clarkb | corvus: left a comment on ^ | 22:53 |
| *** bobh has quit IRC | 22:53 | |
| *** sree has joined #openstack-infra | 22:53 | |
| *** wolverin_ has quit IRC | 22:54 | |
| *** wolverineav has joined #openstack-infra | 22:55 | |
| *** wolverineav has quit IRC | 22:56 | |
| *** wolverineav has joined #openstack-infra | 22:56 | |
| dmsimard | also left another comment | 22:57 |
| dmsimard | need to add a os.path.join | 22:57 |
| *** wolverineav has quit IRC | 22:58 | |
| *** sree has quit IRC | 22:58 | |
| dmsimard | ok I started zuul-executor on ze10 after wiping old builds | 22:58 |
| dmsimard | going to relocate and start rebuilding ze09 | 22:58 |
| dmsimard | clarkb: safe to assume I can already delete the existing instance ? | 22:58 |
| clarkb | dmsimard: I think so. corvus pabelanger fungi ^ can you think of any reason not to? | 22:59 |
| *** wolverin_ has joined #openstack-infra | 22:59 | |
| *** bobh has joined #openstack-infra | 23:00 | |
| dmsimard | #status log (dmsimard) ze10 was rebooted after being hung since january 4th | 23:00 |
| openstackstatus | dmsimard: finished logging | 23:00 |
| *** sree has joined #openstack-infra | 23:02 | |
| *** bobh has quit IRC | 23:04 | |
| *** wolverin_ has quit IRC | 23:04 | |
| *** david-lyle has joined #openstack-infra | 23:04 | |
| *** wolverineav has joined #openstack-infra | 23:05 | |
| fungi | yeah, i would delete ze09 and the existing dns records, then build another to replace it | 23:06 |
| clarkb | note that puppet may fail if it runs when the old dns records are gone | 23:06 |
| fungi | though i suppose that could race puppet updating firewalls | 23:06 |
| fungi | yeah, that | 23:07 |
| *** sree has quit IRC | 23:07 | |
| clarkb | (probably not a huge issue as long as we get new records up soon and odn't forget) | 23:07 |
| fungi | so maybe just update dns | 23:07 |
| fungi | but probably fine either way | 23:07 |
| clarkb | fungi: care to review https://review.openstack.org/#/c/531249/ ? | 23:07 |
| clarkb | hoping that gets in before tonights periodic runs so I can double check that either over weekend or on monday | 23:07 |
| *** wolverin_ has joined #openstack-infra | 23:08 | |
| *** wolverineav has quit IRC | 23:09 | |
| *** wolverineav has joined #openstack-infra | 23:10 | |
| *** wolverin_ has quit IRC | 23:10 | |
| fungi | sure | 23:10 |
| *** sree has joined #openstack-infra | 23:11 | |
| dmsimard | fungi, clarkb: is there logic involved in making /var/lib/zuul it's own partition ? | 23:11 |
| corvus | dmsimard: it's manual :( | 23:11 |
| dmsimard | I don't see any of that in puppet, it just creates the directory tree | 23:11 |
| *** ltomasbo has quit IRC | 23:11 | |
| *** dalvarez has quit IRC | 23:11 | |
| dmsimard | corvus: ok so I shouldn't use launch-node because that'll get puppet to create the directory tree right away ? | 23:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Delete stale jobdirs at startup https://review.openstack.org/531510 | 23:12 |
| dmsimard | sorry, still kind of new to this :) | 23:12 |
| corvus | dmsimard: nah, use launch-node, but then rm or move it to the new partition | 23:13 |
| dmsimard | ack | 23:13 |
| corvus | if you just rm it, puppet will fix on next run | 23:13 |
| *** jpena|off has quit IRC | 23:14 | |
| *** wolverineav has quit IRC | 23:14 | |
| *** sree has quit IRC | 23:15 | |
| *** amoralej|off has quit IRC | 23:16 | |
| *** bobh has joined #openstack-infra | 23:18 | |
| *** bobh has quit IRC | 23:19 | |
| *** sree has joined #openstack-infra | 23:23 | |
| weshay | anyone around? https://review.openstack.org/#/c/530983/ looks stuck in the gate | 23:24 |
| weshay | http://zuulv3.openstack.org/stream.html?uuid=4f3fec55ee9f4148a8360f6df56551da&logfile=console.log | 23:24 |
| *** wolverineav has joined #openstack-infra | 23:26 | |
| *** wolverineav has quit IRC | 23:27 | |
| *** sree has quit IRC | 23:28 | |
| *** wolverineav has joined #openstack-infra | 23:28 | |
| *** Apoorva_ has quit IRC | 23:28 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-snmpd master: Enable snmpd so it works on systemd systems https://review.openstack.org/531520 | 23:29 |
| clarkb | infra-root ^ noticed when poking at git farm performance on cacti (trying to see if there is a noticeable difference in system load and such) | 23:29 |
| *** wolverin_ has joined #openstack-infra | 23:30 | |
| *** ltomasbo has joined #openstack-infra | 23:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Create artifacts dir regardless of afs rpm builds https://review.openstack.org/531249 | 23:30 |
| clarkb | weshay: I'm not sure its stuck. I think we've had prolems iwth the console log streaming not always working if the process has oomed | 23:31 |
| clarkb | weshay: is the blank console log the only data we have to assume its stuck? | 23:31 |
| weshay | ya.. that's all I have | 23:31 |
| corvus | clarkb: want to drop ensure => running? | 23:31 |
| clarkb | corvus: we can, in this case I didn't think it would hurt to have both | 23:31 |
| *** wolveri__ has joined #openstack-infra | 23:31 | |
| *** sree has joined #openstack-infra | 23:32 | |
| corvus | clarkb: probably not, so i +2d. i'd just perfer to stomp out all instances. | 23:32 |
| weshay | clarkb, other jobs that should have started before this one, have completed the same job. that would be the other data point | 23:32 |
| *** wolverineav has quit IRC | 23:32 | |
| dmsimard | clarkb: I'm getting a permission denied error (causing launch-node to fail) when it tries to refresh the inventory but it works with sudo -- should I use sudo when using launch-node ? http://paste.openstack.org/show/639289/ | 23:32 |
| corvus | dmsimard: are you in the right group? | 23:33 |
| clarkb | dmsimard: I think that means your user's group membership isn't correct. Ya check the launch node readme it has details on that | 23:33 |
| corvus | yeah, it's the pgraph at the top | 23:33 |
| dmsimard | hmm I was able to launch nodes before, let's see | 23:33 |
| clarkb | weshay: its possible the job had to be restarted for some reason. I can try and find where it is running really quick to at least track down the possible dead log streaming | 23:34 |
| *** esberglu has quit IRC | 23:34 | |
| corvus | dmsimard, clarkb: maybe the inventory file got remade with bad perms? | 23:34 |
| *** wolverin_ has quit IRC | 23:34 | |
| corvus | i can never remember where the cache is | 23:34 |
| dmsimard | I'm in admin afaict | 23:34 |
| *** esberglu has joined #openstack-infra | 23:34 | |
| clarkb | that could be (and ya I have to reread the launch script to find it) | 23:34 |
| corvus | -rw-rw-r-- 1 root admin 476869 Jan 5 23:30 ansible-inventory.cache | 23:35 |
| *** gyee has quit IRC | 23:35 | |
| dmsimard | corvus: it complains about /var/cache/ansible-inventory* | 23:35 |
| weshay | clarkb, ok.. thank you. | 23:35 |
| dmsimard | /var/cache/ansible-inventory_openstackci-rax is kinda weird because apparently there's no such thing in /var/cache.. only /var/cache/ansible-inventory | 23:35 |
| *** wolveri__ has quit IRC | 23:35 | |
| *** jtomasek has quit IRC | 23:36 | |
| corvus | did something change in os_client_config? | 23:37 |
| corvus | oh no, that's determined only by the inventory plugin | 23:38 |
| corvus | -rwxr-xr-x 1 root root 9771 Jan 2 16:16 /etc/ansible/hosts/openstack | 23:38 |
| corvus | how did that get updated? | 23:38 |
| dmsimard | January 2nd.. | 23:38 |
| corvus | it's puppeted -- copied in place from ansible | 23:39 |
| *** esberglu has quit IRC | 23:39 | |
| corvus | so it's a change to the openstack inventory file in ansible | 23:39 |
| clarkb | weshay: build is running from ze05.openstack.org and has processes running. If I log into the test instance it appears to be running quickstart | 23:39 |
| *** slaweq has joined #openstack-infra | 23:39 | |
| dmsimard | corvus: we had a few changes to clouds.yaml due to OVH in and out | 23:39 |
| weshay | clarkb, ok.. thanks for checking :) | 23:39 |
| dmsimard | can't think of anything else | 23:39 |
| *** e0ne has quit IRC | 23:40 | |
| corvus | the commit that made that cache file format landed on august 10. | 23:41 |
| corvus | the jan 2 change shouldn't matter | 23:42 |
| clarkb | isn't 664 root:admin correct? (I feel like I'm missing what we think changed in appropriately) | 23:42 |
| clarkb | Shrews: what is process for debugging console streaming with the proxy in place? looks like nothing listens on port 79 anymore on the executors? | 23:43 |
| corvus | clarkb: see the paste. it's trying to write to a cache dir that doesn't exist | 23:43 |
| corvus | clarkb: afaik, it's still on 79, but we will move it to an unprivileged port | 23:43 |
| *** slaweq has quit IRC | 23:44 | |
| clarkb | ah ok ya ze01 has a :79 listener but ze05 does not | 23:44 |
| corvus | likely oom killed | 23:44 |
| clarkb | corvus: do we have to restart the zuul-executor daemon to correct that? | 23:44 |
| corvus | clarkb, dmsimard: here's the code regarding the cache: https://github.com/ansible/ansible/commit/d9d983160d9acc45b33a99ed7e570a9351eaf2c9#diff-c3e81a1e2b883ffdfff123da47a1c27dR190 | 23:44 |
| *** sree has quit IRC | 23:44 | |
| corvus | clarkb: yes | 23:45 |
| *** sree has joined #openstack-infra | 23:45 | |
| dmsimard | corvus: that landed in ansible 2.4, env is using 2.1.. | 23:45 |
| corvus | dmsimard: we copy it from the git repo | 23:46 |
| dmsimard | oh | 23:46 |
| clarkb | [Fri Dec 22 13:54:01 2017] Killed process 2955 (zuul-executor) total-vm:653968kB, anon-rss:1476kB, file-rss:5344kB that is the cause I think | 23:47 |
| clarkb | weshay: ^ fyi | 23:47 |
| clarkb | its interesting how much smaller rss is but still enough to get it killed | 23:48 |
| corvus | clarkb: it's killing children of the largest process | 23:48 |
| weshay | k | 23:48 |
| weshay | the poor children | 23:48 |
| corvus | would be a good idea for apache, not so much for zuul | 23:48 |
| dmsimard | the message is literally "sacrifice child" too. | 23:48 |
| weshay | if there is only zuul, the children must be sacrificed ;) | 23:49 |
| corvus | dmsimard: best i can figure, we somehow had an old version of the inventory plugin stuck on there, and now it's been updated | 23:49 |
| dmsimard | corvus: yeah I was trying to see if there was something obvious in the puppet logs | 23:49 |
| corvus | dmsimard: i say run it as root for now... we'll probably need to update our cache directory creation in puppet to create those and set group ownership | 23:49 |
| clarkb | well puppet wasn't working until we fixed ovh | 23:49 |
| clarkb | but ovh wasn't broken tat long | 23:50 |
| corvus | clarkb: yeah, but that change was from many months ago | 23:50 |
| *** ijw has quit IRC | 23:50 | |
| clarkb | fwiw cacti doesn't show anything indicating pti is a major burden on the git servers | 23:53 |
| *** caphrim007_ has joined #openstack-infra | 23:54 | |
| pabelanger | fungi: dmsimard: re: eavesdrop01, no it sounds like there was an issue bringing the server online if no swap exists. It was launched using launch-node.py, I guess we need to maybe add some logic to detect if swap is missing after we run puppet and fail so we don't launch servers without swap? | 23:54 |
| *** caphrim007_ has quit IRC | 23:56 | |
| *** caphrim007 has quit IRC | 23:58 | |
| *** jascott1 has quit IRC | 23:59 | |
| *** sdague has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!