| *** sdake has joined #openstack-infra | 00:00 | |
| nibalizer | ok | 00:00 |
|---|---|---|
| nibalizer | +2a on your patch | 00:00 |
| jhesketh | ta | 00:01 |
| jhesketh | okay looks like ansible is recovering | 00:02 |
| jhesketh | daemontool_: so once ansible finishes the jenkins masters should have your change and therefore a recheck should work | 00:03 |
| nibalizer | jhesketh: looks good | 00:07 |
| jhesketh | I'll poke at the inventory shortly to not fail if a cloud is down | 00:07 |
| nibalizer | its copied out of ansible upstream iirc | 00:08 |
| *** thorst has joined #openstack-infra | 00:08 | |
| *** amitgandhinz has quit IRC | 00:08 | |
| jhesketh | yep | 00:08 |
| *** Qiming_ has quit IRC | 00:14 | |
| *** wmolicki has joined #openstack-infra | 00:16 | |
| *** roxanagh_ has quit IRC | 00:17 | |
| *** roxanaghe has joined #openstack-infra | 00:18 | |
| *** sflanigan has joined #openstack-infra | 00:19 | |
| *** thorst has quit IRC | 00:27 | |
| *** wmolicki has quit IRC | 00:28 | |
| *** wmolicki has joined #openstack-infra | 00:28 | |
| *** beekneemech has quit IRC | 00:29 | |
| *** bnemec has joined #openstack-infra | 00:31 | |
| *** roxanaghe has quit IRC | 00:32 | |
| *** roxanaghe has joined #openstack-infra | 00:34 | |
| *** zeih has joined #openstack-infra | 00:38 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Change Artifacts meeting chair & meeting ID https://review.openstack.org/284487 | 00:42 |
| *** markvoelker has joined #openstack-infra | 00:43 | |
| *** zeih has quit IRC | 00:43 | |
| *** wmolicki has quit IRC | 00:44 | |
| *** sdake has quit IRC | 00:45 | |
| *** markvoelker has quit IRC | 00:47 | |
| *** fedexo has joined #openstack-infra | 00:47 | |
| *** sdake has joined #openstack-infra | 00:50 | |
| *** bgaifullin has quit IRC | 00:55 | |
| *** achanda has quit IRC | 01:00 | |
| *** hayato_ has joined #openstack-infra | 01:00 | |
| *** zeih has joined #openstack-infra | 01:00 | |
| *** Jeffrey4l has joined #openstack-infra | 01:01 | |
| *** Daisy has joined #openstack-infra | 01:03 | |
| *** x00350071 has joined #openstack-infra | 01:04 | |
| *** amitgandhinz has joined #openstack-infra | 01:04 | |
| *** zeih has quit IRC | 01:05 | |
| *** sdake has quit IRC | 01:06 | |
| *** sdake has joined #openstack-infra | 01:08 | |
| *** salv-orlando has joined #openstack-infra | 01:12 | |
| *** Qiming_ has joined #openstack-infra | 01:14 | |
| *** amitgandhinz has quit IRC | 01:17 | |
| *** aysyd has joined #openstack-infra | 01:20 | |
| *** salv-orlando has quit IRC | 01:20 | |
| *** camunoz has quit IRC | 01:26 | |
| *** hichihara has joined #openstack-infra | 01:28 | |
| *** sam_wan has joined #openstack-infra | 01:29 | |
| *** aysyd has quit IRC | 01:31 | |
| *** Daisy has quit IRC | 01:32 | |
| *** Daisy has joined #openstack-infra | 01:32 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Enable network isolation on all the jobs. https://review.openstack.org/285674 | 01:32 |
| *** claudiub has joined #openstack-infra | 01:35 | |
| *** Daisy has quit IRC | 01:36 | |
| *** x00350071 is now known as xiangxinyong | 01:36 | |
| *** sdake has quit IRC | 01:43 | |
| *** camunoz has joined #openstack-infra | 01:43 | |
| *** Daisy has joined #openstack-infra | 01:43 | |
| *** markvoelker has joined #openstack-infra | 01:44 | |
| *** Daisy has quit IRC | 01:46 | |
| *** Daisy has joined #openstack-infra | 01:46 | |
| *** markvoelker has quit IRC | 01:48 | |
| *** Daisy_ has joined #openstack-infra | 01:50 | |
| *** yamamoto_ has joined #openstack-infra | 01:50 | |
| *** Daisy has quit IRC | 01:51 | |
| *** gildub has joined #openstack-infra | 01:52 | |
| *** yamamoto_ has quit IRC | 01:55 | |
| *** Sukhdev has joined #openstack-infra | 01:56 | |
| *** zeih has joined #openstack-infra | 02:01 | |
| *** jpr has quit IRC | 02:04 | |
| *** zeih has quit IRC | 02:06 | |
| *** Daisy_ has quit IRC | 02:06 | |
| *** Qiming_ has quit IRC | 02:07 | |
| *** Qiming has joined #openstack-infra | 02:08 | |
| *** yamamoto_ has joined #openstack-infra | 02:08 | |
| *** Daisy has joined #openstack-infra | 02:09 | |
| *** yamamoto_ has quit IRC | 02:09 | |
| *** achanda has joined #openstack-infra | 02:10 | |
| *** Sukhdev has quit IRC | 02:13 | |
| *** Sukhdev has joined #openstack-infra | 02:13 | |
| *** amitgandhinz has joined #openstack-infra | 02:14 | |
| *** achanda has quit IRC | 02:14 | |
| *** sripriya has joined #openstack-infra | 02:15 | |
| *** achanda has joined #openstack-infra | 02:15 | |
| *** camunoz has quit IRC | 02:21 | |
| *** achanda has quit IRC | 02:22 | |
| *** dkranz has quit IRC | 02:24 | |
| *** ajmiller has quit IRC | 02:25 | |
| *** rguillebert has quit IRC | 02:27 | |
| *** amitgandhinz has quit IRC | 02:27 | |
| *** thorst has joined #openstack-infra | 02:28 | |
| *** Sukhdev has quit IRC | 02:31 | |
| *** sdake has joined #openstack-infra | 02:31 | |
| *** camunoz has joined #openstack-infra | 02:33 | |
| *** rhallisey has quit IRC | 02:33 | |
| *** yamamoto has joined #openstack-infra | 02:33 | |
| *** Keedya has joined #openstack-infra | 02:34 | |
| *** thorst has quit IRC | 02:36 | |
| *** salv-orlando has joined #openstack-infra | 02:37 | |
| *** Daisy has quit IRC | 02:38 | |
| *** Daisy has joined #openstack-infra | 02:38 | |
| *** binbincong_ has quit IRC | 02:39 | |
| *** yamamoto has quit IRC | 02:40 | |
| *** Keedya has quit IRC | 02:42 | |
| *** Daisy has quit IRC | 02:43 | |
| *** Keedya has joined #openstack-infra | 02:43 | |
| *** Daisy has joined #openstack-infra | 02:43 | |
| *** markvoelker has joined #openstack-infra | 02:44 | |
| *** adreznec has quit IRC | 02:45 | |
| *** cbader_ has quit IRC | 02:45 | |
| *** camunoz has quit IRC | 02:46 | |
| *** hichihara has quit IRC | 02:46 | |
| Keedya | Hi there, can I get some reviews on https://review.openstack.org/#/c/284827/ | 02:46 |
| *** _joes_ has quit IRC | 02:46 | |
| *** lennyb__ has quit IRC | 02:46 | |
| *** camunoz has joined #openstack-infra | 02:46 | |
| Keedya | Happy Sunday :) | 02:46 |
| *** adreznec has joined #openstack-infra | 02:47 | |
| *** jokke_ has quit IRC | 02:47 | |
| *** jokke_ has joined #openstack-infra | 02:47 | |
| *** lennyb__ has joined #openstack-infra | 02:47 | |
| *** _joes_ has joined #openstack-infra | 02:48 | |
| *** hichihara has joined #openstack-infra | 02:48 | |
| *** ianychoi has quit IRC | 02:49 | |
| *** markvoelker has quit IRC | 02:49 | |
| *** ianychoi has joined #openstack-infra | 02:49 | |
| *** Keedya has quit IRC | 02:49 | |
| *** Keedya has joined #openstack-infra | 02:50 | |
| *** boris-42 has joined #openstack-infra | 02:50 | |
| *** Sukhdev has joined #openstack-infra | 02:52 | |
| *** roxanaghe has joined #openstack-infra | 02:52 | |
| *** salv-orlando has quit IRC | 02:53 | |
| *** Keedya has quit IRC | 02:55 | |
| *** binbincong has joined #openstack-infra | 02:56 | |
| *** binbincong has quit IRC | 03:05 | |
| *** binbincong has joined #openstack-infra | 03:05 | |
| *** julim has quit IRC | 03:07 | |
| *** vgridnev has joined #openstack-infra | 03:07 | |
| *** nelsnels_ has joined #openstack-infra | 03:09 | |
| *** nelsnelson has quit IRC | 03:12 | |
| *** roxanaghe has quit IRC | 03:14 | |
| *** sdake has quit IRC | 03:19 | |
| *** baoli has joined #openstack-infra | 03:21 | |
| *** tojuvone_ has quit IRC | 03:21 | |
| *** yamamoto_ has joined #openstack-infra | 03:23 | |
| *** amitgandhinz has joined #openstack-infra | 03:24 | |
| *** tojuvone has joined #openstack-infra | 03:28 | |
| *** Daisy has quit IRC | 03:33 | |
| *** Daisy has joined #openstack-infra | 03:34 | |
| *** Daisy has quit IRC | 03:34 | |
| *** Daisy has joined #openstack-infra | 03:34 | |
| *** thorst has joined #openstack-infra | 03:36 | |
| *** amitgandhinz has quit IRC | 03:37 | |
| *** achanda has joined #openstack-infra | 03:37 | |
| *** roxanaghe has joined #openstack-infra | 03:39 | |
| *** hayato__ has joined #openstack-infra | 03:39 | |
| *** hayato_ has quit IRC | 03:41 | |
| *** thorst has quit IRC | 03:41 | |
| *** roxanaghe has quit IRC | 03:42 | |
| *** roxanaghe has joined #openstack-infra | 03:42 | |
| *** achanda has quit IRC | 03:42 | |
| *** Daisy has quit IRC | 03:43 | |
| *** links has joined #openstack-infra | 03:44 | |
| *** Daisy has joined #openstack-infra | 03:44 | |
| openstackgerrit | Merged openstack-infra/system-config: Infra-cloud-west is currently offline https://review.openstack.org/285854 | 03:45 |
| *** sdake has joined #openstack-infra | 03:47 | |
| *** Daisy has quit IRC | 03:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Reinstall requests on Fedora builds https://review.openstack.org/285876 | 03:49 |
| *** baoli has quit IRC | 03:49 | |
| *** achanda has joined #openstack-infra | 03:49 | |
| *** Daisy has joined #openstack-infra | 03:52 | |
| *** stevemar has joined #openstack-infra | 03:53 | |
| *** Daisy has quit IRC | 03:54 | |
| *** Daisy has joined #openstack-infra | 03:54 | |
| *** achanda has quit IRC | 03:55 | |
| *** binbincong has quit IRC | 03:56 | |
| *** salv-orlando has joined #openstack-infra | 03:57 | |
| *** binbincong has joined #openstack-infra | 03:58 | |
| *** Daisy has quit IRC | 03:59 | |
| *** Daisy has joined #openstack-infra | 04:01 | |
| *** links has quit IRC | 04:01 | |
| *** zeih has joined #openstack-infra | 04:03 | |
| *** salv-orlando has quit IRC | 04:03 | |
| *** camunoz has quit IRC | 04:03 | |
| *** Sukhdev has quit IRC | 04:04 | |
| *** hayato_ has joined #openstack-infra | 04:05 | |
| *** Daisy has quit IRC | 04:06 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix PATH when being run from unactivated venv https://review.openstack.org/285878 | 04:06 |
| *** zeih has quit IRC | 04:07 | |
| *** dimtruck is now known as zz_dimtruck | 04:07 | |
| *** hayato__ has quit IRC | 04:08 | |
| *** hayato__ has joined #openstack-infra | 04:09 | |
| *** hayato___ has joined #openstack-infra | 04:10 | |
| *** hayato_ has quit IRC | 04:10 | |
| *** hayato_ has joined #openstack-infra | 04:11 | |
| *** hayato__ has quit IRC | 04:11 | |
| *** hayato__ has joined #openstack-infra | 04:11 | |
| *** david-lyle has joined #openstack-infra | 04:13 | |
| *** hayato___ has quit IRC | 04:15 | |
| *** hayato_ has quit IRC | 04:15 | |
| *** camunoz has joined #openstack-infra | 04:16 | |
| *** claudiub has quit IRC | 04:17 | |
| *** links has joined #openstack-infra | 04:18 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/shade: Catch failures with particular clouds https://review.openstack.org/285882 | 04:19 |
| *** salv-orlando has joined #openstack-infra | 04:32 | |
| *** amitgandhinz has joined #openstack-infra | 04:34 | |
| *** Qiming_ has joined #openstack-infra | 04:35 | |
| *** Qiming has quit IRC | 04:35 | |
| *** david-lyle has quit IRC | 04:36 | |
| *** salv-orlando has quit IRC | 04:38 | |
| *** vgridnev has quit IRC | 04:42 | |
| *** markvoelker has joined #openstack-infra | 04:45 | |
| *** NTolerance has joined #openstack-infra | 04:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Reinstall requests on Fedora builds https://review.openstack.org/285876 | 04:47 |
| *** amitgandhinz has quit IRC | 04:47 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Prioritize venv python on hose https://review.openstack.org/285885 | 04:49 |
| *** nt has quit IRC | 04:49 | |
| *** NTolerance is now known as nt | 04:49 | |
| *** markvoelker has quit IRC | 04:50 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Prioritize venv python on host https://review.openstack.org/285885 | 04:50 |
| *** david-lyle has joined #openstack-infra | 04:50 | |
| *** thorst has joined #openstack-infra | 04:54 | |
| *** achanda has joined #openstack-infra | 04:55 | |
| *** david-lyle has quit IRC | 04:56 | |
| *** achanda has quit IRC | 05:01 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Make debootstrap cache opt-in https://review.openstack.org/285886 | 05:03 |
| *** thorst has quit IRC | 05:03 | |
| openstackgerrit | Yang Hongyang proposed openstack-infra/project-config: Add non-voting python34 job to magnum project https://review.openstack.org/285887 | 05:04 |
| openstackgerrit | Yang Hongyang proposed openstack-infra/project-config: Add voting python34 jobs for python-magnumclient https://review.openstack.org/285888 | 05:04 |
| *** Daisy_ has joined #openstack-infra | 05:08 | |
| *** fabo_ is now known as fabo | 05:16 | |
| *** yuanying has quit IRC | 05:24 | |
| *** yuanying_ has joined #openstack-infra | 05:24 | |
| *** asselin__ has quit IRC | 05:25 | |
| *** exploreshaifali has joined #openstack-infra | 05:27 | |
| *** exploreshaifali has quit IRC | 05:34 | |
| *** [1]Thelo has joined #openstack-infra | 05:36 | |
| *** salv-orlando has joined #openstack-infra | 05:36 | |
| *** Thelo has quit IRC | 05:39 | |
| *** [1]Thelo is now known as Thelo | 05:39 | |
| *** maishsk has quit IRC | 05:39 | |
| *** sridhar_ram has joined #openstack-infra | 05:42 | |
| *** sridhar_ram has quit IRC | 05:42 | |
| *** watanabe_isao has joined #openstack-infra | 05:42 | |
| *** amitgandhinz has joined #openstack-infra | 05:44 | |
| *** roxanaghe has quit IRC | 05:44 | |
| *** roxanaghe has joined #openstack-infra | 05:45 | |
| *** salv-orlando has quit IRC | 05:47 | |
| *** fawadkhaliq has joined #openstack-infra | 05:54 | |
| *** abregman has joined #openstack-infra | 05:54 | |
| *** amotoki has joined #openstack-infra | 05:56 | |
| *** gongysh has joined #openstack-infra | 05:57 | |
| *** amitgandhinz has quit IRC | 05:57 | |
| *** exploreshaifali has joined #openstack-infra | 05:58 | |
| *** achanda has joined #openstack-infra | 05:58 | |
| *** thorst has joined #openstack-infra | 06:00 | |
| *** baoli has joined #openstack-infra | 06:01 | |
| *** achanda has quit IRC | 06:02 | |
| *** Daisy has joined #openstack-infra | 06:04 | |
| *** baoli has quit IRC | 06:05 | |
| *** Daisy_ has quit IRC | 06:07 | |
| *** thorst has quit IRC | 06:07 | |
| *** achanda has joined #openstack-infra | 06:11 | |
| *** Sukhdev has joined #openstack-infra | 06:13 | |
| *** zhurong has joined #openstack-infra | 06:15 | |
| *** dguitarbite has joined #openstack-infra | 06:20 | |
| *** calebb has joined #openstack-infra | 06:20 | |
| *** dguitarbite has quit IRC | 06:20 | |
| *** BobBall_AWOL has joined #openstack-infra | 06:21 | |
| *** Daisy has quit IRC | 06:22 | |
| *** Daisy has joined #openstack-infra | 06:22 | |
| *** dguitarbite has joined #openstack-infra | 06:23 | |
| *** dguitarbite has quit IRC | 06:24 | |
| *** dguitarbite has joined #openstack-infra | 06:24 | |
| *** calebb has quit IRC | 06:25 | |
| *** BobBall_1WOL has quit IRC | 06:25 | |
| openstackgerrit | abregman proposed openstack-dev/pbr: Fail on warnings when warnerrors is True https://review.openstack.org/285897 | 06:26 |
| *** Daisy_ has joined #openstack-infra | 06:26 | |
| *** Daisy has quit IRC | 06:26 | |
| *** |-paul-| has joined #openstack-infra | 06:26 | |
| *** BobBall_1WOL has joined #openstack-infra | 06:26 | |
| openstackgerrit | Jens proposed openstack-infra/git-review: Make it possible to configure draft as default push mode https://review.openstack.org/220426 | 06:26 |
| *** BobBall_AWOL has quit IRC | 06:26 | |
| *** _nadya_ has joined #openstack-infra | 06:28 | |
| *** kushal has joined #openstack-infra | 06:28 | |
| *** _nadya_ has quit IRC | 06:29 | |
| *** zhurong has quit IRC | 06:29 | |
| *** zhurong has joined #openstack-infra | 06:30 | |
| *** Daisy has joined #openstack-infra | 06:31 | |
| *** nikhil has quit IRC | 06:32 | |
| *** korzen has joined #openstack-infra | 06:35 | |
| *** camunoz has quit IRC | 06:35 | |
| *** Daisy_ has quit IRC | 06:35 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/285901 | 06:36 |
| *** camunoz has joined #openstack-infra | 06:38 | |
| *** salv-orlando has joined #openstack-infra | 06:42 | |
| *** zhurong has quit IRC | 06:43 | |
| *** sdake has quit IRC | 06:43 | |
| *** rcernin has joined #openstack-infra | 06:45 | |
| *** ianw has quit IRC | 06:45 | |
| *** markvoelker has joined #openstack-infra | 06:46 | |
| *** rcernin has quit IRC | 06:49 | |
| *** ianw has joined #openstack-infra | 06:50 | |
| *** markvoelker has quit IRC | 06:50 | |
| *** amitgandhinz has joined #openstack-infra | 06:54 | |
| *** AJaeger has quit IRC | 06:58 | |
| *** fedexo has quit IRC | 06:59 | |
| *** sdake has joined #openstack-infra | 06:59 | |
| *** korzen has quit IRC | 07:00 | |
| *** jaosorior has joined #openstack-infra | 07:01 | |
| *** mrunge_ is now known as mrunge | 07:01 | |
| *** gildub has quit IRC | 07:01 | |
| *** AJaeger has joined #openstack-infra | 07:02 | |
| *** hayato__ has quit IRC | 07:04 | |
| openstackgerrit | abregman proposed openstack-dev/pbr: Fail on warnings when warnerrors is True https://review.openstack.org/285897 | 07:05 |
| *** thorst has joined #openstack-infra | 07:05 | |
| *** hichihara has quit IRC | 07:07 | |
| *** amitgandhinz has quit IRC | 07:08 | |
| *** yolanda has joined #openstack-infra | 07:08 | |
| *** thorst has quit IRC | 07:12 | |
| *** _nadya_ has joined #openstack-infra | 07:13 | |
| *** fawadk has joined #openstack-infra | 07:15 | |
| *** fawadkhaliq has quit IRC | 07:15 | |
| *** fawadkhaliq has joined #openstack-infra | 07:15 | |
| *** roxanaghe has quit IRC | 07:17 | |
| *** sdake has quit IRC | 07:17 | |
| *** amotoki has quit IRC | 07:19 | |
| *** fawadk has quit IRC | 07:20 | |
| *** exploreshaifali has quit IRC | 07:24 | |
| *** rcernin has joined #openstack-infra | 07:24 | |
| *** vgridnev has joined #openstack-infra | 07:27 | |
| *** vgridnev has quit IRC | 07:27 | |
| *** scheuran has joined #openstack-infra | 07:30 | |
| *** amotoki has joined #openstack-infra | 07:31 | |
| *** nikhil has joined #openstack-infra | 07:32 | |
| *** rossella_s has quit IRC | 07:32 | |
| *** rossella_s has joined #openstack-infra | 07:33 | |
| *** amotoki has quit IRC | 07:35 | |
| *** Daisy has quit IRC | 07:35 | |
| *** Daisy has joined #openstack-infra | 07:36 | |
| *** claudiub has joined #openstack-infra | 07:37 | |
| AJaeger | infra-root, it looks like we're not building any nodes anymore ;( | 07:38 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add experimental pylint job to Rally https://review.openstack.org/285915 | 07:38 |
| AJaeger | infra-root, please check http://grafana.openstack.org/dashboard/db/nodepool | 07:38 |
| yolanda | hi AJaeger, i'm back but i need to step out for some errands, for a pair of hours | 07:39 |
| yolanda | i hope that there is any other infra-root that can take care, if not i'll look when i come back | 07:39 |
| *** gongysh has quit IRC | 07:39 | |
| AJaeger | yolanda: thanks! Let's see whether jhesketh is still awake... | 07:40 |
| *** Daisy has quit IRC | 07:41 | |
| AJaeger | Looking at timing, I have a theory... | 07:41 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: gate-rally-install-devstack-centos7 job is ready for check & gate queue https://review.openstack.org/266253 | 07:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove infracloud-west https://review.openstack.org/285917 | 07:43 |
| AJaeger | jhesketh, infra-root, please check ^ | 07:43 |
| *** pcaruana has joined #openstack-infra | 07:43 | |
| *** fawadkhaliq has quit IRC | 07:43 | |
| *** sripriya has quit IRC | 07:46 | |
| AJaeger | jhesketh, infra-root: 285917 will not get any node due to these problems, you need to ninja merge it. But first please check the nodepool logs whether this is really the problem. | 07:47 |
| jhesketh | AJaeger: looking | 07:47 |
| AJaeger | jhesketh: If I look at grafana, it looks like after your change "Infra-cloud-west is currently offline" merged no new nodes where build... | 07:48 |
| AJaeger | thanks, jhesketh | 07:48 |
| jhesketh | AJaeger: hmm probably broke nodepool :-s | 07:48 |
| *** kozhukal` has joined #openstack-infra | 07:48 | |
| jhesketh | as it probably expects the credentials | 07:49 |
| *** amotoki has joined #openstack-infra | 07:49 | |
| jhesketh | AJaeger: nice pick up | 07:49 |
| *** amotoki has quit IRC | 07:49 | |
| *** gus has quit IRC | 07:50 | |
| *** Daisy has joined #openstack-infra | 07:52 | |
| *** gus has joined #openstack-infra | 07:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove infracloud-west https://review.openstack.org/285917 | 07:53 |
| *** gongysh has joined #openstack-infra | 07:55 | |
| AJaeger | jhesketh: do you want to apply the change manually to nodepool - or are you confident that puppet/ansible will pick it up? | 07:55 |
| jhesketh | I'm confident it'll pick it up... the run starts in 5min | 07:55 |
| jhesketh | I'm contemplating running it or applying it manually though | 07:55 |
| jhesketh | I'm not sure if nodepool needs a restart to reload a cloud... | 07:55 |
| * jhesketh checks | 07:56 | |
| abregman | hi. can anyone help with reviewing this: https://review.openstack.org/#/c/285897 ? | 07:57 |
| *** maishsk has joined #openstack-infra | 07:57 | |
| *** kozhukal` has quit IRC | 07:57 | |
| *** kozhukal` has joined #openstack-infra | 07:58 | |
| AJaeger | abregman: please explain a bit what your change is so that the right people can look at it. | 07:58 |
| *** kozhukal` is now known as kozhukalov` | 07:58 | |
| *** kozhukalov` has quit IRC | 07:59 | |
| abregman | sure. at the moment the docs gate for every project in OpenStack is not really working since pbr skip warnings, although warnerror is set in setup.cfg | 07:59 |
| AJaeger | jhesketh: looking at grafana, I see that some jenkins are out of nodes, others are not - and jenkins03 is deleting 50+ nodes for quite some time. Do we have some node leakage on them? | 07:59 |
| *** maishsk has quit IRC | 07:59 | |
| abregman | so this patch is fixing it by setting it to True with warnerrors = True | 07:59 |
| *** max_lobur has joined #openstack-infra | 07:59 | |
| AJaeger | abregman: So, that's a pbr change? Please say so - and talk to pbr cores. | 07:59 |
| abregman | AJaeger: hmm who are the pbr cores? | 08:00 |
| AJaeger | abregman: check it yourself - go to review.o.o, projects -> List , search for pbr and then look at at the access | 08:01 |
| abregman | AJaeger: ack. thanks | 08:01 |
| *** maishsk has joined #openstack-infra | 08:01 | |
| *** Daisy has quit IRC | 08:02 | |
| *** pasquier-s_ has quit IRC | 08:03 | |
| *** mattymo has quit IRC | 08:03 | |
| *** dkaigarodsev has quit IRC | 08:04 | |
| *** amitgandhinz has joined #openstack-infra | 08:04 | |
| jhesketh | (looks like nodepool reloads configs itself... ansible is running currently so hopefully the new change will be adopted shortly) | 08:04 |
| * jhesketh watches it | 08:04 | |
| *** dkaigarodsev has joined #openstack-infra | 08:05 | |
| *** pasquier-s has joined #openstack-infra | 08:05 | |
| *** Daisy has joined #openstack-infra | 08:05 | |
| jhesketh | AJaeger: not sure what's going on with jenkins03 there... am poking | 08:05 |
| *** mattymo has joined #openstack-infra | 08:05 | |
| *** ihrachys has joined #openstack-infra | 08:06 | |
| max_lobur | Hi Everyone! Can someone from project-config review please https://review.openstack.org/#/c/281301/ (adding the new proj, already +2) | 08:06 |
| *** jtomasek has joined #openstack-infra | 08:07 | |
| *** wwwbukolaycom has quit IRC | 08:09 | |
| *** thorst has joined #openstack-infra | 08:11 | |
| *** zeih_ has joined #openstack-infra | 08:11 | |
| *** zeih_ has quit IRC | 08:12 | |
| *** zeih_ has joined #openstack-infra | 08:12 | |
| *** kushal has quit IRC | 08:13 | |
| *** bgaifullin has joined #openstack-infra | 08:15 | |
| *** hayato_ has joined #openstack-infra | 08:15 | |
| *** bgaifullin has quit IRC | 08:15 | |
| *** [HeOS] has joined #openstack-infra | 08:15 | |
| *** hayato_ has quit IRC | 08:15 | |
| *** zhurong has joined #openstack-infra | 08:16 | |
| *** hayato_ has joined #openstack-infra | 08:16 | |
| jhesketh | AJaeger: okay new nodes building | 08:16 |
| jhesketh | AJaeger: I think nodepool was also stuck from cleaning up nodes, so I'll see if the nodes fix themselves from jenkins03 | 08:17 |
| *** amitgandhinz has quit IRC | 08:17 | |
| *** HeOS has quit IRC | 08:17 | |
| *** thorst has quit IRC | 08:17 | |
| *** exploreshaifali has joined #openstack-infra | 08:18 | |
| *** jed56 has joined #openstack-infra | 08:20 | |
| *** arxcruz has joined #openstack-infra | 08:21 | |
| *** vincentll has joined #openstack-infra | 08:22 | |
| *** jlanoux has joined #openstack-infra | 08:23 | |
| *** hashar has joined #openstack-infra | 08:24 | |
| *** dizquierdo has joined #openstack-infra | 08:25 | |
| *** ifarkas has joined #openstack-infra | 08:25 | |
| *** matrohon has joined #openstack-infra | 08:25 | |
| *** vgridnev has joined #openstack-infra | 08:25 | |
| *** achanda has quit IRC | 08:26 | |
| *** fawadkhaliq has joined #openstack-infra | 08:27 | |
| AJaeger | jhesketh: thanks, yeah, see them building now (had to step out) | 08:27 |
| *** exploreshaifali has quit IRC | 08:28 | |
| *** vgridnev has quit IRC | 08:28 | |
| *** gongysh has quit IRC | 08:29 | |
| bogdando | AJaeger, could you please accept the revert of the change https://review.openstack.org/#/c/285476/ ? | 08:29 |
| *** bgaifullin has joined #openstack-infra | 08:29 | |
| *** rcernin has quit IRC | 08:29 | |
| bogdando | it fails for unknown tox reasons :) and blocks all patches to the affected repo , like this https://review.openstack.org/#/c/281967/ ;( | 08:30 |
| *** vgridnev has joined #openstack-infra | 08:30 | |
| AJaeger | bogdando: you really didn't test docs building locally? ;( | 08:31 |
| bogdando | if you know how to fix this, I'd be very happy though | 08:31 |
| bogdando | I do, but may be I should have done it to the *clean* VM | 08:32 |
| bogdando | also I'm not sure how to trigger the exactly gate job at my env. Tox worked for me, for example | 08:33 |
| AJaeger | bogdando: invocation is " tox -evenv -- python setup.py build_sphinx" | 08:33 |
| AJaeger | your tox.ini looks very strange, compare with other repos... | 08:34 |
| bogdando | as there is no python things to test, only docs | 08:34 |
| AJaeger | bogdando: problem is you create: | 08:34 |
| AJaeger | venv create: /home/jenkins/workspace/gate-fuel-noop-fixtures-docs/.tox/2.7 | 08:35 |
| AJaeger | But it should be venv | 08:35 |
| *** amotoki has joined #openstack-infra | 08:35 | |
| bogdando | Ok, I will put -1 WIP for a while, may be I will be lucky to fix this in couple of hours ... | 08:35 |
| bogdando | thank you | 08:36 |
| *** amotoki has quit IRC | 08:36 | |
| *** achanda has joined #openstack-infra | 08:37 | |
| *** rcernin has joined #openstack-infra | 08:37 | |
| AJaeger | bogdando: https://review.openstack.org/285937 | 08:37 |
| AJaeger | bogdando: your tox.ini was far too sophisticated, I'm for small is beautiful ;) | 08:38 |
| *** fawadkhaliq has quit IRC | 08:38 | |
| bogdando | AJaeger, oh, thank you! You've saved my couple of hours | 08:38 |
| AJaeger | bogdando: change is untested ;) | 08:38 |
| bogdando | AJaeger, looks working | 08:41 |
| AJaeger | bogdando: then let's wait for Zuul to finish testing and merge it... Our systems are busy right now, might take an hour or two. | 08:43 |
| bogdando | okay, thank you | 08:43 |
| AJaeger | jhesketh: can you reach jenkins03? I cannot ;( | 08:44 |
| *** exploreshaifali has joined #openstack-infra | 08:44 | |
| *** amotoki has joined #openstack-infra | 08:45 | |
| *** gildub has joined #openstack-infra | 08:45 | |
| *** d0ugal_ has quit IRC | 08:46 | |
| *** d0ugal has joined #openstack-infra | 08:46 | |
| *** gildub has quit IRC | 08:46 | |
| *** markvoelker has joined #openstack-infra | 08:47 | |
| AJaeger | jhesketh: ah, now it's there - slow... | 08:47 |
| *** gildub has joined #openstack-infra | 08:48 | |
| *** watanabe_isao has quit IRC | 08:49 | |
| *** markvoelker has quit IRC | 08:51 | |
| jhesketh | AJaeger: hmm the load looks normal | 08:53 |
| jhesketh | loading the front page of jenkins for any masters is usually slow though | 08:53 |
| jhesketh | as for why it has so many in delete state, that's unclear | 08:54 |
| jhesketh | they are also across many clouds | 08:54 |
| *** maishsk has quit IRC | 08:55 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack/diskimage-builder: Get max retries for the interface from environment https://review.openstack.org/285942 | 08:55 |
| jhesketh | AJaeger: I might do a safe restart on it... | 08:55 |
| *** fhubik has joined #openstack-infra | 08:55 | |
| *** nmagnezi has joined #openstack-infra | 08:56 | |
| *** dingyichen has quit IRC | 08:56 | |
| *** notnownikki has joined #openstack-infra | 08:57 | |
| *** lezbar has joined #openstack-infra | 08:58 | |
| *** maishsk has joined #openstack-infra | 08:58 | |
| *** maishsk has quit IRC | 08:58 | |
| *** cody-somerville has quit IRC | 08:59 | |
| *** jordanP has joined #openstack-infra | 09:02 | |
| *** shardy has joined #openstack-infra | 09:03 | |
| *** maishsk has joined #openstack-infra | 09:04 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Add a standalone map page https://review.openstack.org/285943 | 09:05 |
| *** esikachev has joined #openstack-infra | 09:05 | |
| *** mrmartin has joined #openstack-infra | 09:06 | |
| *** achanda has quit IRC | 09:08 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add pylint, coverage, py34 to sahara-tests https://review.openstack.org/284693 | 09:08 |
| *** acabot has joined #openstack-infra | 09:09 | |
| *** _degorenko|afk is now known as degorenko | 09:10 | |
| *** amotoki has quit IRC | 09:10 | |
| xiangxinyong | hi anteaya | 09:11 |
| *** achanda has joined #openstack-infra | 09:12 | |
| *** amotoki has joined #openstack-infra | 09:13 | |
| *** Sukhdev has quit IRC | 09:13 | |
| *** amitgandhinz has joined #openstack-infra | 09:14 | |
| *** thorst has joined #openstack-infra | 09:15 | |
| *** dizquierdo has quit IRC | 09:15 | |
| *** Daisy has quit IRC | 09:15 | |
| *** nmagnezi has quit IRC | 09:15 | |
| *** Daisy has joined #openstack-infra | 09:15 | |
| *** asettle has joined #openstack-infra | 09:19 | |
| *** jistr has joined #openstack-infra | 09:20 | |
| *** Daisy has quit IRC | 09:20 | |
| *** Hal has joined #openstack-infra | 09:21 | |
| *** Hal is now known as Guest5714 | 09:21 | |
| *** derekh has joined #openstack-infra | 09:21 | |
| *** korzen has joined #openstack-infra | 09:22 | |
| *** thorst has quit IRC | 09:22 | |
| *** amotoki has quit IRC | 09:24 | |
| *** patrickeast has quit IRC | 09:24 | |
| *** patrickeast has joined #openstack-infra | 09:25 | |
| *** zeih_ has quit IRC | 09:26 | |
| *** amitgandhinz has quit IRC | 09:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Disable Liberty translations for Server projects https://review.openstack.org/285949 | 09:28 |
| *** nmagnezi has joined #openstack-infra | 09:28 | |
| *** sorantis has joined #openstack-infra | 09:31 | |
| *** zeih_ has joined #openstack-infra | 09:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:32 | |
| *** zhurong has quit IRC | 09:34 | |
| *** amotoki has joined #openstack-infra | 09:34 | |
| *** BobBall_1WOL is now known as BobBall | 09:35 | |
| *** e0ne has joined #openstack-infra | 09:43 | |
| *** sheeprine has quit IRC | 09:47 | |
| yolanda | AJaeger, i'm back. How are things ? | 09:48 |
| AJaeger | yolanda: jhesketh and I fixed nodepool to produce nodes again | 09:48 |
| AJaeger | it's slowly recovering. | 09:49 |
| yolanda | AJaeger, what was the problem? | 09:49 |
| *** kushal has joined #openstack-infra | 09:49 | |
| *** sheeprine has joined #openstack-infra | 09:49 | |
| AJaeger | 285917 fixed it - jhesketh removed infracloud-west in system-config but it was still in project-config | 09:50 |
| AJaeger | looking at http://grafana.openstack.org/dashboard/db/nodepool I'm not happy with the large number of deletes for jenkins03 | 09:50 |
| yolanda | oh, i thought there was a change from crinkle from saturday, to remove infracloud-west | 09:50 |
| AJaeger | jhesketh was considering a safe restart, not sure whether he did it | 09:50 |
| jhesketh | yeah I actually knew it would be in nodepool, I just didn't think it would cause the scheduler to stop working... I might try and make nodepool more resilient to that | 09:50 |
| jhesketh | AJaeger: yeah in progress | 09:51 |
| jhesketh | it needs to finish the building jobs | 09:51 |
| AJaeger | jhesketh: indeed... | 09:51 |
| jhesketh | yolanda: quite likely, but it never merged and I didn't look for it when debugging this morning | 09:51 |
| *** sdake has joined #openstack-infra | 09:51 | |
| *** Daisy_ has joined #openstack-infra | 09:52 | |
| *** achanda has quit IRC | 09:53 | |
| *** boris-42 has quit IRC | 09:54 | |
| *** ihrachys has quit IRC | 09:54 | |
| AJaeger | yolanda, jhesketh: I would have expected us to use all nodes by now but looking at zuul graph, there's still room left, so something is still odd (might be jenkins03) | 09:54 |
| *** e0ne has quit IRC | 09:55 | |
| yolanda | AJaeger, i'm trying to enter into jenkins03 but looks super slow | 09:56 |
| jhesketh | yolanda: I have it doing a graceful restart fyi | 09:57 |
| yolanda | jhesketh, so that change landed on the weekend https://review.openstack.org/#/c/285443/1/nodepool/nodepool.yaml | 09:57 |
| yolanda | it sees max-servers to -1 for weset | 09:57 |
| yolanda | west | 09:57 |
| *** Daisy_ has quit IRC | 09:57 | |
| yolanda | isn't enough? | 09:57 |
| jhesketh | yolanda: right, so the problem was that clouds.yaml still had infra-west in it | 09:58 |
| jhesketh | which made ansible fail | 09:58 |
| jhesketh | so no infra changes were being applied | 09:58 |
| jhesketh | so I removed infra-west from clouds.yaml, but nodepool also uses that | 09:58 |
| yolanda | oh... i didn't think that an extra entry on clouds.yaml could make it fail, if nodepool didn't use it | 09:58 |
| jhesketh | and nodepool does not gracefully handle not having the right configuration or a missing cloud | 09:58 |
| jhesketh | yolanda: well it wasn't nodepool, it was ansible doing the puppet runs.. nodepool failed due to my ansible fix | 09:58 |
| *** salv-orl_ has joined #openstack-infra | 09:59 | |
| *** ihrachys has joined #openstack-infra | 09:59 | |
| yolanda | ah i see. This should be more resilient to failures | 09:59 |
| *** thorst has joined #openstack-infra | 09:59 | |
| yolanda | jhesketh, i started some work on making that ansible puppet more resilient: https://review.openstack.org/284999 | 10:00 |
| yolanda | i can take a look at this ansible-puppet issue | 10:01 |
| jhesketh | cool | 10:02 |
| jhesketh | yolanda: already have | 10:02 |
| *** salv-orlando has quit IRC | 10:02 | |
| jhesketh | https://review.openstack.org/#/c/285882/ | 10:02 |
| jhesketh | https://github.com/ansible/ansible/pull/14699 | 10:02 |
| yolanda | ah cool | 10:02 |
| lucasagomes | hi folks, if you have a time mind taking a look at this devstack-gate patch? https://review.openstack.org/#/c/284036/ this is a fairly small (3LOC) patch that will allow Ironic to enable drivers such as pxe_ipmitool to run in the gate. Thank you | 10:03 |
| *** thorst has quit IRC | 10:06 | |
| *** sdake has quit IRC | 10:06 | |
| *** wznoinsk__ is now known as wznoinsk | 10:07 | |
| *** rguillebert has joined #openstack-infra | 10:07 | |
| yolanda | AJaeger, i'm looking at https://review.openstack.org/#/c/285722. What's the reasoning for start using ubuntu-trusty vs bare-trusty ? | 10:12 |
| *** craige has quit IRC | 10:14 | |
| AJaeger | yolanda: this is part of fungi's move from bare-trusty to ubuntu-trusty. Marked it as WIP since I want to coordinate appropriate time with him. | 10:14 |
| AJaeger | yolanda: bare-trusty only lives in RAX clouds, we want to move away from those images and use only ubuntu-trusty | 10:14 |
| yolanda | AJaeger, so the final goal is to increase the nodes available for that tests? | 10:15 |
| AJaeger | and have ubuntu-trusty in all clouds available | 10:15 |
| yolanda | ok | 10:15 |
| *** gildub has quit IRC | 10:15 | |
| AJaeger | increase nodes available - and remove single point of failure | 10:15 |
| AJaeger | End goal is consolidating the number of different images | 10:16 |
| *** dtantsur|afk is now known as dtantsur | 10:16 | |
| yolanda | i see, so a single ubuntu-trusty generated by dib, and used everywhere | 10:16 |
| AJaeger | exactly | 10:17 |
| AJaeger | yolanda: fungi has done a couple of bindep changes to move us in that direction - but those only run experimental jobs | 10:17 |
| AJaeger | I think we're ready to make the switch now but want fungi to coordinate that - therefore WIP. So, reviews welcome | 10:18 |
| yolanda | AJaeger how is status on those tests? have they good success rate? | 10:18 |
| AJaeger | yolanda: pabelanger and fungi hunted down one problem where the two nodes gave different results - and now they look fine. | 10:19 |
| AJaeger | But too early to speak about success rate. | 10:19 |
| AJaeger | They do the same now ;) | 10:19 |
| AJaeger | yolanda: https://review.openstack.org/#/q/topic:bindep-testing | 10:19 |
| *** sputnik13 has joined #openstack-infra | 10:20 | |
| *** Qiming_ has quit IRC | 10:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Add dashboard and client for Smaug https://review.openstack.org/281019 | 10:23 |
| *** daemontool_ has quit IRC | 10:23 | |
| openstackgerrit | Merged openstack-infra/project-config: Add broadview-ui project https://review.openstack.org/283101 | 10:23 |
| openstackgerrit | Merged openstack-infra/project-config: Remove tempest-lib-src experimental jobs https://review.openstack.org/284216 | 10:23 |
| openstackgerrit | Merged openstack-infra/project-config: Update zuul layout for Puppet OpenStack jobs on Trusty https://review.openstack.org/285591 | 10:23 |
| *** amitgandhinz has joined #openstack-infra | 10:24 | |
| yolanda | i had a very long review queue, but trying to unlock things... | 10:24 |
| AJaeger | yolanda: thanks a lot! | 10:24 |
| AJaeger | jhesketh: got an 502 proxy error on https://jenkins03.openstack.org/job/gate-tempest-dsvm-full/33050/ | 10:24 |
| yolanda | AJaeger, please let me know if you need reviews on something in particular, i may loose something with that week without reviews | 10:25 |
| AJaeger | yolanda: nothing critical right now, thanks | 10:26 |
| jhesketh | AJaeger: I think that's a timeout thing | 10:26 |
| AJaeger | jhesketh: got now 33050 which according to jenkins03 is finished | 10:26 |
| AJaeger | jhesketh: but looking at http://status.openstack.org/zuul/ it's not finished | 10:26 |
| jhesketh | I think it's struggling to keep up | 10:27 |
| AJaeger | (that's the top change in the gate) | 10:27 |
| openstackgerrit | Merged openstack-infra/project-config: Make tweaks to neutron-plus job https://review.openstack.org/285116 | 10:27 |
| jhesketh | so it's probably dropping things | 10:27 |
| jhesketh | darn | 10:27 |
| yolanda | jhesketh, i added a fix to the play to run jjb on hosts, to try avoid cache corruption https://review.openstack.org/284462 | 10:27 |
| openstackgerrit | Merged openstack-infra/project-config: Promote lbaas namespace driver to check, non-voting https://review.openstack.org/285456 | 10:27 |
| AJaeger | jhesketh: that keystone change failed, can you kill it somehow? | 10:27 |
| AJaeger | keystone will not merge anything today - they have a leap year bug: | 10:28 |
| AJaeger | expires_at = datetime.datetime.utcnow().replace(year=2031) | 10:28 |
| AJaeger | ValueError: day is out of range for month | 10:28 |
| AJaeger | ;) | 10:28 |
| yolanda | oh, leap years! a nightmare | 10:28 |
| *** daemontool has joined #openstack-infra | 10:29 | |
| jhesketh | AJaeger: I'm trying to load jenkins03 to take a look, but it's slow | 10:30 |
| jhesketh | I'll need to log into it to kill it gracefully which probably isn't going to be fun | 10:30 |
| openstackgerrit | Merged openstack-infra/project-config: Remove gate-openstack-ansible-commit-nv job https://review.openstack.org/285407 | 10:32 |
| openstackgerrit | Merged openstack-infra/project-config: Add a new job to run neutron_vpnaas tempest tests https://review.openstack.org/279785 | 10:33 |
| *** sputnik13 has quit IRC | 10:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove redundant job to stop running the same tests https://review.openstack.org/283869 | 10:34 |
| openstackgerrit | Merged openstack-infra/project-config: add reno jobs for oslo.db https://review.openstack.org/283765 | 10:34 |
| openstackgerrit | Dmitry Tantsur proposed openstack/requirements: Bump python-ironic-inspector-client to 1.5.0 https://review.openstack.org/285972 | 10:34 |
| *** amitgandhinz has quit IRC | 10:37 | |
| *** sfinucan has joined #openstack-infra | 10:37 | |
| *** hayato_ has quit IRC | 10:39 | |
| *** daemontool_ has joined #openstack-infra | 10:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Retire cloudv-ostf-adapter repo https://review.openstack.org/285352 | 10:39 |
| openstackgerrit | Merged openstack-infra/project-config: Remove magnumclient bandit job https://review.openstack.org/285127 | 10:39 |
| openstackgerrit | Merged openstack-infra/project-config: Add senlin-dashboard to PROJECTS https://review.openstack.org/283433 | 10:39 |
| *** daemontool has quit IRC | 10:41 | |
| *** daemontool_ is now known as daemontool | 10:42 | |
| *** salv-orlando has joined #openstack-infra | 10:42 | |
| *** salv-orlando has quit IRC | 10:43 | |
| *** salv-orl_ has quit IRC | 10:43 | |
| *** sdake has joined #openstack-infra | 10:43 | |
| *** salv-orlando has joined #openstack-infra | 10:43 | |
| openstackgerrit | Merged openstack-infra/storyboard: Add Due Dates to relevant WSME models https://review.openstack.org/284315 | 10:45 |
| openstackgerrit | Merged openstack-infra/storyboard: Update `PUT /v1/worklists/:id/items/:id` to allow assigning a due date https://review.openstack.org/284316 | 10:46 |
| *** markvoelker has joined #openstack-infra | 10:47 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add mirror server https://review.openstack.org/238414 | 10:48 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Used the mirror server for the centos image and git repos https://review.openstack.org/285257 | 10:48 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use instack-qcow2 from cache if available and appropriate https://review.openstack.org/285259 | 10:48 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Upload the instack qcow image after periodic jobs https://review.openstack.org/285258 | 10:48 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job https://review.openstack.org/229789 | 10:48 |
| *** markvoelker has quit IRC | 10:52 | |
| *** yamamoto_ has quit IRC | 10:53 | |
| *** sdake has quit IRC | 10:53 | |
| *** mrmartin has quit IRC | 10:53 | |
| *** andreykurilin__ has joined #openstack-infra | 10:54 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Archive cards instead of deleting them https://review.openstack.org/285392 | 10:54 |
| *** sdague has joined #openstack-infra | 10:55 | |
| *** daemontool__ has joined #openstack-infra | 10:56 | |
| *** flaper87 has quit IRC | 10:56 | |
| *** flaper87 has joined #openstack-infra | 10:56 | |
| *** daemontool has quit IRC | 10:57 | |
| *** korzen_ has joined #openstack-infra | 10:58 | |
| *** korzen has quit IRC | 10:58 | |
| *** sam_wan has quit IRC | 10:59 | |
| *** amotoki has quit IRC | 11:00 | |
| *** amotoki has joined #openstack-infra | 11:03 | |
| *** thorst has joined #openstack-infra | 11:06 | |
| *** exploreshaifali has quit IRC | 11:06 | |
| *** sdake has joined #openstack-infra | 11:06 | |
| *** tpsilva has joined #openstack-infra | 11:09 | |
| *** craige has joined #openstack-infra | 11:09 | |
| *** thorst has quit IRC | 11:11 | |
| *** amotoki has quit IRC | 11:11 | |
| *** sdake has quit IRC | 11:16 | |
| *** Qiming has joined #openstack-infra | 11:16 | |
| Qiming | path submitted 7 hours ago, it is still not triggering Ci, any hints? | 11:18 |
| Qiming | s/path/patch | 11:18 |
| ttx | Qiming: the gate is pretty loaded right now, there is a long backlog | 11:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting python34 job to magnum project https://review.openstack.org/285887 | 11:20 |
| Qiming | thanks ttx, will find some other things to work on, :) | 11:20 |
| ttx | Current estimates on the check queue ~ 5 hours | 11:21 |
| openstackgerrit | Merged openstack-infra/project-config: Add voting python34 jobs for python-magnumclient https://review.openstack.org/285888 | 11:21 |
| sdague | heh | 11:21 |
| sdague | leap day | 11:21 |
| ttx | sdague: fun heh | 11:21 |
| *** asselin__ has joined #openstack-infra | 11:22 | |
| *** fhubik is now known as fhubik_brb | 11:22 | |
| ttx | This morning I almost tweeted "make use of the extra day to get things in before th FF crunch, then I looked up zuul status and decided it wasn't the best day to do that after all | 11:22 |
| sdague | no... not even that | 11:22 |
| sdague | keystone doesn't do leap days - http://logs.openstack.org/78/284778/3/gate/gate-keystone-python27/f6b7c0a/ | 11:22 |
| ttx | sdague: right, I just tweeted about that one instead :) | 11:23 |
| sdague | ah, ok, I haven't looked at the twitters yet | 11:23 |
| *** daemontool_ has joined #openstack-infra | 11:24 | |
| * ttx likes leap day blunders | 11:24 | |
| *** achanda has joined #openstack-infra | 11:24 | |
| ttx | that one just sounds perfectly alright (except it starts failing as you get closer to 2031 but meh) | 11:24 |
| sdague | has anyone posted that fix yet? | 11:24 |
| ttx | sdague: yolanda AJaeger and jhesketh were discussing it earlier | 11:25 |
| ttx | They may not have | 11:25 |
| jhesketh | I wasn't looking for a fix sorry | 11:25 |
| *** asselin__ has quit IRC | 11:26 | |
| sdague | ok, I'll post the fix | 11:27 |
| ttx | sdague: looks like the answer is "no" | 11:27 |
| *** daemontool__ has quit IRC | 11:27 | |
| * ttx lunches | 11:28 | |
| yolanda | i haven't sent any fix | 11:28 |
| *** achanda has quit IRC | 11:29 | |
| *** dims has joined #openstack-infra | 11:29 | |
| *** yamamoto has joined #openstack-infra | 11:31 | |
| *** maishsk has quit IRC | 11:32 | |
| *** dims has quit IRC | 11:32 | |
| *** fhubik_brb is now known as fhubik | 11:32 | |
| *** dims has joined #openstack-infra | 11:32 | |
| *** rossella_s has quit IRC | 11:32 | |
| max_lobur | sdague: Hi! Could you take a look please https://review.openstack.org/#/c/281301/ (adding a proj to project-config) | 11:32 |
| *** bgaifullin has quit IRC | 11:32 | |
| *** rossella_s has joined #openstack-infra | 11:33 | |
| *** bgaifullin has joined #openstack-infra | 11:33 | |
| *** yamamoto_ has joined #openstack-infra | 11:33 | |
| *** amitgandhinz has joined #openstack-infra | 11:33 | |
| *** korzen has joined #openstack-infra | 11:34 | |
| *** korzen_ has quit IRC | 11:35 | |
| *** sorantis has quit IRC | 11:35 | |
| *** maishsk has joined #openstack-infra | 11:35 | |
| *** otsuka_ has joined #openstack-infra | 11:36 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** Daisy has joined #openstack-infra | 11:36 | |
| sdague | yolanda / jhesketh did zuul just flip out an not do anythign over the weekend. the nodepool pattern + the check queue looks like what happens when work doesn't process for a long time then flushes all at once | 11:37 |
| jhesketh | sdague: nodepool was offline due to infra-cloud credentials being unavailable | 11:38 |
| jhesketh | I should have picked up on it earlier :-( | 11:38 |
| AJaeger | sdague: the keystone fix is open for grabs ;) | 11:38 |
| sdague | AJaeger: I pushed it | 11:38 |
| sdague | jhesketh: ah | 11:39 |
| sdague | jhesketh: sounds like a good enhancement :) | 11:39 |
| *** otsuka has quit IRC | 11:39 | |
| sdague | especially given the vast number of clouds we get under it now | 11:40 |
| dmellado | Hi is there any cause about the CI being *really* slow today? | 11:40 |
| AJaeger | dmellado: it has a long backlog, have patience, please | 11:40 |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Fix hostname entries in glean https://review.openstack.org/285433 | 11:40 |
| dmellado | AJaeger: ack, I was thinking about if there was something broken, as it has been waiting way longer than almost ever xD | 11:40 |
| AJaeger | sdague: nova did not publish any tarballs or docs for a week due to constraints changes, please see http://lists.openstack.org/pipermail/openstack-infra/2016-February/003932.html | 11:41 |
| dmellado | and I was seeing every gate but one still in 'queued' state | 11:41 |
| dmellado | but thanks for letting me know AJaeger ;) | 11:41 |
| sdague | dmellado: there was something broken over the weekend, now there is a lot flushing | 11:42 |
| dmellado | sdague: I see, thanks! | 11:42 |
| *** ldnunes has joined #openstack-infra | 11:43 | |
| AJaeger | sdague: downtime was only this morning for 6 hours or so... | 11:43 |
| *** Daisy has quit IRC | 11:46 | |
| sdague | AJaeger: right, but we're coming up on M3, and we're already way over capacity | 11:46 |
| sdague | I'll be surprised if the check queue is less than 6 hours delay this entire week | 11:46 |
| *** Daisy has joined #openstack-infra | 11:46 | |
| *** bgaifullin has quit IRC | 11:47 | |
| *** exploreshaifali has joined #openstack-infra | 11:47 | |
| *** amitgandhinz has quit IRC | 11:47 | |
| sdague | AJaeger: you have WIP https://review.openstack.org/#/c/285826 this | 11:49 |
| AJaeger | sdague: Yes, wanted to give lifeless, Nakato and you a chance to review ;) | 11:50 |
| sdague | can you unWIP it and I'll approve it | 11:50 |
| AJaeger | done | 11:50 |
| *** Daisy has quit IRC | 11:51 | |
| sdague | I think on stuff like that keeping it mergable is good. Few nova cores are going to weigh in on a bit like that that don't understand what's going on | 11:51 |
| AJaeger | sdague: I wanted review from outside of nova ;) | 11:52 |
| * AJaeger removed -1 from both nova and glance changes now | 11:52 | |
| *** cody-somerville has joined #openstack-infra | 11:53 | |
| *** bgaifullin has joined #openstack-infra | 11:53 | |
| sdague | yolanda: is it possible to kill all the kolla changes in check queue. Looking at sdake's email they all will fail | 11:54 |
| AJaeger | sdague: could you remove two more constraint related changes: https://review.openstack.org/285720 and https://review.openstack.org/285493 , please? | 11:54 |
| AJaeger | s/remove/review/ | 11:54 |
| sdague | and the kolla rush isn't helping the capacity | 11:54 |
| yolanda | sdague i can be possible yes | 11:54 |
| yolanda | checking | 11:54 |
| sdague | well except his fix for it - https://review.openstack.org/#/c/285875/ | 11:55 |
| AJaeger | sdague: now I get it - the kolla change did not merge yet ;( | 11:55 |
| *** marcusvrn_ has joined #openstack-infra | 11:55 | |
| sdague | http://lists.openstack.org/pipermail/openstack-dev/2016-February/087738.html | 11:57 |
| sdague | AJaeger: the kolla change hasn't even gotten tests run on it yet | 11:57 |
| sdague | because all the other kolla changes rushing the world | 11:57 |
| AJaeger | ;( | 11:58 |
| *** htruta` is now known as htruta | 11:58 | |
| sdague | and other changes | 11:58 |
| sdague | it's not just them | 11:58 |
| sdague | but we are just way over capacity | 11:59 |
| yolanda | i killled one kolla service, the others have not even reached jenkins | 11:59 |
| *** skolekonov_ has joined #openstack-infra | 12:00 | |
| sdague | cool | 12:00 |
| sdague | as long as https://review.openstack.org/#/c/285875/ stays in the queue to get tests run on | 12:01 |
| sdague | that's their important fix | 12:01 |
| skolekonov_ | Hi folks. Is jenkins03.openstack.org works ok? I see a long chain in gate pipeline and I can't open a job on this jenkins | 12:01 |
| skolekonov_ | *working | 12:02 |
| AJaeger | skolekonov_: jhesketh is shutting down jenkins03 currently | 12:02 |
| AJaeger | skolekonov_: we have a long backlog, please have patience | 12:02 |
| skolekonov_ | AJaeger, ok, thanks | 12:03 |
| jhesketh | yeah jenkins03 is clearing it's queues | 12:03 |
| *** zeih_ has quit IRC | 12:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix pylint to work with constraints https://review.openstack.org/285720 | 12:03 |
| AJaeger | yolanda, sdague, jhesketh : What about sending out a status notice like "Our systems have a long backlog, please have patience and avoid rechecks." | 12:03 |
| jhesketh | AJaeger: yep, good idea | 12:04 |
| jhesketh | #status notice Infra currently has a long backlog. Please be patient and where possible avoid rechecks while it catches up. | 12:05 |
| openstackstatus | jhesketh: sending notice | 12:05 |
| openstackgerrit | Merged openstack-infra/project-config: Remove cinder constraints jobs https://review.openstack.org/285493 | 12:05 |
| *** lucasagomes is now known as lucas-hungry | 12:06 | |
| -openstackstatus- NOTICE: Infra currently has a long backlog. Please be patient and where possible avoid rechecks while it catches up. | 12:06 | |
| *** sorantis has joined #openstack-infra | 12:07 | |
| openstackstatus | jhesketh: finished sending notice | 12:07 |
| *** jpr has joined #openstack-infra | 12:08 | |
| *** rfolco has joined #openstack-infra | 12:08 | |
| *** amrith is now known as _amrith_ | 12:11 | |
| odyssey4me | AJaeger yolanda we have a time sensitive review (adding new repositories) which needs attention - could you please take a look at it when you have a moment? | 12:12 |
| bgaifullin | Hi. Folks please review https://review.openstack.org/#/c/279526/ and merge if it is OK | 12:12 |
| *** otsuka has joined #openstack-infra | 12:13 | |
| *** fhubik is now known as fhubik_brb | 12:14 | |
| *** bgaifullin has quit IRC | 12:15 | |
| *** otsuka_ has quit IRC | 12:15 | |
| *** bgaifullin has joined #openstack-infra | 12:16 | |
| *** bgaifullin has quit IRC | 12:18 | |
| *** zeih has joined #openstack-infra | 12:18 | |
| *** daemontool_ has quit IRC | 12:18 | |
| openstackgerrit | xiangxinyong proposed openstack-infra/project-config: Add smaug-dashboard and python-smaugclient to gerritbot https://review.openstack.org/286005 | 12:20 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 12:20 |
| *** daemontool has joined #openstack-infra | 12:20 | |
| *** khappone has quit IRC | 12:21 | |
| *** khappone has joined #openstack-infra | 12:21 | |
| *** baoli has joined #openstack-infra | 12:22 | |
| *** baoli has quit IRC | 12:22 | |
| kozhukalov | AJaeger: could you please take a look at https://review.openstack.org/#/c/260455/ once again (Vitaly has answered Dmitry's question)? | 12:23 |
| *** otsuka_ has joined #openstack-infra | 12:23 | |
| *** hashar is now known as hasharLunch | 12:25 | |
| *** achanda has joined #openstack-infra | 12:26 | |
| *** otsuka has quit IRC | 12:26 | |
| *** zeih has quit IRC | 12:28 | |
| *** amotoki has joined #openstack-infra | 12:30 | |
| *** zeih has joined #openstack-infra | 12:30 | |
| *** achanda has quit IRC | 12:30 | |
| *** bgaifullin has joined #openstack-infra | 12:31 | |
| *** gordc has joined #openstack-infra | 12:32 | |
| *** thorst has joined #openstack-infra | 12:32 | |
| *** daemontool has quit IRC | 12:32 | |
| *** jaosorior has quit IRC | 12:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Create openstack/packetary-specs repository https://review.openstack.org/279526 | 12:34 |
| *** jaosorior has joined #openstack-infra | 12:34 | |
| *** daemontool has joined #openstack-infra | 12:34 | |
| *** smarcet has joined #openstack-infra | 12:35 | |
| *** rhallisey has joined #openstack-infra | 12:36 | |
| smarcet | fungi: mordred: clarkb: morning guys, please review https://review.openstack.org/#/c/285475/ when u get a chance :) | 12:37 |
| *** otsuka has joined #openstack-infra | 12:37 | |
| *** exploreshaifali has quit IRC | 12:38 | |
| *** sorantis has quit IRC | 12:39 | |
| *** sorantis has joined #openstack-infra | 12:40 | |
| *** otsuka_ has quit IRC | 12:41 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new projects for the OSA role break out https://review.openstack.org/284512 | 12:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new projects for the OSA role break out https://review.openstack.org/284512 | 12:43 |
| *** amitgandhinz has joined #openstack-infra | 12:43 | |
| *** amotoki has quit IRC | 12:45 | |
| *** sheeprine has quit IRC | 12:45 | |
| *** sheeprine has joined #openstack-infra | 12:46 | |
| *** markvoelker has joined #openstack-infra | 12:48 | |
| *** fhubik_brb is now known as fhubik | 12:52 | |
| *** markvoelker has quit IRC | 12:53 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new projects for the OSA role break out https://review.openstack.org/284512 | 12:53 |
| *** pcaruana has quit IRC | 12:53 | |
| jamespage | AJaeger, hello - we're ready to move forward on the migration of openstack charm development under https://review.openstack.org/#/c/232705 | 12:54 |
| jamespage | any chance we can get that moving today? | 12:54 |
| *** baoli has joined #openstack-infra | 12:55 | |
| *** daemontool has quit IRC | 12:55 | |
| *** coreyob has joined #openstack-infra | 12:55 | |
| *** daemontool has joined #openstack-infra | 12:55 | |
| *** ldnunes has quit IRC | 12:56 | |
| *** daemontool has quit IRC | 12:56 | |
| *** jpr has quit IRC | 12:56 | |
| *** julim has joined #openstack-infra | 12:56 | |
| *** amitgandhinz has quit IRC | 12:57 | |
| *** rlandy has joined #openstack-infra | 12:57 | |
| jhesketh | yolanda: I might delete all the nodes on jenkins03 from nodepool | 12:58 |
| jhesketh | then restart jenkins03 forcefully | 12:58 |
| *** exploreshaifali has joined #openstack-infra | 12:58 | |
| jhesketh | the soft-restart doesn't appear to be working | 12:58 |
| *** daemontool has joined #openstack-infra | 12:59 | |
| *** ldnunes has joined #openstack-infra | 12:59 | |
| *** amotoki has joined #openstack-infra | 13:02 | |
| *** zeih has quit IRC | 13:02 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 13:03 |
| *** andreykurilin__ has quit IRC | 13:03 | |
| *** mrmartin has joined #openstack-infra | 13:03 | |
| yolanda | jhesketh, need any help ? | 13:04 |
| jhesketh | nope, done already | 13:04 |
| yolanda | also, i think there is a play for jenkins restarts | 13:04 |
| jhesketh | had to kill the jenkins process | 13:04 |
| *** alivigni has joined #openstack-infra | 13:05 | |
| jhesketh | yolanda: yes, I was using that for the soft restart | 13:05 |
| jhesketh | which at least stopped it from getting more jobs while it was finishing off the ones it had... but it had completed all of that and was still not very responsive | 13:05 |
| jhesketh | and it was hogging nodeds | 13:05 |
| jhesketh | *nodes | 13:05 |
| *** lucas-hungry is now known as lucasagomes | 13:06 | |
| *** dtardivel has joined #openstack-infra | 13:07 | |
| *** pcaruana has joined #openstack-infra | 13:08 | |
| *** sheeprine has quit IRC | 13:09 | |
| *** sheeprine has joined #openstack-infra | 13:10 | |
| *** ociuhandu has quit IRC | 13:11 | |
| *** yaume has joined #openstack-infra | 13:12 | |
| hasharLunch | if soft restart does not work, you might want to get a thread dump to identify the lock | 13:13 |
| *** hasharLunch is now known as hashar | 13:13 | |
| hashar | on my setup I had multiple issues with the IRC notification plugins. Of course YMMV | 13:14 |
| *** rfolco_ has joined #openstack-infra | 13:14 | |
| *** toabctl has quit IRC | 13:14 | |
| *** dkranz has joined #openstack-infra | 13:15 | |
| jhesketh | okay that appears to have unstuck jenkins03 | 13:15 |
| *** rfolco has quit IRC | 13:16 | |
| *** trash has quit IRC | 13:16 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Archive cards instead of deleting them https://review.openstack.org/285392 | 13:17 |
| *** otsuka has quit IRC | 13:18 | |
| *** otsuka has joined #openstack-infra | 13:18 | |
| *** links has quit IRC | 13:18 | |
| *** toabctl has joined #openstack-infra | 13:18 | |
| *** otsuka has left #openstack-infra | 13:18 | |
| *** trash has joined #openstack-infra | 13:20 | |
| *** dprince has joined #openstack-infra | 13:23 | |
| *** |-paul-| has quit IRC | 13:23 | |
| *** fhubik is now known as fhubik_brb | 13:23 | |
| *** _amrith_ is now known as amrith | 13:23 | |
| *** edmondsw has joined #openstack-infra | 13:25 | |
| *** skolekonov_ has quit IRC | 13:26 | |
| *** kushal has quit IRC | 13:27 | |
| *** exploreshaifali has quit IRC | 13:28 | |
| *** dmellado is now known as dmellado|lunch | 13:28 | |
| *** dmellado|lunch is now known as dmellado | 13:28 | |
| *** fhubik_brb is now known as fhubik | 13:29 | |
| *** sheeprine has quit IRC | 13:30 | |
| *** sheeprine has joined #openstack-infra | 13:30 | |
| *** regXboi has joined #openstack-infra | 13:31 | |
| *** ociuhandu has joined #openstack-infra | 13:31 | |
| *** amrith is now known as _amrith_ | 13:33 | |
| openstackgerrit | Oleksandr Kyrylchuk proposed openstack-infra/project-config: Remove cloudv-ostf-adapter from Infrastructure Systems https://review.openstack.org/285399 | 13:34 |
| *** annegentle has joined #openstack-infra | 13:35 | |
| *** jtomasek_ has joined #openstack-infra | 13:35 | |
| *** keedya has joined #openstack-infra | 13:35 | |
| *** amotoki has quit IRC | 13:39 | |
| *** tongli has joined #openstack-infra | 13:40 | |
| *** jpr has joined #openstack-infra | 13:40 | |
| *** fawadkhaliq has joined #openstack-infra | 13:43 | |
| *** aysyd has joined #openstack-infra | 13:45 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: node deletion delay is now configurable https://review.openstack.org/245220 | 13:47 |
| *** markvoelker has joined #openstack-infra | 13:47 | |
| *** baoli has quit IRC | 13:47 | |
| *** sambetts has joined #openstack-infra | 13:48 | |
| *** daemontool has quit IRC | 13:48 | |
| *** baoli has joined #openstack-infra | 13:48 | |
| *** aysyd has quit IRC | 13:52 | |
| *** amotoki has joined #openstack-infra | 13:53 | |
| *** achanda has joined #openstack-infra | 13:53 | |
| *** daemontool has joined #openstack-infra | 13:54 | |
| *** aysyd has joined #openstack-infra | 13:55 | |
| *** jtomasek_ has quit IRC | 13:55 | |
| *** daemontool_ has joined #openstack-infra | 13:55 | |
| *** links has joined #openstack-infra | 13:59 | |
| *** daemontool has quit IRC | 13:59 | |
| *** bswartz has joined #openstack-infra | 14:01 | |
| openstackgerrit | Vincent proposed openstack-infra/project-config: add Synergy project https://review.openstack.org/283157 | 14:02 |
| max_lobur | Hi Everyone! Can someone from project-config review please https://review.openstack.org/#/c/281301/ (adding the new proj, already +2) | 14:04 |
| tobiash_ | hi, just two small questions about the zuul-merger | 14:05 |
| tobiash_ | is it processing the merges for different projects single threaded or in parallel? | 14:05 |
| tobiash_ | how many instances do you have on the openstack ci? | 14:06 |
| *** otsuka has joined #openstack-infra | 14:06 | |
| tobiash_ | our (currently single instance) zuul-merger seems to get slow when rebasing larger patch series | 14:07 |
| *** otsuka has quit IRC | 14:07 | |
| *** otsuka has joined #openstack-infra | 14:08 | |
| *** calebb has joined #openstack-infra | 14:08 | |
| odyssey4me | AJaeger I see your email about a recheck automatically doing the check on the latest base. Is that really so? I'm under the impression that for the check queue it does a checkout, whereas in the gate queue it does a cherry-pick on top of the latest HEAD? | 14:11 |
| *** mriedem has joined #openstack-infra | 14:11 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list https://review.openstack.org/286062 | 14:12 |
| *** zz_dimtruck is now known as dimtruck | 14:12 | |
| *** achanda has quit IRC | 14:13 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 14:13 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Add experimental Manila LVM job with minimal services https://review.openstack.org/285828 | 14:14 |
| AJaeger | odyssey4me: AFAIK we always merge the changes | 14:15 |
| AJaeger | That's why a change that would create a merge-conflict when rebased will not get tested in the check queue at all... | 14:15 |
| *** zeih has joined #openstack-infra | 14:16 | |
| *** links has quit IRC | 14:16 | |
| *** boris-42 has joined #openstack-infra | 14:16 | |
| AJaeger | odyssey4me: rebasing on master is only usefull if you have merge-conflicts. OTherwise it's not needed, it will make comparison between patchsets really hard | 14:17 |
| dhellmann | fungi : I have another pypi upload failure we didn't notice from a few weeks ago. Do you think you'll have time to retry it for us today? | 14:17 |
| odyssey4me | AJaeger yes, that's why I'm interested - this is good feedback and something that I can share with our team who is under a different impression | 14:17 |
| dhellmann | fungi : http://logs.openstack.org/86/8640c9f08ae1700bd08fe53970532373c8345764/release/python-ironic-inspector-client-pypi-both-upload/7ae83a1/console.html.gz | 14:17 |
| *** calebb has quit IRC | 14:18 | |
| AJaeger | odyssey4me: In the mail, the change did not merge - so it should have asked to rebase *on top of the unmerged change*. | 14:18 |
| *** ajmiller has joined #openstack-infra | 14:18 | |
| *** Qiming has quit IRC | 14:19 | |
| *** Qiming has joined #openstack-infra | 14:19 | |
| *** tiswanso has joined #openstack-infra | 14:20 | |
| kozhukalov | AJaeger: could you please take a look once again https://review.openstack.org/#/c/260455/ ? | 14:21 |
| *** dizquierdo has joined #openstack-infra | 14:21 | |
| AJaeger | kozhukalov: what is that one about, please give details | 14:21 |
| kozhukalov | moving javascript code from fuel-web to a separate repo fuel-ui | 14:22 |
| kozhukalov | AJaeger: we have answered Dmitry's question about tests | 14:22 |
| AJaeger | kozhukalov: will review next time I go through the queue... | 14:22 |
| kozhukalov | nice, thanks a lot | 14:23 |
| *** otsuka has left #openstack-infra | 14:23 | |
| *** denisra_ has quit IRC | 14:25 | |
| *** ajmiller has quit IRC | 14:25 | |
| *** denisra_ has joined #openstack-infra | 14:25 | |
| *** jcoufal has joined #openstack-infra | 14:26 | |
| *** andymaier has joined #openstack-infra | 14:26 | |
| *** vgridnev has quit IRC | 14:26 | |
| *** vgridnev has joined #openstack-infra | 14:26 | |
| *** mrmartin has quit IRC | 14:28 | |
| *** jsavak has joined #openstack-infra | 14:30 | |
| korzen | Hi, is there anyone able to get my patch +2? https://review.openstack.org/#/c/250215 I've got +1 from armax, ihrachys and sc68cal, it is the DVR multinode grenade job for Neutron upgrade tests | 14:30 |
| korzen | sdague ^ ? | 14:31 |
| korzen | anteaya? | 14:31 |
| *** kevinsho_ has joined #openstack-infra | 14:31 | |
| *** eharney has joined #openstack-infra | 14:34 | |
| *** sdake has joined #openstack-infra | 14:35 | |
| *** asselin__ has joined #openstack-infra | 14:35 | |
| *** dimtruck is now known as zz_dimtruck | 14:37 | |
| anteaya | infra-root: looks like jenkins 02 and 04 are off doing their own thing | 14:37 |
| anteaya | would be great to have them playing along | 14:37 |
| *** pcaruana has quit IRC | 14:38 | |
| anteaya | and bluebox | 14:39 |
| anteaya | 's error node launch attempts graph looks fairly full: http://grafana.openstack.org/dashboard/db/nodepool-blue-box | 14:39 |
| anteaya | mordred: ^ | 14:39 |
| *** zz_dimtruck is now known as dimtruck | 14:39 | |
| annegentle | what's the date for cutoff of summit atc invites? | 14:39 |
| annegentle | cutoff date would be simpler, heh | 14:39 |
| anteaya | annegentle: fungi would know that | 14:40 |
| *** asselin__ has quit IRC | 14:40 | |
| annegentle | anteaya: figures since fungi is probably running the script | 14:40 |
| anteaya | so far I haven't seen him online yet today, should be about soon | 14:40 |
| anteaya | annegentle: yup | 14:40 |
| annegentle | anteaya: ok, thanks | 14:40 |
| anteaya | I believe he has two meetings in about 20 minutes | 14:40 |
| anteaya | annegentle: np | 14:40 |
| keedya | Good Morning all, can I get some reviews on https://review.openstack.org/#/c/284827/ ? | 14:40 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:42 | |
| pabelanger | morning | 14:42 |
| *** x00350071 has joined #openstack-infra | 14:42 | |
| max_lobur | and here please :) https://review.openstack.org/#/c/281301/ needs one more +2 | 14:43 |
| *** mrmartin has joined #openstack-infra | 14:44 | |
| pabelanger | AJaeger: yolanda: I talked a little to fungi last week about bindep. I think we might be ready for some jobs. Will wait for fungi to take point, but haven't seen anything other issues | 14:44 |
| yolanda | anteaya, will take a look at jenkins | 14:44 |
| *** xiangxinyong has quit IRC | 14:45 | |
| AJaeger | pabelanger, fungi: I just put together a small etherpad with reviews: https://etherpad.openstack.org/p/bindep | 14:46 |
| openstackgerrit | Sergey Belous proposed openstack-infra/project-config: do not merge, test job that install os-vif https://review.openstack.org/286085 | 14:46 |
| *** dimtruck is now known as zz_dimtruck | 14:47 | |
| pabelanger | AJaeger: cool, looking | 14:48 |
| *** woodster_ has joined #openstack-infra | 14:49 | |
| AJaeger | pabelanger, fungi: I'd like fungi to say which jobs to merge first (and review whether he's happy with changes as they are) | 14:49 |
| *** _amrith_ is now known as amrith | 14:49 | |
| *** amitgandhinz has joined #openstack-infra | 14:49 | |
| anteaya | yolanda: thank you | 14:49 |
| yolanda | we also had problems with jenkins03 before | 14:50 |
| anteaya | yeah I saw on the graph | 14:50 |
| anteaya | thanks for getting jenkins 03 back in the game! | 14:50 |
| *** pradk has joined #openstack-infra | 14:50 | |
| AJaeger | anteaya: which graph are you lookin at? | 14:50 |
| anteaya | AJaeger: http://grafana.openstack.org/dashboard/db/nodepool | 14:51 |
| anteaya | jenkins 03 has some history there | 14:51 |
| AJaeger | ok - that's what I did earlier as well... | 14:51 |
| anteaya | great, haven't read backscroll yet | 14:51 |
| pabelanger | AJaeger: agreed | 14:51 |
| AJaeger | you're right - 2 and 4 have a straight line, didn't notice earlier ;(((( | 14:52 |
| *** pcaruana has joined #openstack-infra | 14:52 | |
| lucasagomes | hi folks, if you have a time mind taking a look at this devstack-gate patch? https://review.openstack.org/#/c/284036/ this is a fairly small (3LOC) patch that will allow Ironic to enable drivers such as pxe_ipmitool to run in the gate. Thank you | 14:53 |
| lucasagomes | sdague, sorry to ping you directly, but if you have a min later on today ^ | 14:53 |
| AJaeger | pabelanger: thanks for review | 14:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for project-config https://review.openstack.org/285722 | 14:54 |
| *** asselin has joined #openstack-infra | 14:54 | |
| *** yamahata_ has quit IRC | 14:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for project-config https://review.openstack.org/285722 | 14:55 |
| yolanda | jenkins04 is totally broken, cannot be restarted from the play | 14:55 |
| AJaeger | pabelanger: now fixed ^ | 14:56 |
| *** doug-fish has joined #openstack-infra | 14:56 | |
| *** thiagop has joined #openstack-infra | 14:56 | |
| pabelanger | AJaeger: \o/ | 14:56 |
| AJaeger | yolanda: ;( | 14:57 |
| AJaeger | yolanda: and 02? | 14:57 |
| yolanda | it's restarting now | 14:57 |
| AJaeger | great! | 14:57 |
| *** sdake has quit IRC | 14:58 | |
| AJaeger | pabelanger: did I list all changes on the etherpad or are you aware of more? | 14:59 |
| anteaya | AJaeger: oh that's okay, thanks for getting 03 back online | 15:00 |
| anteaya | yolanda: thanks for taking a look at them and getting 02 coming back | 15:01 |
| pabelanger | AJaeger: no, I think we are good. | 15:02 |
| *** lennyb__ is now known as lennyb | 15:03 | |
| yolanda | they should be online now | 15:03 |
| anteaya | thank you | 15:03 |
| yolanda | i'm going to check the remaining nodepool nodes there, as the plays did fail due to 502 error | 15:03 |
| anteaya | thank you | 15:03 |
| *** matbu is now known as perrin | 15:05 | |
| *** kgiusti has joined #openstack-infra | 15:05 | |
| *** perrin is now known as traffyy | 15:05 | |
| *** rbrndt_ has joined #openstack-infra | 15:05 | |
| *** traffyy is now known as matbu | 15:05 | |
| *** vgridnev has quit IRC | 15:05 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Install dstat after we set the fedora mirror https://review.openstack.org/286101 | 15:06 |
| *** jpr has quit IRC | 15:08 | |
| *** kushal has joined #openstack-infra | 15:11 | |
| *** asselin__ has joined #openstack-infra | 15:12 | |
| *** annegentle has quit IRC | 15:12 | |
| *** sorantis has quit IRC | 15:13 | |
| *** vgridnev has joined #openstack-infra | 15:13 | |
| *** asselin has quit IRC | 15:15 | |
| *** ajmiller has joined #openstack-infra | 15:16 | |
| *** vgridnev has quit IRC | 15:16 | |
| *** mtanino has joined #openstack-infra | 15:16 | |
| *** asselin has joined #openstack-infra | 15:18 | |
| *** mestery has quit IRC | 15:19 | |
| *** asselin__ has quit IRC | 15:19 | |
| *** mestery has joined #openstack-infra | 15:19 | |
| *** asselin has quit IRC | 15:22 | |
| *** asselin has joined #openstack-infra | 15:23 | |
| *** vgridnev has joined #openstack-infra | 15:25 | |
| *** vgridnev has quit IRC | 15:26 | |
| anteaya | yolanda: can you take another look at jenkins 02? currently it has over 80 ready nodes but none in use | 15:26 |
| yolanda | ok | 15:26 |
| anteaya | thanks | 15:26 |
| *** fabio_ has joined #openstack-infra | 15:27 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/project-config: Adjust acls for fuel-noop-fixtures https://review.openstack.org/286109 | 15:27 |
| *** MarkAtwood has joined #openstack-infra | 15:28 | |
| *** mrmartin has quit IRC | 15:28 | |
| *** vgridnev has joined #openstack-infra | 15:28 | |
| *** kingia has quit IRC | 15:28 | |
| BobBall | Looking at the stable/liberty test run at http://logs.openstack.org/41/281341/2/check/gate-tempest-dsvm-full/48bb5a4/logs/devstacklog.txt.gz can someone explain why devstack logs several repos as using master (e.g. "2016-02-26 09:43:17.524 | ++ GITBRANCH["ceilometermiddleware"]=master") rather than using stable/liberty? | 15:28 |
| *** kingia has joined #openstack-infra | 15:28 | |
| *** mriedem is now known as mriedem_meeting | 15:30 | |
| *** zz_dimtruck is now known as dimtruck | 15:30 | |
| *** sripriya has joined #openstack-infra | 15:31 | |
| anteaya | BobBall: if I look at that I see all the services using stable/liberty and all the clients and libs using master | 15:32 |
| anteaya | http://logs.openstack.org/41/281341/2/check/gate-tempest-dsvm-full/48bb5a4/logs/devstacklog.txt.gz#_2016-02-26_09_43_17_507 | 15:32 |
| openstackgerrit | Francesco Longo proposed openstack-infra/project-config: Added IoTronic project. https://review.openstack.org/286113 | 15:33 |
| anteaya | I believe the stable jobs use the stable/foo branch of the services and the master branch of clients and libs by design | 15:33 |
| anteaya | mriedem_meeting: can you confirm, after you have finished your meeting? | 15:33 |
| *** amrith is now known as _amrith_ | 15:33 | |
| *** esikachev has quit IRC | 15:33 | |
| *** mizar has joined #openstack-infra | 15:35 | |
| *** annegentle has joined #openstack-infra | 15:35 | |
| *** mdenny has joined #openstack-infra | 15:35 | |
| *** mizar is now known as flongo | 15:35 | |
| *** jaosorior is now known as jaosorior_away | 15:37 | |
| *** [1]Thelo has joined #openstack-infra | 15:37 | |
| *** pblaho_ is now known as pblaho | 15:38 | |
| sripriya | infra-cores: could you please help review a minor update for tacker project, https://review.openstack.org/#/c/284470/. Thank you | 15:38 |
| *** Thelo has quit IRC | 15:40 | |
| *** [1]Thelo is now known as Thelo | 15:40 | |
| *** annegentle has quit IRC | 15:42 | |
| *** annegentle has joined #openstack-infra | 15:42 | |
| *** _amrith_ is now known as amrith | 15:44 | |
| *** kushal has quit IRC | 15:45 | |
| *** kdas_ has joined #openstack-infra | 15:45 | |
| *** Sukhdev has joined #openstack-infra | 15:46 | |
| odyssey4me | pabelanger do you have a minute to discuss how https://github.com/openstack/windmill/blob/master/tools/install_roles.sh works? (can you pop into #openstack-ansible to chat) ? | 15:46 |
| pabelanger | odyssey4me: sure, let me grab a coffee | 15:46 |
| *** vgridnev has quit IRC | 15:47 | |
| *** vgridnev has joined #openstack-infra | 15:48 | |
| *** jlanoux has quit IRC | 15:48 | |
| *** sorantis has joined #openstack-infra | 15:48 | |
| bogdando | AJaeger, your tox.ini fix's working like a charm. Thank you! Could you please also take a look this one https://review.openstack.org/#/c/286109 ? | 15:49 |
| *** denisra_ is now known as denisra | 15:49 | |
| *** armax has joined #openstack-infra | 15:50 | |
| *** mriedem_meeting is now known as mriedem | 15:51 | |
| *** julim has quit IRC | 15:51 | |
| *** Qiming has quit IRC | 15:52 | |
| mriedem | anteaya: the dsvm jobs use pypi released clients/libs | 15:53 |
| anteaya | yolanda: any report on what jenkins 02 is doing? | 15:53 |
| *** julim has joined #openstack-infra | 15:53 | |
| mriedem | anteaya: unless they are the -src- jobs for the client/lib repo itself | 15:53 |
| *** wolsen has joined #openstack-infra | 15:53 | |
| anteaya | mriedem: thank you | 15:53 |
| yolanda | i'm removing stale nodes there, is that behaving better? | 15:53 |
| anteaya | BobBall: ^^ | 15:53 |
| *** dtantsur is now known as dtantsur|brb | 15:53 | |
| anteaya | yolanda: well grafana says 02 has 95 ready nodes and 0 running nodes: http://grafana.openstack.org/dashboard/db/nodepool | 15:54 |
| yolanda | i see jobs running on jenkins02 | 15:54 |
| anteaya | so it is sucking up the nodes but not putting them back into play | 15:54 |
| *** amrith is now known as _amrith_ | 15:54 | |
| anteaya | yolanda: perhaps grafana is having a derp | 15:54 |
| *** _amrith_ is now known as amrith | 15:54 | |
| yolanda | https://jenkins02.openstack.org/computer/ , there are plenty of jobs running | 15:55 |
| *** dkehn has quit IRC | 15:55 | |
| anteaya | yolanda: awesome, I'd go by that link then and say the grafana graph for jenkins 02 used nodes is not behaving as I would expect | 15:56 |
| anteaya | pabelanger: any idea why grafana has jenkins 02 used nodes at 0 and https://jenkins02.openstack.org/computer/ shows jobs running? | 15:56 |
| *** zeih has quit IRC | 15:56 | |
| *** vgridnev has quit IRC | 15:57 | |
| *** angdraug has joined #openstack-infra | 15:57 | |
| *** vgridnev has joined #openstack-infra | 15:57 | |
| *** zeih has joined #openstack-infra | 15:57 | |
| *** kdas_ has quit IRC | 15:58 | |
| jeblair | anteaya, yolanda: nodepool reports that data to grafana, but no one has checked nodepool. perhaps someone should do that before we decide the graph is in error? | 15:58 |
| *** vgridnev has quit IRC | 15:58 | |
| anteaya | jeblair: ah good point | 15:58 |
| *** roxanaghe has joined #openstack-infra | 15:58 | |
| *** maishsk has quit IRC | 15:58 | |
| mriedem | clarkb: fungi: can you tell if the e-r daemon is processing results? | 15:58 |
| jeblair | anteaya: based on what i'm seeing, i suspect that jenkins02 is not reporting start or finish events to nodepool via zeromq | 15:58 |
| anteaya | ah, that would make sense | 15:59 |
| anteaya | thank you | 15:59 |
| anteaya | jeblair: and good morning | 15:59 |
| *** salv-orl_ has joined #openstack-infra | 16:00 | |
| jeblair | that will cause a significant waste of resources. jenkins02 should be removed or restarted again. | 16:00 |
| anteaya | oh | 16:00 |
| anteaya | I didn't know that | 16:00 |
| anteaya | thank you | 16:00 |
| jeblair | anteaya: all those ready nodes are really used nodes which should be re-used, but won't be for 8 hours. | 16:00 |
| yolanda | i did a restart like an hour ago. The play crashed with 502 and left lots of stale nodes, but i cleaned | 16:01 |
| *** sc68cal has joined #openstack-infra | 16:01 | |
| jeblair | yolanda: apparently zmq did not restart correctly | 16:01 |
| anteaya | jeblair: oh | 16:01 |
| yolanda | shall i restart again? | 16:01 |
| *** dkehn has joined #openstack-infra | 16:02 | |
| yolanda | jeblair, restarting again | 16:02 |
| *** arxcruz has quit IRC | 16:03 | |
| *** salv-orlando has quit IRC | 16:03 | |
| yolanda | jeblair also, what's better way to operate when the restart is not clean? what's best way to clean the stale nodes? | 16:03 |
| *** jsavak has quit IRC | 16:03 | |
| *** jsavak has joined #openstack-infra | 16:04 | |
| *** nmagnezi has quit IRC | 16:05 | |
| pabelanger | anteaya: sorry, no | 16:05 |
| fungi | yolanda: take the jenkins service on the master down completely (no java processes remaining in the process table), then mark all its nodes for deletion in nodepool and delete all the slave entries from its master config.xml on disk, then start jenkins back up again | 16:05 |
| yolanda | fungi thanks | 16:06 |
| yolanda | that should be documented... | 16:06 |
| fungi | that way there are no unaccounted for ready/used nodes in nodepool and no risk that used nodes might get reused when jenkins puts them back online when it starts up | 16:06 |
| anteaya | pabelanger: thanks it looks like issue is that jenkins 02 is not reporting correctly to nodepool via zeromq, yolanda is working on fixing the issue | 16:07 |
| *** asselin__ has joined #openstack-infra | 16:07 | |
| anteaya | pabelanger: well understanding first and then fixing | 16:07 |
| yolanda | i ran the play to restart jenkins02 again | 16:07 |
| *** fabio_ has quit IRC | 16:08 | |
| *** pedroalvarez_ is now known as pedroalvarez | 16:08 | |
| *** asselin has quit IRC | 16:10 | |
| *** kushal has joined #openstack-infra | 16:10 | |
| openstackgerrit | Sergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif https://review.openstack.org/286085 | 16:12 |
| openstackgerrit | Kevin Carter proposed openstack-infra/project-config: Added new projects for the OSA role break out https://review.openstack.org/284512 | 16:12 |
| *** abregman has quit IRC | 16:13 | |
| *** zeih has quit IRC | 16:13 | |
| openstackgerrit | Sergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif https://review.openstack.org/286085 | 16:14 |
| openstackgerrit | Thomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs https://review.openstack.org/286136 | 16:15 |
| *** jlanoux has joined #openstack-infra | 16:16 | |
| *** rcernin has quit IRC | 16:16 | |
| *** esikachev has joined #openstack-infra | 16:17 | |
| *** fawadkhaliq has quit IRC | 16:17 | |
| *** pcaruana has quit IRC | 16:17 | |
| *** stevelle_ is now known as stevelle | 16:18 | |
| clarkb | odyssey4me: we always merge with the appropriate target ref before trsting. in check that is head of the branch proposed against and in gate that is merged atop the theoretical future state that is being tested | 16:18 |
| pabelanger | anteaya: sounds good | 16:18 |
| odyssey4me | clarkb really good to know | 16:19 |
| yolanda | fungi, jeblair, i have to leave now, but left jenkins02 restarting. maybe is better if i stop that and some of you force the stop and clean up the nodes? | 16:19 |
| *** jpr has joined #openstack-infra | 16:20 | |
| fungi | yolanda: i'm going to make a pass through all the jenkins masters shortly and see what needs cleaning up | 16:22 |
| fungi | thanks for working on that | 16:22 |
| yolanda | fungi ok | 16:23 |
| yolanda | i will leave for a pair of hours | 16:23 |
| anteaya | yolanda: thank you | 16:23 |
| anteaya | fungi: thank you too | 16:23 |
| yolanda | anteaya, fungi, anytime. Bye! | 16:23 |
| anteaya | bye | 16:23 |
| fungi | and yeah, was in morning meetings which ran long, so mostly caught up on scrollback but that's why i was basically silent | 16:24 |
| anteaya | fungi: yup, figured as much | 16:24 |
| anteaya | happy meetings | 16:24 |
| fungi | working on fixing the ironic-inspector-client 1.4.0 release | 16:24 |
| anteaya | cacti says gerrit and zuul look okay | 16:24 |
| hrubi | Hi, zuul combines change queues of project when they share the same job. I have a common job which I want to run for several projects, but it is not of an integration nature, thus I don't want it to trigger the shared change queue behaviour. Would it make sense to have an option for turning this behaviour off for certain job? | 16:25 |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-cfgdb' https://review.openstack.org/286137 | 16:25 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 16:25 | |
| anteaya | and the other jenkinsii appear to be jenkining | 16:25 |
| anteaya | and the gate is moving | 16:25 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 16:25 | |
| fungi | and to the earlier comments that the ci system was "broken" or "down" all weekend, not true. it was humming along and completely caught up when i was looking at it yesterday afternoon. the nodepool config issue didn't merge and starve us for resources until utc monday | 16:26 |
| fungi | thanks jhesketh and yolanda for working through that | 16:26 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceph pure virtual method called bug 1551305 https://review.openstack.org/286140 | 16:27 |
| openstack | bug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed] https://launchpad.net/bugs/1551305 | 16:27 |
| *** abitha has joined #openstack-infra | 16:27 | |
| clarkb | hrubi: you can just use a different job name | 16:27 |
| fungi | hrubi: we do that fairly trivially with jenkins-job-builder by making the job a job-template which takes a name suffix parameter, and then using different suffixes when instantiating it for different projects | 16:28 |
| *** lxsli has joined #openstack-infra | 16:28 | |
| fungi | you could even just use the {name} parameter to differentiate it completely on project name | 16:28 |
| fungi | we do that with the majority of our non-integration-test jobs in fact (unit tests, static analysis, documentation builds, et cetera) | 16:29 |
| *** esikachev has quit IRC | 16:29 | |
| *** lxsli has quit IRC | 16:29 | |
| *** tiswanso has quit IRC | 16:30 | |
| *** lxsli has joined #openstack-infra | 16:30 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repository for fuel-plugin-murano https://review.openstack.org/269567 | 16:30 |
| *** kushal has quit IRC | 16:31 | |
| *** ashleighfarnham has joined #openstack-infra | 16:31 | |
| *** korzen_ has joined #openstack-infra | 16:31 | |
| *** tiswanso has joined #openstack-infra | 16:31 | |
| *** fhubik is now known as fhubik_brb | 16:31 | |
| *** austin81 has joined #openstack-infra | 16:32 | |
| *** korzen has quit IRC | 16:32 | |
| *** jtomasek_ has joined #openstack-infra | 16:33 | |
| *** kushal has joined #openstack-infra | 16:33 | |
| *** lxsli has quit IRC | 16:33 | |
| hrubi | clarkb, fungi: thanks. I thought this was overkill for my use case when having ~40 projects and wanted to add one common job to all of them. I would now have 40 new jobs for the same exact thing. would a patch with such option I mentioned be voted down? :) | 16:33 |
| *** jtomasek has quit IRC | 16:34 | |
| *** sputnik13 has joined #openstack-infra | 16:34 | |
| *** kushal has quit IRC | 16:34 | |
| *** lxsli has joined #openstack-infra | 16:34 | |
| *** kushal has joined #openstack-infra | 16:34 | |
| *** electrofelix has joined #openstack-infra | 16:36 | |
| clarkb | hrubi considering that zuulv3 will address this and this is how we deal with it today I doubt zuulv2 would want a patch like that | 16:36 |
| jklare | AJaeger hey, still around? | 16:37 |
| clarkb | (better to fix in zuulv3 which is part of the spec than cobble a workaround in zuulv2 when a workaround already exists) | 16:37 |
| hrubi | clarkb: alright, thanks. I'll go with separate job names then | 16:37 |
| *** markus_z has joined #openstack-infra | 16:37 | |
| *** sbalukoff has quit IRC | 16:38 | |
| *** sputnik13 has quit IRC | 16:38 | |
| *** keedya has quit IRC | 16:39 | |
| prometheanfire | someone mind +w this (dib) has two +2 already https://review.openstack.org/#/c/281960/ | 16:39 |
| *** keedya has joined #openstack-infra | 16:39 | |
| jklare | does anybody know a good way to run a jenkins job for multiple projects always on centos AND ubuntu without duplicating a lot of code | 16:39 |
| fungi | jklare: sure, job-templates | 16:40 |
| jklare | fungi sure, but i think the {node} is usually handed over in the project | 16:41 |
| jklare | fungi or can i use an array here? | 16:41 |
| fungi | jklare: you can use an array. see http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n4852 for an example | 16:42 |
| fungi | jklare: the gate-bindep-fallback-{node} template instantiation there | 16:42 |
| stevemar | Shrews: mordred: btw, we reverted the silly truncated thing for ksc and released a new version: https://review.openstack.org/#/c/285707/ | 16:43 |
| fungi | i need to clean that up too. someone added an unnecessary {suffix} parameter | 16:43 |
| *** calebb has joined #openstack-infra | 16:43 | |
| jklare | fungi and the job will be run on all of them always or just on one of them by random | 16:43 |
| *** thorst is now known as thorst_afk | 16:44 | |
| fungi | jklare: that's just defining the jobs. to run the jobs you need to tell zuul to do that, so for example this project-template in its layout added to projects which need it http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n272 | 16:44 |
| jklare | fungi ahhh i see, thanks | 16:45 |
| *** kushal has quit IRC | 16:45 | |
| max_lobur | anteaya: sdague are you around? do you have a sec to review https://review.openstack.org/#/c/281301/ (needs another +2) | 16:45 |
| *** sorantis has quit IRC | 16:45 | |
| fungi | jklare: in that example we're voting/gating on devstack-centos7 and ubuntu-trusty nodes, and running the fedora-23 version of the job only in experimental | 16:45 |
| *** asilenkov_ has joined #openstack-infra | 16:46 | |
| *** katyafervent_awa has quit IRC | 16:46 | |
| *** asilenkov has quit IRC | 16:46 | |
| *** pkarikh has quit IRC | 16:46 | |
| markus_z | Zara: Hi, we talked some weeks ago about my "requirements" of a bug tracker and I wrote some lines down, maybe we can discuss it shortly? | 16:46 |
| *** katyafervent_awa has joined #openstack-infra | 16:47 | |
| *** pkarikh has joined #openstack-infra | 16:47 | |
| *** vgridnev has joined #openstack-infra | 16:48 | |
| markus_z | Zara: It's at the bottom of https://etherpad.openstack.org/p/nova-bugs-team under the section "temp/misc/stuff". Not very elaborate, but maybe it helps. | 16:49 |
| mriedem | clarkb: check it out http://logs.openstack.org/periodic-stable/periodic-cinder-python27-liberty/c36b836/console.html#_2016-02-28_06_09_35_759 | 16:49 |
| jklare | fungi it there something like a bare-centos7 node, like the bare-trusty one? | 16:49 |
| mriedem | that debug log is in each test run over 65K times | 16:49 |
| *** sputnik13 has joined #openstack-infra | 16:51 | |
| *** scheuran has quit IRC | 16:51 | |
| fungi | jklare: no, we're in the process of moving off the bare-.* nodes anyway | 16:51 |
| *** sorantis has joined #openstack-infra | 16:51 | |
| Zara | markus_z: ah, thanks for those, reading now | 16:51 |
| jklare | fungi ok, i will try the devstack-centos7 one :) | 16:51 |
| *** annegentle has quit IRC | 16:52 | |
| fungi | jklare: in the near term we expect to have all jobs running on devstack-.* nodes (which are likely to also be renamed) | 16:52 |
| *** thingee has quit IRC | 16:53 | |
| fungi | jklare: the "bare" and "devstack" prefixes are misleading anyway. "bare" means we don't run devstack there so preinstall a ton of packages different jobs might need. "devstack" means we're going to run jobs which, like devstack, install their own dependencies | 16:53 |
| *** thingee has joined #openstack-infra | 16:53 | |
| *** fhubik_brb is now known as fhubik | 16:53 | |
| jklare | fungi ah ok | 16:53 |
| *** mrmartin has joined #openstack-infra | 16:53 | |
| fungi | jklare: so the devstack-.* nodes are a fairly minimal blank slate | 16:53 |
| fungi | that "ubuntu-trusty" node is really just "devstack-trusty" under a new name | 16:54 |
| jklare | fungi i guess i will have to just give it a try to figure out which packages we need for the chef-stuff we are doing | 16:54 |
| *** sputnik13 has quit IRC | 16:54 | |
| *** yamamoto_ has quit IRC | 16:54 | |
| jklare | fungi thanks for explaining :) | 16:55 |
| *** yamamoto has joined #openstack-infra | 16:55 | |
| fungi | jklare: yep. if you want something with the equivalent of the packageset from the bare-.* workers, try adding the install-distro-packages macro like we're doing in our experimental jobs for the switch off bare-.* nodes: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/experimental-workers.yaml | 16:55 |
| *** daemontool_ has quit IRC | 16:55 | |
| *** dtantsur|brb is now known as dtantsur | 16:56 | |
| mordred | stevemar: woot! | 16:56 |
| fungi | jklare: those are equivalents of jobs which run on bare-.* but instead install packages and in some cases do database setup too | 16:56 |
| *** matrohon has quit IRC | 16:57 | |
| *** apoorvad has joined #openstack-infra | 16:57 | |
| pleia2 | good morning | 16:59 |
| bkero | Good morning | 17:00 |
| *** sbalukoff has joined #openstack-infra | 17:00 | |
| mrmartin | good morning | 17:00 |
| *** vincentll has quit IRC | 17:00 | |
| jpmaxman | Hi | 17:01 |
| anteaya | a crowd | 17:01 |
| mrmartin | so wiki.o.o fix, what is the plan? | 17:02 |
| *** Jeffrey4l has quit IRC | 17:03 | |
| *** hashar has quit IRC | 17:04 | |
| clarkb | mrmartin: is htat all stdout? | 17:04 |
| clarkb | er | 17:04 |
| clarkb | mriedem: ^ | 17:04 |
| fungi | dhellmann: python-ironic-inspector-client-1.4.0.tar.gz is up at https://pypi.python.org/pypi/python-ironic-inspector-client now. did you also need the release announcement job run for it? | 17:04 |
| jpmaxman | Well let's start what if anything has been done? | 17:04 |
| mriedem | clarkb: yeah | 17:04 |
| mriedem | i think a logging fixture in the cinder tests is enabling debug logging for the libs so it steps on oslo.log's default levels | 17:04 |
| *** yamahata has joined #openstack-infra | 17:04 | |
| mriedem | i'm writing a test to see | 17:04 |
| jpmaxman | When I dropped off on Saturday people were going down lots of paths simultaneously | 17:05 |
| *** exploreshaifali has joined #openstack-infra | 17:05 | |
| jpmaxman | Did any changes actually make it to the server? | 17:05 |
| Zara | markus_z: that is helpful, thank you. some of those things are either implemented or on the todo list, but a lot of it is new (pretty much everything involving more sophisticated commenting), so that's interesting. it's nearly the end of the working day for me, and I expect to be busy tomorrow, but Wednesday might be a good day to talk more; what (UTC) time would work for you? | 17:05 |
| jpmaxman | I noticed that the spam is still coming in as of last night. | 17:05 |
| anteaya | mrmartin jpmaxman I suggest taking a look at the logs from the weekend: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ | 17:05 |
| mrmartin | @pleia2, jpmaxman: I suggested to disable the password auth to prevent spam account registration: https://review.openstack.org/#/c/285669/ | 17:05 |
| anteaya | I haven't had a chance myself yet so don't know the status | 17:05 |
| *** sdake has joined #openstack-infra | 17:06 | |
| jpmaxman | Ok I suggest we do one thing at a time to identify if it is the right fix or not | 17:06 |
| mrmartin | anteaya: I checked the logs, and also checked the wiki database | 17:06 |
| * anteaya continues to consume the split the design summit mailing list thread | 17:06 | |
| lucasagomes | hi folks, if you have a time mind taking a look at this devstack-gate patch? https://review.openstack.org/#/c/284036/ this is a fairly small (3LOC) patch that will allow Ironic to enable drivers such as pxe_ipmitool to run in the gate. Thank you | 17:06 |
| markus_z | Zara: I'm in UTC+1, Wednesday is perfect for me, very few meetings. I usually start at 9am | 17:06 |
| Sam-I-Am | anteaya: i'm sorry? | 17:06 |
| anteaya | mrmartin: great well if you read the logs you know the status better than I | 17:06 |
| fungi | pabelanger: had a solution which restricted account creation to wiki admins | 17:06 |
| anteaya | Sam-I-Am: what did I do? | 17:06 |
| *** kushal has joined #openstack-infra | 17:07 | |
| jpmaxman | So did anything actually make it to the server ? | 17:07 |
| fungi | https://review.openstack.org/285723 | 17:07 |
| Sam-I-Am | anteaya: reading the summit split thread | 17:07 |
| anteaya | Sam-I-Am: did I ping you by mistake? | 17:07 |
| Sam-I-Am | anteaya: no, i was just making a joke-ish | 17:07 |
| anteaya | Sam-I-Am: yes, I am reading the summit split thread, hence have not read the backlog from the weekend | 17:07 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 17:07 |
| anteaya | Sam-I-Am: oh I see, sorry had my busy hat on and I missed it | 17:07 |
| fungi | jpmaxman: i think solutions were still under discussion, since it's assumed we'll have to script some massive cleanup regardless of whether it goes on for another day or two at this point | 17:07 |
| *** jaosorior_away is now known as jaosorior | 17:08 | |
| fungi | i'm personally fine with password auth temporarily breaking the ability of #status and #success to update wiki pages until we get the stray accounts deleted and can switch to disallowing account creation by non-admins instead of completely disallowing password authentication | 17:09 |
| Sam-I-Am | anteaya: tis ok | 17:09 |
| Zara | markus_z: ah, great! :) I'm also relatively free all day Weds (well, we have the storyboard meeting at 3pm UTC, but the conversation may be relevant to that anyway :)). will talk in more detail then! | 17:09 |
| mrmartin | pabelanger's patch seems to be ok for me. | 17:09 |
| *** dizquierdo has quit IRC | 17:10 | |
| *** yamamoto has quit IRC | 17:10 | |
| fungi | i may not be completely caught up on the wiki discussion, but i think the path forward is probably to land mrmartin's change to disallow password auth and pabelanger's change to disallow account creation by non-admins, then delete all accounts with no openid link besides our two bot accounts, then reenable password auth | 17:10 |
| zaro | morning | 17:11 |
| fungi | since the latest substantiated theory seems to be that these are not acutally from compromised launchpad user accounts but rather just non-openid accounts | 17:11 |
| anteaya | Sam-I-Am: thanks | 17:11 |
| mrmartin | ok, I need to rebase the auth patch | 17:11 |
| anteaya | zaro: morning | 17:11 |
| mrmartin | first | 17:11 |
| *** fhubik has quit IRC | 17:11 | |
| fungi | as such, it should be fairly easy to spot them and remove them from the database | 17:12 |
| openstackgerrit | Merged openstack-infra/project-config: Create fuel-ui project https://review.openstack.org/260455 | 17:12 |
| fungi | however, we likely also want to retain a list of the account ids so that we can attempt to identify any edits they performed prior to their deletion | 17:13 |
| *** amotoki has quit IRC | 17:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add OSIC clouds.yaml details https://review.openstack.org/285473 | 17:13 |
| fungi | Ryan_Lane: i saw you appeared over the weekend too. do you have any input on whether there's a sane way to find and unwind edits/moves performed by those accounts? | 17:13 |
| jpmaxman | Fungi instead of removing I think we can change permissions so they can't create or edit | 17:14 |
| *** boris-42 has quit IRC | 17:14 | |
| jpmaxman | So I am fine to make that change and see what happens | 17:14 |
| fungi | sure, that's also fine | 17:14 |
| fungi | delete or disable | 17:14 |
| *** sridhar_ram has joined #openstack-infra | 17:14 | |
| *** sorantis has quit IRC | 17:14 | |
| jpmaxman | Given that we have captcha in place and the spam continues I am still concerned this may be an exploit | 17:14 |
| *** sputnik13 has joined #openstack-infra | 17:15 | |
| jpmaxman | My main point on Saturday is before we account for all the side effects of a change let's see if the change actually fixes the problem. | 17:15 |
| fungi | i wouldn't be surprised to find that they're updating via the api, so bypassing any captcha enforcement. i don't know that would strictly qualify as an exploit | 17:15 |
| jpmaxman | I still think making the captcha impossible to answer for a period of time would be a telling change too | 17:15 |
| fungi | but once they can't create password-auth accounts they should cease to be able to use the api too | 17:16 |
| *** esikachev has joined #openstack-infra | 17:16 | |
| jeblair | fungi: oh, when was it substantiated that they are non-openid accounts? | 17:16 |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-mediawiki: Disable standard password based auth https://review.openstack.org/285669 | 17:16 |
| jeblair | fungi: certainly when we started looking at this, they had openid urls associated with them | 17:16 |
| fungi | jeblair: when mrmartin started looking in the db all the accounts he found had no openid associated | 17:17 |
| jpmaxman | True. But good to know. Anyway my overarching advice is simply to quickly apply one change at a time to identify the fix more quickly then come up with a proper patch | 17:17 |
| mrmartin | yes, it is definietly with password auth | 17:17 |
| mrmartin | ok so the patch got a rebase now. | 17:17 |
| jpmaxman | If we want to start with user account creation being disabled that is fine with me | 17:17 |
| *** korzen_ has quit IRC | 17:17 | |
| *** andreykurilin__ has joined #openstack-infra | 17:18 | |
| *** tphummel has joined #openstack-infra | 17:18 | |
| *** openstackgerrit has quit IRC | 17:18 | |
| *** openstackgerrit has joined #openstack-infra | 17:18 | |
| *** vgridnev has quit IRC | 17:19 | |
| *** sc68cal has quit IRC | 17:22 | |
| *** sdake has quit IRC | 17:22 | |
| jeblair | fungi, mrmartin: there may now be non-openid users, probably because we have enabled their creation by removing the impossible-to-answer question. but when we started looking into this, the new accounts were definitely openid accounts. | 17:22 |
| jeblair | fungi, mrmartin: as an example, look at user 6498. | 17:22 |
| *** sdake has joined #openstack-infra | 17:23 | |
| *** ayoung has quit IRC | 17:23 | |
| jeblair | select * from user, user_openid where user_id=6498 and uoi_user=user_id; | 17:23 |
| pleia2 | when I looked they were also all openid accounts | 17:23 |
| fungi | yeah, i agree turning on the captcha by adding guessable q&a (via mechanical turk, et cetera) has probably increased spam by allowing non-openid account creation | 17:23 |
| pleia2 | so this is new | 17:23 |
| *** asettle has quit IRC | 17:23 | |
| Ryan_Lane | I added special:nuke for that | 17:24 |
| Ryan_Lane | For removing old edits | 17:24 |
| Ryan_Lane | It should let you remove all edits by a user or ip I think | 17:24 |
| fungi | so https://review.openstack.org/285723 seems like a more sane way to disable non-openid account creation than our old solution of having an unguessable answer | 17:25 |
| mrmartin | yeah, but 6498 don't have a valid user_password field | 17:25 |
| Ryan_Lane | Someone was mentioning that launchpad has an issue with people making fake openid accounts? | 17:25 |
| jeblair | mrmartin: yes, that's my point. it is an openid account from launchpad. | 17:25 |
| mrmartin | so, the infected accounts, have a non-empty user_password field. | 17:25 |
| fungi | Ryan_Lane: yeah, pleia2 said the ubuntu wiki is overrun with spam in recent weeks as well | 17:26 |
| jeblair | mrmartin: that is only the case recently, since the account creation question was changed | 17:26 |
| mrmartin | the field value for openid one's are '' | 17:26 |
| Ryan_Lane | You can setup autoconfirmed | 17:26 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:26 | |
| *** jcoufal has quit IRC | 17:26 | |
| *** daemontool_ has joined #openstack-infra | 17:26 | |
| Ryan_Lane | So that new users can't create new articles | 17:26 |
| *** jistr has quit IRC | 17:26 | |
| Ryan_Lane | There's also the thing I linked the other day, which will block bad ips | 17:26 |
| Ryan_Lane | Can use that in combination to disallow autoconfirmed users in the ip blacklist from editing at all | 17:27 |
| fungi | most of the spam seems to be they'll edit a page (often a "talk" page) to replace its content with spam, then redirect other high-traffic pages to it | 17:27 |
| fungi | hrm, have we possibly done anything to disable listing of the special pages? https://wiki.openstack.org/wiki/Special:SpecialPages is blank for me now even though i'm logged in | 17:29 |
| *** vgridnev has joined #openstack-infra | 17:29 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Install dstat after we set the fedora mirror https://review.openstack.org/286101 | 17:29 |
| pabelanger | ohai | 17:29 |
| *** vgridnev has quit IRC | 17:29 | |
| pabelanger | fungi: clarkb needs to revert the chmod 0000 to RunJobs.php | 17:30 |
| *** david-lyle has joined #openstack-infra | 17:30 | |
| clarkb | pabelanger: we decided that wasn't helpful? | 17:30 |
| *** sripriya has quit IRC | 17:30 | |
| clarkb | thats a simple chmod 644 $file | 17:30 |
| jpmaxman | Why don't we then start by making the answer unguessable. This will stop the new account creation. And we can see if it stops the spam. | 17:30 |
| *** kushal has quit IRC | 17:30 | |
| pabelanger | clarkb: not sure honestly, maybe we can revert it now. Do that current changes and see if runjobs.php is used again | 17:31 |
| jpmaxman | If it does not we can start disabling extensions until we find what vector the spammers are using | 17:31 |
| fungi | jpmaxman: well, it won't entirely stop the spam, because it was unguessable up until a few days ago | 17:31 |
| fungi | jpmaxman: it was made guessable in an attempt to stem the flow of spam | 17:31 |
| pabelanger | right | 17:31 |
| jpmaxman | Fungi not exactly | 17:31 |
| fungi | jpmaxman: please elaborate | 17:31 |
| jpmaxman | It was enabled for page edit and creation in an attempt to stop the spam. It was made guessable to allow humans to still edit the wiki | 17:32 |
| fungi | jpmaxman: oh, you mean leave the q&a enforced for page edits but put it back to being unguessable | 17:32 |
| jpmaxman | Yes | 17:32 |
| fungi | so basically turn off edits for the whole wiki to see if that turns off edits for the whole wiki? | 17:32 |
| *** electrofelix has quit IRC | 17:32 | |
| jpmaxman | Not for long | 17:32 |
| jpmaxman | Just maybe 30 min | 17:33 |
| jpmaxman | To see if it stops the spam | 17:33 |
| *** kushal has joined #openstack-infra | 17:33 | |
| jpmaxman | If it doesn't we know they are getting in via a different path | 17:33 |
| *** degorenko is now known as _degorenko|afk | 17:33 | |
| *** jsavak has quit IRC | 17:33 | |
| pabelanger | if that is the case, then we are in worst shape right? | 17:34 |
| *** jsavak has joined #openstack-infra | 17:34 | |
| mrmartin | don't we like to turn off puppet runs on wiki.o.o and do Settings.php modification manually until we find out what is working? | 17:34 |
| mrmartin | pabelanger: if that is the case, we still need to see something in access logs | 17:34 |
| pleia2 | we've resisted doing that because having an editable wiki is important to the project, but if people feel it's valuable as a test, we can try it and just let people know | 17:34 |
| *** electrofelix has joined #openstack-infra | 17:35 | |
| *** sputnik13 has quit IRC | 17:35 | |
| fungi | it basically boils down to "let's test our mediawiki installation for an unknown vulnerability" | 17:35 |
| *** abitha has quit IRC | 17:35 | |
| pabelanger | I think there is 2 paths we are working on. 1st, what jpmaxman is suggestion, disable edits, if we are still getting spammed, then we have some security issue. 2nd, is try to configure wiki.o.o how we needed it too, which is taking longer. | 17:35 |
| fungi | what is the cadence of spam updates? we'd need to leave it broken long enough to be reasonably sure it's preventing spamming. how long is long enough? | 17:35 |
| jpmaxman | https://wiki.openstack.org/w/index.php?title=Special:RecentChanges&limit=50 | 17:36 |
| fungi | i'm not understanding where the 30-minute estimate is coming from | 17:36 |
| jpmaxman | they are coming in almost minute by minute | 17:36 |
| fungi | okay | 17:36 |
| fungi | that's helpful | 17:36 |
| *** sputnik13 has joined #openstack-infra | 17:36 | |
| *** fawadkhaliq has joined #openstack-infra | 17:36 | |
| jpmaxman | I just think it is somewhere to start that seems logical | 17:36 |
| jpmaxman | I'm open to other suggestions | 17:36 |
| Shrews | mordred: i think we're probably due for a shade release: http://docs.openstack.org/infra/shade/releasenotes.html#id1 | 17:37 |
| jpmaxman | but we need to start narrowing this down | 17:37 |
| pabelanger | I am also not sure out blacklist spam is working properly, it would be nice to have an admin confirm in the logs we are rejecting some pages | 17:37 |
| fungi | pabelanger: i can do that now | 17:37 |
| pabelanger | https://wiki.openstack.org/wiki/MediaWiki:Spam-blacklist now exists | 17:37 |
| pabelanger | so we can start adding URLs into that | 17:37 |
| pabelanger | to see if page creation is affected | 17:38 |
| pabelanger | but, I also think there is a download process from wikipedia but unsure how to confirm that | 17:38 |
| jpmaxman | my problem with blacklists is that even if we successfully block it still doesn't tell us how they were getting in in the first place | 17:38 |
| mrmartin | let's try to prevent spam account creation first. | 17:39 |
| *** sfinucan has quit IRC | 17:39 | |
| pabelanger | well, we have a few patch up to stop user creation. which will force people to launchpad.net | 17:39 |
| *** flongo has quit IRC | 17:39 | |
| pabelanger | untested | 17:39 |
| pabelanger | https://review.openstack.org/#/c/285723/ | 17:39 |
| fungi | at which point we're back to where we were last week, only compromised/fake lp accounts are able to spam | 17:39 |
| *** flongo has joined #openstack-infra | 17:40 | |
| *** davideagnello has joined #openstack-infra | 17:40 | |
| fungi | which i think eliminates their use of the api as long as we also disable all the password-based auth accounts which got added in the interim | 17:40 |
| fungi | and forces them to have to bypass the captcha | 17:40 |
| *** abregman has joined #openstack-infra | 17:41 | |
| pabelanger | right. We can then start the process to purge accounts | 17:41 |
| pabelanger | if we are considering the possibility of a security exploit, we should also bring wiki-dev.o.o online for precise see that is needed to start the migration. | 17:42 |
| fungi | here's an example post from the apache access log: | 17:42 |
| fungi | 63.141.249.202 - - [28/Feb/2016:07:01:12 +0000] "POST /wiki/File:Support_Help_@@@_(((1-800-860-9230!_QuickBooks_((TechNICAL))_Support_phone_number,,_QuickBooks_PRO_Customer_support_phone_number.pdf/trackback/ HTTP/1.1" 302 705 "http://wiki.openstack.org/wiki/File:Support_Help_@@@_(((1-800-860-9230!_QuickBooks_((TechNICAL))_Support_phone_number,,_QuickBooks_PRO_Customer_support_phone_number.pdf" | 17:42 |
| fungi | "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" | 17:42 |
| pabelanger | we need to do that anyways, but will depend on how far we can kick it down the road | 17:43 |
| mrmartin | before the purge we need to disable fake account creation. | 17:43 |
| pabelanger | mrmartin: yes, my patch will help with that | 17:43 |
| pabelanger | another though it to maybe run awstats on our http logs and see if there is a pattern | 17:44 |
| pabelanger | or what ever is the latest and greatest http log stats program | 17:44 |
| *** ihrachys has quit IRC | 17:45 | |
| mrmartin | ok, then someone should accept pabelanger's patch, and go on | 17:46 |
| mriedem | i'm trying to find the thing in project-config that only runs unit tests/pep8 on unit test only changes | 17:46 |
| mriedem | nvm, found it | 17:46 |
| pabelanger | mriedem: zuul/layout.yaml | 17:46 |
| fungi | i'm looking for patterns in the access log right now, mostly for a clue to what's being used to do this | 17:47 |
| *** achanda has joined #openstack-infra | 17:47 | |
| fungi | the user agent string, which is most likely faked, is typical of ie11 on win7 | 17:47 |
| kozhukalov | guys could you please add me to the group https://review.openstack.org/#/admin/groups/1303,members? I am an author of the patch https://review.openstack.org/#/c/260455/ (kozhukalov Vladimir Kozhukalov vkozhukalov@mirantis.com) | 17:47 |
| *** harlowja_at_home has joined #openstack-infra | 17:48 | |
| *** rbradfor_ is now known as rbradfor | 17:49 | |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: Set persister to only run monasca tempest tests https://review.openstack.org/286179 | 17:49 |
| *** ajmiller has quit IRC | 17:50 | |
| anteaya | fungi: are you saying a browser has opened an exploit in wiki access? | 17:51 |
| fungi | anteaya: i am not saying anything, just looking at the details of the post calls in the apache access log | 17:51 |
| pabelanger | fungi: not sure I want to, but we could add some logic into apache to limit access based on User-Agent | 17:51 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Skip dsvm jobs on cinder docs/reno/unit test changes https://review.openstack.org/286180 | 17:51 |
| fungi | it's more likely a non-browser spambot which is faking its user agent string to look like ie11/win7 | 17:52 |
| anteaya | fungi: very good, sorry to jump to conclusions | 17:52 |
| *** rkukura has joined #openstack-infra | 17:52 | |
| anteaya | ah | 17:52 |
| *** ajmiller has joined #openstack-infra | 17:52 | |
| *** annegentle has joined #openstack-infra | 17:53 | |
| fungi | the usual tactic for attackers in cases like this is to claim to be the most common user agent possible, so that blocking by user agent is an unattractive option | 17:53 |
| *** jsavak has quit IRC | 17:53 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list https://review.openstack.org/286062 | 17:53 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 17:53 |
| *** derekh has quit IRC | 17:53 | |
| pleia2 | and it's trivial to do | 17:54 |
| *** tiswanso has quit IRC | 17:54 | |
| openstackgerrit | Oleksandr Kyrylchuk proposed openstack-infra/project-config: Remove cloudv-ostf-adapter from Infrastructure Systems https://review.openstack.org/285399 | 17:54 |
| fungi | ooh, they're doing file uploads too | 17:54 |
| *** exploreshaifali has quit IRC | 17:54 | |
| pleia2 | (there's verification that what the user agent says is true) | 17:54 |
| *** sdake has quit IRC | 17:54 | |
| pleia2 | er NO verification | 17:54 |
| pabelanger | fungi: ya, PDFs | 17:54 |
| nibalizer | how can I help? | 17:55 |
| *** david-lyle has quit IRC | 17:55 | |
| *** david-lyle_ has joined #openstack-infra | 17:55 | |
| *** dkranz_ has joined #openstack-infra | 17:56 | |
| *** kencjohnston has joined #openstack-infra | 17:56 | |
| fungi | nibalizer: mostly we're trying to gather as much information about the spam situation as possible on wiki.o.o while we also try to limit the damage being caused | 17:56 |
| *** dkranz has quit IRC | 17:56 | |
| *** tongli has quit IRC | 17:56 | |
| pabelanger | argument could be made, PDFs on wiki.o.o is bad to start with. Meaning we could filter our file extensions we support: https://www.mediawiki.org/wiki/Manual:$wgFileExtensions | 17:56 |
| *** sdake has joined #openstack-infra | 17:57 | |
| *** claudiub has quit IRC | 17:57 | |
| fungi | so there's a patch now approved to disable new non-openid account creation except for sysops. another patch soon to merge i think to temporarily disable password-based auth while we work on disabling accounts which were created via non-openid means | 17:57 |
| *** jsavak has joined #openstack-infra | 17:57 | |
| fungi | and discussing other avenues we can close off, filters we can get into place, cleanup tools at our disposal | 17:58 |
| anteaya | is there an etherpad up for this work? | 17:58 |
| anteaya | or a storyboard board? | 17:58 |
| anteaya | or can I create one? | 17:58 |
| pabelanger | anteaya: I'd be open to creating an etherpad for this | 17:58 |
| pleia2 | there is one that jpmaxman made, sec | 17:58 |
| anteaya | great, would you like to or shall I? | 17:58 |
| anteaya | pleia2: ah great | 17:58 |
| *** kushal has quit IRC | 17:58 | |
| pleia2 | https://etherpad.openstack.org/p/wiki.openstack.org | 17:59 |
| *** krtaylor has quit IRC | 17:59 | |
| *** annegentle has quit IRC | 17:59 | |
| anteaya | thank you | 17:59 |
| *** dkranz_ is now known as dkranz-meeting | 17:59 | |
| *** dtantsur is now known as dtantsur|afk | 17:59 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Prevent new user registrations except by sysops https://review.openstack.org/285723 | 18:00 |
| *** claudiub has joined #openstack-infra | 18:01 | |
| *** jlanoux has quit IRC | 18:01 | |
| *** Guest67703 is now known as me_ | 18:01 | |
| *** me_ is now known as med_ | 18:01 | |
| *** med_ has quit IRC | 18:01 | |
| *** med_ has joined #openstack-infra | 18:01 | |
| *** baoli has quit IRC | 18:01 | |
| *** esikachev has quit IRC | 18:02 | |
| *** baoli has joined #openstack-infra | 18:03 | |
| *** baoli has quit IRC | 18:03 | |
| *** tiswanso has joined #openstack-infra | 18:03 | |
| *** kencjohnston has quit IRC | 18:04 | |
| *** achanda has quit IRC | 18:04 | |
| *** sridhar_ram1 has joined #openstack-infra | 18:05 | |
| *** lucasagomes is now known as lucas-dinner | 18:05 | |
| *** sridhar_ram has quit IRC | 18:07 | |
| *** annegentle has joined #openstack-infra | 18:07 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Get more info from ps command https://review.openstack.org/282508 | 18:07 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove unused urllib https://review.openstack.org/261182 | 18:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-mediawiki: Disable PDF uploads by users https://review.openstack.org/286189 | 18:07 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Run top in batch mode https://review.openstack.org/273452 | 18:07 |
| *** sarob has joined #openstack-infra | 18:09 | |
| fungi | still working on correlating the recent changes page to apache access logs | 18:09 |
| *** maishsk has joined #openstack-infra | 18:10 | |
| *** matrohon has joined #openstack-infra | 18:10 | |
| *** baoli has joined #openstack-infra | 18:10 | |
| *** krtaylor has joined #openstack-infra | 18:10 | |
| fungi | aha, they're in the general access.log not the mediawiki-access.log | 18:11 |
| mrmartin | yeah, the mediawiki-access.log is for the runjobs only | 18:11 |
| fungi | this is looking much more like what i was expecting at least | 18:11 |
| fungi | and different user agent strings on these too | 18:12 |
| *** bgaifullin has quit IRC | 18:13 | |
| *** Swami has joined #openstack-infra | 18:14 | |
| *** maishsk has quit IRC | 18:15 | |
| *** maishsk has joined #openstack-infra | 18:15 | |
| fungi | ip addresses seem to be from hosting and internet service provider networks all over the world, pretty typical botnet pattern | 18:15 |
| *** yamamoto has joined #openstack-infra | 18:16 | |
| dhellmann | fungi : thanks for dealing with that release. I can run the announce script myself locally. | 18:17 |
| *** _nadya_ has quit IRC | 18:18 | |
| armax | anteaya: ping | 18:18 |
| anteaya | armax: hey there | 18:19 |
| armax | anteaya: hi, question | 18:19 |
| * anteaya listens | 18:19 | |
| armax | anteaya: the other day you showed me this page | 18:19 |
| armax | http://status.openstack.org//elastic-recheck/data/uncategorized.html | 18:19 |
| anteaya | I did so | 18:19 |
| armax | do job pop here automatically? | 18:19 |
| armax | what I mean is | 18:19 |
| *** sc68cal has joined #openstack-infra | 18:19 | |
| armax | I see for instance gate-neutron-dsvm-api | 18:19 |
| anteaya | any that don't have an elastic recheck query filed for them, yes | 18:19 |
| *** baoli_ has joined #openstack-infra | 18:19 | |
| armax | ok | 18:20 |
| armax | how do we dismiss those that are due to infra failures | 18:20 |
| anteaya | how do you mean? | 18:20 |
| armax | for instance job http://status.openstack.org//elastic-recheck/data/uncategorized.html#gate-tempest-dsvm-full has 3 | 18:20 |
| armax | anteaya: hang on | 18:21 |
| *** yamamoto has quit IRC | 18:21 | |
| armax | anteaya: that’s not the link I wanted to share | 18:21 |
| armax | anteaya: this one http://status.openstack.org//elastic-recheck/data/uncategorized.html#gate-tempest-dsvm-neutron-full | 18:21 |
| armax | both unclassified failures don’t seem due to intermittent neutron issues | 18:22 |
| *** baoli has quit IRC | 18:22 | |
| *** electrofelix has quit IRC | 18:22 | |
| anteaya | that may be true | 18:22 |
| armax | perhaps you have some guide for me to read? | 18:22 |
| anteaya | however there is still a part of the sytem that is not working as expected | 18:22 |
| anteaya | for elastic-recheck? | 18:22 |
| * anteaya gets the link | 18:22 | |
| fungi | Ryan_Lane: any guesses what this would be trying to accomplish? "GET /w/load.php?debug=false&lang=en-gb&modules=ext.echo.badge%7Cext.uls.nojs%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.ui.button&only=styles&skin=strapping&* HTTP/1.1" | 18:23 |
| fungi | Ryan_Lane: i see it being tried by one of the ip addresses which was also spamming pages | 18:23 |
| anteaya | armax: http://docs.openstack.org/infra/elastic-recheck/ | 18:23 |
| anteaya | armax: also dougwig had a good chat with mriedem on friday about elastic-recheck, and kevinbenton was there too | 18:23 |
| *** thorst_afk is now known as thorst | 18:23 | |
| armax | anteaya: ok | 18:23 |
| anteaya | armax: they might remember some helpful stuff as well | 18:24 |
| anteaya | armax: thanks for asking | 18:24 |
| nibalizer | fungi: im gonna pull some of these longer log files locally so I can peer at them better | 18:24 |
| *** Guest63721 is now known as mfisch | 18:24 | |
| anteaya | armax: if a job fails and it is due to an infra issue, we still need an e-r query filed about it, so we can track it | 18:24 |
| *** maishsk has quit IRC | 18:24 | |
| armax | anteaya: ok | 18:24 |
| anteaya | armax: sometimes our hosts differ or change things, but we don't know about it unless we can track it | 18:24 |
| anteaya | armax: thanks :) | 18:25 |
| *** mfisch is now known as Guest2004 | 18:25 | |
| armax | anteaya: that makes sense | 18:25 |
| anteaya | great, thank you | 18:25 |
| *** _nadya_ has joined #openstack-infra | 18:25 | |
| fungi | Ryan_Lane: nevermind, i think this is just how css is linked in the theme | 18:25 |
| fungi | though that does point to the attacks likely using actual browsers | 18:25 |
| armax | anteaya: I am still somewhat wondering how come the neutron functional job doesn’t show up on this dashboard | 18:25 |
| fungi | i'm seeing http post calls for action=edit, and then followup http get calls for the resulting page _and_ linked css | 18:26 |
| armax | anteaya: on the unclassified dashboard I mean | 18:26 |
| anteaya | armax: good question, well we noticed on friday that the elastic-recheck bot isn't behaving as mriedem expects, but we haven't had much time to dig into it with clarkb yet | 18:26 |
| anteaya | armax: yes, we expected more unclassified results than we were seeing on friday | 18:27 |
| mriedem | armax: does the neutron functional job run in the gate queue or only check queue? | 18:27 |
| armax | anteaya: ok, I’ll look into those are reported | 18:27 |
| armax | mriedem: check only | 18:27 |
| mriedem | and is it voting? | 18:27 |
| armax | mriedem: voiting | 18:27 |
| anteaya | armax: currenting infra is focused on addressing our wiki spam issue that has been going on for about 2 weeks | 18:27 |
| *** krtaylor has quit IRC | 18:27 | |
| mriedem | armax: the uncategorized bugs page is only gate queue failures | 18:27 |
| armax | mriedem: ack | 18:27 |
| anteaya | so after that is over was hoping to find clarkb and ask him about it | 18:27 |
| anteaya | mriedem: ah thank you | 18:28 |
| clarkb | mriedem: armax anteaya all of the e-r html /json files are up to date | 18:28 |
| clarkb | I think only the bot is having trouble? | 18:28 |
| mriedem | clarkb: yeah | 18:28 |
| armax | mriedem: the functional job was on the gate queue at one point, how long do these go back? | 18:28 |
| mriedem | armax: 10 days | 18:28 |
| *** Guest2004 is now known as mfisch | 18:28 | |
| armax | mriedem: that explains it | 18:28 |
| *** mfisch has quit IRC | 18:28 | |
| *** mfisch has joined #openstack-infra | 18:28 | |
| armax | mriedem: thanks | 18:28 |
| *** bpokorny has joined #openstack-infra | 18:28 | |
| clarkb | mriedem: freenode has been flapping I am almost certain that the bot got lost in that | 18:28 |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard https://review.openstack.org/286194 | 18:29 |
| mriedem | armax: np, that's why you can only filter back to 10 days on the page | 18:29 |
| mriedem | Generated at: 2016-02-29T18:20 (View: 24 hours, 2 days, 7 days, 10 days) | 18:29 |
| *** baoli_ has quit IRC | 18:29 | |
| armax | mriedem: oh, I see that now | 18:29 |
| fungi | nibalizer: picking the client ip address of a particular offending spam and then filtering the log for that and analyzing the flow of http requests is interesting | 18:29 |
| clarkb | though looks like it is connected now according to the PING PONG log messages | 18:29 |
| * armax blind on monday morning | 18:29 | |
| *** boris-42 has joined #openstack-infra | 18:30 | |
| *** baoli has joined #openstack-infra | 18:31 | |
| mriedem | sdague: in case you want to reduce unnecessary test node usage https://review.openstack.org/#/c/286180/ | 18:31 |
| *** thorst is now known as thorst_afk | 18:31 | |
| *** abitha has joined #openstack-infra | 18:31 | |
| *** doug-fish has quit IRC | 18:32 | |
| *** marekd has joined #openstack-infra | 18:32 | |
| anteaya | armax: thanks for asking about it :) | 18:32 |
| *** ashleighfarnham has quit IRC | 18:32 | |
| armax | anteaya: thank you | 18:32 |
| fungi | nibalizer: on the one i'm looking at (), the first we ever hear from this client is a get request for one of the existing spam files in the wiki, with a referrer claimed to be https://www.google.co.in/ | 18:32 |
| *** ashleighfarnham has joined #openstack-infra | 18:32 | |
| fungi | nibalizer: it's not until a couple minutes later that it issues its first post, which is an upload of a different file | 18:33 |
| anteaya | armax: welcome :) | 18:33 |
| fungi | nibalizer: er, () was supposed to be (43.252.25.79) | 18:33 |
| *** sridhar_ram1 is now known as sridhar_ram | 18:34 | |
| *** jamesmcarthur has joined #openstack-infra | 18:34 | |
| *** notnownikki has quit IRC | 18:35 | |
| pabelanger | according to https://wiki.openstack.org/wiki/Special:ListGroupRights only admins can use createaccount now | 18:36 |
| pabelanger | we need to test if launchpad.net accounts are still able to be created | 18:36 |
| pabelanger | I think I can test that with one of my bot launchpad accounts | 18:36 |
| *** annegentle has quit IRC | 18:37 | |
| *** cloudtrainme has joined #openstack-infra | 18:37 | |
| mrmartin | I can check the last spam account user id in the database. | 18:37 |
| fungi | nibalizer: so this is the history of the page in question: https://wiki.openstack.org/w/index.php?title=File:Tech_(Support)%E2%80%A6.))))1%2B844%2B461%2B2828((((!!!_Pogo_tech_Support_phone_number,,_Pogo_techNIcal_support_phone_number._usa.pdf&action=history | 18:37 |
| nibalizer | is it weird that I tend to think of pogo as a real company? | 18:38 |
| mrmartin | ok, so just for the record the last password based user account is: 7791 | KatieBatista12 | 18:38 |
| fungi | nibalizer: it looks like the file was uploaded a couple days ago, with a different account at a different ip address, and this new ip address retrieved the page for it today, then updated the page description | 18:38 |
| *** sripriya has joined #openstack-infra | 18:38 | |
| fungi | this is pretty sophisticated | 18:38 |
| *** krtaylor has joined #openstack-infra | 18:39 | |
| anteaya | mrmartin: is there a timestamp associated with account creation on that account? | 18:40 |
| anteaya | fungi: so they might be updating via email notifications? | 18:40 |
| mrmartin | anteaya: user_touched: 20160229180338 | 18:40 |
| *** yaume has quit IRC | 18:41 | |
| anteaya | mrmartin: thank you | 18:41 |
| fungi | mrmartin: anteaya: that would be just shy of 40 minutes ago then | 18:41 |
| anteaya | yup | 18:41 |
| *** kevinsho_ has quit IRC | 18:41 | |
| fungi | assuming that's a squashed iso-8601 timestamp in utc, which it looks to be | 18:41 |
| *** baoli has quit IRC | 18:42 | |
| anteaya | iso 8601 seems to stick a T in there according to: https://en.wikipedia.org/wiki/ISO_8601 | 18:42 |
| anteaya | I might be reading that wrong | 18:43 |
| pabelanger | clarkb: fungi: do you mind chmod 0644 runjobs.php? | 18:43 |
| *** sdake has quit IRC | 18:43 | |
| fungi | pabelanger: happen to know the full path? | 18:43 |
| clarkb | pabelanger: I can do it | 18:43 |
| *** flongo has quit IRC | 18:43 | |
| fungi | ahh, clarkb would know since it's likely in his command history | 18:43 |
| *** baoli has joined #openstack-infra | 18:44 | |
| pabelanger | so, login into wiki.o.o for the first time with a new launchpad.net id, can only see the following: | 18:44 |
| pabelanger | All users need a nickname; you can choose one from the options below. | 18:44 |
| pabelanger | Username and account choice | 18:44 |
| pabelanger | Log in / create account with OpenIDCancel | 18:44 |
| pabelanger | clicking loging doesn't do much but loop back to the page | 18:44 |
| pabelanger | reading up on openid extensions for suggestions | 18:44 |
| *** angdraug has quit IRC | 18:45 | |
| clarkb | all done | 18:45 |
| *** pvaneck has joined #openstack-infra | 18:45 | |
| *** Sukhdev has quit IRC | 18:45 | |
| *** doug-fish has joined #openstack-infra | 18:46 | |
| *** nmagnezi has joined #openstack-infra | 18:46 | |
| *** doug-fish has quit IRC | 18:46 | |
| *** doug-fish has joined #openstack-infra | 18:47 | |
| fungi | anteaya: the note in section 4.3.2 says the T is effectively optional | 18:48 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Added summit_type_id/event_type_id filter to events endpoint https://review.openstack.org/286199 | 18:49 |
| *** notnownikki has joined #openstack-infra | 18:49 | |
| fungi | in the 8601:2004(E) revision anyway | 18:49 |
| *** kencjohnston has joined #openstack-infra | 18:49 | |
| fungi | or so i gather from context, since i haven't bought a copy of the standard | 18:50 |
| *** e0ne has joined #openstack-infra | 18:51 | |
| anteaya | thank you | 18:51 |
| *** baoli has quit IRC | 18:52 | |
| fungi | gotta love standards bodies who charge inordinate sums of money for copies of their standards (even digital copies) | 18:52 |
| *** thorst_afk is now known as thorst | 18:52 | |
| anteaya | there it is footnote 19, thanks | 18:52 |
| anteaya | :( | 18:53 |
| clarkb | fungi: you know the standard is good when you have to pay lots of money to read it | 18:53 |
| clarkb | see also SQL | 18:53 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:53 | |
| pabelanger | So, reading https://www.mediawiki.org/wiki/Extension:OpenID I think we need some updates to our config. By setting createaccount = False, we removed the ablity for the user to pick there user name. So, I think we need to set $wgOpenIDUseEmailAsNickname = True and possible $wgOpenIDProposeUsernameFromSREG = True too | 18:53 |
| fungi | in the past i've gotten iso standards in digital form on the cheap through ansi since they have an agreement to allow ansi to distribute at least some iso standards documents to american companies/citizens at a fraction of the iso direct cost | 18:53 |
| *** baoli has joined #openstack-infra | 18:54 | |
| fungi | but still, paying for access to standards seems counter to the idea of having a standard | 18:54 |
| *** jaosorior has quit IRC | 18:54 | |
| anteaya | indeed | 18:54 |
| *** baoli has quit IRC | 18:55 | |
| pabelanger | mrmartin: looking at https://wiki.openstack.org/w/index.php?title=Special%3AListUsers&username=&group=&creationSort=1&desc=1&limit=50 I see Helen as latest user created | 18:55 |
| fungi | people will often either break copyright law to code to an illegally-obtained copy of the standard, or just declare access to the standard unworkable and make up their own nonstandard implementation | 18:55 |
| *** maishsk has joined #openstack-infra | 18:56 | |
| clarkb | jeblair: new raspi 3 has integrated wireless | 18:56 |
| anteaya | pabelanger: timestamp? | 18:56 |
| pabelanger | anteaya: after KatieBatista12 | 18:56 |
| pabelanger | 29 Feb 2016 @ 18:06 | 18:56 |
| mrmartin | pabelanger yeah, but Helen is an openid account | 18:56 |
| fungi | pabelanger: mrmartin: Helen also updated a page at 18:54:05 according to https://wiki.openstack.org/wiki/Special:Log/upload | 18:57 |
| fungi | with spam content | 18:57 |
| pabelanger | mrmartin: Ah, good to know | 18:57 |
| mrmartin | the last password auth account is still KatieBatista12 | 18:57 |
| anteaya | :( | 18:57 |
| fungi | so new openid account, spamming the page | 18:57 |
| mrmartin | awesome | 18:57 |
| pabelanger | fungi: see my comments above about openid creation | 18:57 |
| pabelanger | right now, I cannot create a new users using it | 18:57 |
| *** baoli has joined #openstack-infra | 18:58 | |
| mrmartin | you should ask the spammer's tool to create a new account. | 18:58 |
| pabelanger | indeed | 18:58 |
| *** jaosorior has joined #openstack-infra | 18:58 | |
| *** _nadya_ has quit IRC | 18:58 | |
| fungi | i'm looking in the apache logs now for an indication of what urls they hit for that | 18:59 |
| mrmartin | but this means that launchpad is full with fake accounts | 18:59 |
| anteaya | so thus far we have had zero success closing any spammer account creation vector | 18:59 |
| *** andymaier has quit IRC | 18:59 | |
| pabelanger | Right, if launchpad.net is full of spammers, then we need to up the game for blacklisting URLs | 18:59 |
| pabelanger | I'll be surprised if QuickBooks is using openstack | 19:00 |
| *** maishsk has quit IRC | 19:00 | |
| fungi | i also don't think quickbooks needs to be linking to their website from our wiki even if they are using openstacjk | 19:01 |
| fungi | er, openstacj | 19:01 |
| fungi | gah | 19:01 |
| fungi | i give up typing | 19:01 |
| *** rockyg has joined #openstack-infra | 19:01 | |
| anteaya | it is overrated anyway | 19:02 |
| pabelanger | so far new users look good | 19:03 |
| pabelanger | but, pretty sure we broke any user creation :) | 19:03 |
| pabelanger | s/broke/stopped | 19:03 |
| *** doug-fis_ has joined #openstack-infra | 19:03 | |
| clarkb | if https://review.openstack.org/#/c/285473/ passes testing this time around it would be great to get that in | 19:04 |
| clarkb | then I can work on getting osic up and running | 19:04 |
| jpmaxman | just fYI still spam coming in | 19:04 |
| mrmartin | what I missed during the registration was the user confirmation email | 19:04 |
| jpmaxman | https://wiki.openstack.org/w/index.php?title=Special:RecentChanges&limit=50 | 19:04 |
| *** asselin__ has quit IRC | 19:05 | |
| jpmaxman | brb | 19:05 |
| pabelanger | so, lets land: https://review.openstack.org/#/c/286189/ to disble PDF uploads | 19:05 |
| *** arif-ali has quit IRC | 19:06 | |
| *** doug-fi__ has joined #openstack-infra | 19:06 | |
| *** doug-fi__ has quit IRC | 19:06 | |
| *** doug-fi__ has joined #openstack-infra | 19:07 | |
| *** julim has quit IRC | 19:07 | |
| *** jaosorior has quit IRC | 19:07 | |
| *** doug-fish has quit IRC | 19:07 | |
| *** doug-fis_ has quit IRC | 19:08 | |
| *** maishsk has joined #openstack-infra | 19:08 | |
| *** julim has joined #openstack-infra | 19:10 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Added summit_type_id/event_type_id filter to events endpoint https://review.openstack.org/286199 | 19:10 |
| fungi | that was probably added because some people were actually legitimately wanting to upload pdf content for reference, but it's not actually searchable in mediawiki so i agree that's probably safe | 19:10 |
| fungi | or at least something we'd want to discourage anyway | 19:11 |
| fungi | so i don't object | 19:11 |
| *** dims_ has joined #openstack-infra | 19:11 | |
| *** dims has quit IRC | 19:11 | |
| *** annegentle has joined #openstack-infra | 19:12 | |
| *** flongo has joined #openstack-infra | 19:12 | |
| *** annegentle has quit IRC | 19:13 | |
| *** angdraug has joined #openstack-infra | 19:13 | |
| *** mtanino has quit IRC | 19:14 | |
| anteaya | I'm open to even reducing the ability of our users to use the wiki temporarily until we can identify what will stop the spamming | 19:15 |
| mrmartin | would be great to identify a pattern here | 19:16 |
| *** doug-fi__ is now known as doug-fish | 19:16 | |
| *** annegentle has joined #openstack-infra | 19:16 | |
| fungi | yeah, i'm trying to analyze Helen's use now | 19:16 |
| *** ifarkas has quit IRC | 19:16 | |
| fungi | curious to see whether Helen is coming from more than one ip address for example | 19:16 |
| *** jsavak has quit IRC | 19:16 | |
| *** tiswanso has quit IRC | 19:16 | |
| *** jsavak has joined #openstack-infra | 19:17 | |
| pabelanger | A lot of the changes I see now are related to PDF files | 19:17 |
| *** baoli has quit IRC | 19:18 | |
| fungi | and whether other users than Helen are coming from the same address too | 19:18 |
| mrmartin | but they are also creating pages | 19:18 |
| anteaya | pabelanger: can you take your workflow -1 off of 286189? | 19:18 |
| *** baoli has joined #openstack-infra | 19:18 | |
| pabelanger | anteaya: done | 19:18 |
| openstackgerrit | Austin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP) https://review.openstack.org/279317 | 19:19 |
| prometheanfire | any DIB cores around to give this +W? it has the two +2 it needed, not sure why it didn't get workflow. https://review.openstack.org/#/c/281960/ | 19:19 |
| fungi | so... at the same time Helen created https://wiki.openstack.org/w/index.php?title=File:Call_(((1-800-919-0992_QuickBooks_POS_technical_support_phone_number_support_Phone_Number_USA.pdf&action=history i see a get of that url from 69.42.58.191 in the apache logs. i'm having trouble tracking down the actual file upload in the logs | 19:19 |
| pabelanger | If I can figure out how to add 1-800-919-0992 to spam | 19:19 |
| pabelanger | that will get a lot of quickbooks pages | 19:19 |
| *** tiswanso has joined #openstack-infra | 19:19 | |
| fungi | but that ip address was making other requests from the wiki well before Helen's account was created | 19:19 |
| *** ihrachys has joined #openstack-infra | 19:20 | |
| *** e0ne has quit IRC | 19:20 | |
| anteaya | pabelanger: thank you | 19:20 |
| fungi | so as i would expect, confirmation that there are multiple accounts being used from the same client addresses | 19:20 |
| anteaya | clarkb: openstackjenkins-osic is only mentioned in this file: https://review.openstack.org/#/c/285473/4/modules/openstack_project/templates/puppetmaster/all-clouds.yaml.erb what is it? | 19:21 |
| clarkb | anteaya: it is the account that nodepool uses | 19:21 |
| *** _nadya_ has joined #openstack-infra | 19:21 | |
| anteaya | clarkb: ah thank you | 19:21 |
| clarkb | anteaya: in each of our clouds we have two accounts, the account for services (mirrors etc) and the account to run tests in | 19:22 |
| clarkb | openstackjenkins == run tests in and openstackci == services | 19:22 |
| anteaya | oh | 19:22 |
| anteaya | thank you, I didn't know that | 19:22 |
| anteaya | clarkb: can you review 286189? | 19:23 |
| *** kzaitsev_mb has quit IRC | 19:23 | |
| anteaya | we are hoping that may help | 19:23 |
| *** baoli has quit IRC | 19:23 | |
| *** baoli has joined #openstack-infra | 19:23 | |
| clarkb | anteaya: have we seen people POSTing pdfs as part of the spamming? | 19:23 |
| fungi | anteaya: another way of looking at it is that any machines booted by nodepool go in their own isolated project (openstackjenkins) so that we don't have to trust nodepool with credentials to our other servers | 19:23 |
| clarkb | where people == spam bots? | 19:23 |
| openstackgerrit | Austin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP) https://review.openstack.org/279317 | 19:23 |
| anteaya | clarkb: yes | 19:24 |
| fungi | clarkb: a majority of the activity is uploads of pdfs | 19:24 |
| anteaya | fungi: that is a good way of looking at it, thank you | 19:24 |
| fungi | though i've no doubt that as soon as we stop allowing pdf uploads, they'll just fall back to doing more spam content in pages | 19:24 |
| fungi | which they're already doing quite a bit of anyway | 19:24 |
| pleia2 | just +Aed the pdf one | 19:24 |
| pleia2 | but yeah, I'm sure inline spam will come back very fast | 19:25 |
| anteaya | pleia2: thank you | 19:25 |
| *** ybathia has joined #openstack-infra | 19:25 | |
| *** maishsk has quit IRC | 19:25 | |
| * anteaya eats some food | 19:25 | |
| *** baoli has quit IRC | 19:26 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Added summit_type_id/event_type_id filter to events endpoint https://review.openstack.org/286199 | 19:27 |
| *** dkehn has quit IRC | 19:27 | |
| openstackgerrit | James E. Blair proposed openstack/gertty: Make topic selection a select list https://review.openstack.org/286238 | 19:27 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Indent projects under topics https://review.openstack.org/286239 | 19:27 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add configuration information to docs https://review.openstack.org/286240 | 19:27 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove obsolete dvipng install https://review.openstack.org/286241 | 19:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Remove obsolete dvipng install https://review.openstack.org/286242 | 19:30 |
| clarkb | there are a bunch of older ready nodes in nodepool that jenkinses don't seem to know about, I am going to mark them delete so that we can put that quota to use | 19:31 |
| *** rossella_s has quit IRC | 19:32 | |
| *** austin81 has left #openstack-infra | 19:32 | |
| fungi | fun. so looking at the openstack_wiki db, Helen (user.user_id 7792, created today at 18:06:28) has a user.user_email of surajchand770@gmail.com and user_openid.uoi_openid url https://login.launchpad.net/+id/QbnH7Rm which i think corresponds to https://launchpad.net/~suraj but someone probably needs to use the launchpad.people.getByOpenIDIdentifier() api method to confirm | 19:33 |
| *** ybathia has quit IRC | 19:33 | |
| *** bgaifullin has joined #openstack-infra | 19:33 | |
| fungi | clarkb: oh, yeah, i was going to start digging into seeing which jenkinses are broken and in need of cleanup a couple hours ago before i got distracted by wikispam | 19:34 |
| *** IlyaG has joined #openstack-infra | 19:34 | |
| fungi | i'll get back to that | 19:34 |
| *** IlyaG has quit IRC | 19:34 | |
| clarkb | fungi: just open that login url in your browser | 19:34 |
| clarkb | fungi: it will confirm that it belongs to suraj | 19:34 |
| mrmartin | member since: 2010-07-15 ? | 19:34 |
| AJaeger | fungi, can I distract you for two mins about bindep/ubuntu-trusty? | 19:34 |
| fungi | clarkb: all that tells you is an identifier, but i've seen there be duplicates | 19:34 |
| fungi | AJaeger: probably not today, no | 19:35 |
| *** rossella_s has joined #openstack-infra | 19:35 | |
| AJaeger | ok, fungi | 19:35 |
| clarkb | fungi: you cna have multiple openids that map to a single ~user | 19:35 |
| fungi | clarkb: the openid url returns what they set their visible name to, but it's not necessarily == to their lp username | 19:35 |
| clarkb | fungi: but Idon't think you can have multiple accounts with different openids and ~users | 19:35 |
| *** mtanino has joined #openstack-infra | 19:35 | |
| clarkb | fungi: huh | 19:35 |
| fungi | clarkb: it's not actually a proper lookup, which is why we got them to add that reverse lookup api call for us | 19:35 |
| *** sambetts is now known as sambetts|afk | 19:36 | |
| fungi | clarkb: for example digging my openid url out of the gerrit db and loading it in a browser, it says "OpenID Identity URL for Jeremy Stanley" but the launchpad.people.getByOpenIDIdentifier() should properly reverse that to https://launchpad.net/~fungi | 19:38 |
| *** dkehn has joined #openstack-infra | 19:38 | |
| *** baoli has joined #openstack-infra | 19:39 | |
| fungi | and there's no unique key in lp for the account display name afaik | 19:39 |
| fungi | so this could be any random lp account with display name set to suraj | 19:39 |
| pabelanger | okay, I'm going to start deleting old PDF files on the wiki | 19:40 |
| mrmartin | a real captcha at the new user registration page (even from openid) would solve this issue. | 19:40 |
| clarkb | most of the "ready" nodes belonged to jenkins02 | 19:40 |
| clarkb | so I am ssuming we removed them from the jenkins master but not nodeool when it got restarted | 19:40 |
| clarkb | fungi: huh fun | 19:40 |
| fungi | clarkb: yep, that was a known problem then. when yolanda tried to restart jenkins02 earlier nodepool wasn't seeing zmq from it | 19:40 |
| fungi | or that's what it seemed like anyway | 19:40 |
| *** jordanP has quit IRC | 19:41 | |
| fungi | so used or finished slaves weren't transitioning out of ready state in nodepool | 19:41 |
| anteaya | pabelanger: ack | 19:41 |
| kevinbenton | anteaya: hey, i remember you mentioned infra had an issue with neutron security groups. who was it? | 19:42 |
| anteaya | kevinbenton: hi yes thank you for rememebering | 19:42 |
| nibalizer | kevinbenton: it was in infracloud | 19:42 |
| clarkb | fungi: they were all at least 3 hours old and some were 5 days old so I went ahead and set them all to delete | 19:42 |
| anteaya | can we try again perhaps tomorrow or wednesday | 19:42 |
| *** tongli has joined #openstack-infra | 19:42 | |
| anteaya | kevinbenton: the channel is focused on trying to address spam in the wiki | 19:42 |
| fungi | jenkins03 looks like it's got a ton of cruft nodes. going to start working on cleaning it up next | 19:42 |
| anteaya | kevinbenton: but we would really like to have the neutron default security groups chat | 19:43 |
| fungi | also checking bluebox for leaked floating ips | 19:43 |
| *** jordanP has joined #openstack-infra | 19:43 | |
| kevinbenton | anteaya: ack. maybe tomorrow? | 19:43 |
| *** jordanP has quit IRC | 19:43 | |
| anteaya | jenkins03 was also restarted today according to AJaeger and yolanda, I never did get the backscroll read | 19:43 |
| anteaya | kevinbenton: awesome thank you, let's try tomorrow | 19:43 |
| kevinbenton | anteaya: sounds good | 19:43 |
| fungi | anteaya: in that case it never got its list of workers cleaned up or something | 19:43 |
| anteaya | kevinbenton: thanks for remembering | 19:43 |
| *** ihrachys has quit IRC | 19:44 | |
| anteaya | fungi: that sounds likely | 19:44 |
| AJaeger | jhesketh restarted jenkins03 | 19:44 |
| fungi | anteaya: it seems to have lots of offline slaves that refer to nodepool nodes which were deleted some time ago | 19:44 |
| anteaya | :( | 19:44 |
| fungi | so i'm guessing he didn't clean out the slave entries in its config when he restarted it | 19:44 |
| pabelanger | I am sad that https://wiki.openstack.org/w/index.php?title=Special%3ALog&type=spamblacklist is empty | 19:44 |
| anteaya | not the most efficient | 19:44 |
| anteaya | fungi: likely | 19:44 |
| pabelanger | Need to look into why | 19:45 |
| anteaya | fungi: I wonder if he knows to do that? | 19:45 |
| AJaeger | anteaya: commented on 286242 - see https://review.openstack.org/#/q/status:open+topic:pngmath | 19:45 |
| anteaya | pabelanger: I can't see it, I'm not an administrator | 19:45 |
| anteaya | AJaeger: okay thanks | 19:45 |
| fungi | anteaya: no idea. the closest thing we have to documentation of a manual restart process is clarkb's ansible playbook | 19:45 |
| anteaya | after the fun dies down perhaps we could maybe look at some docs for this process | 19:46 |
| clarkb | which does not remove things from nodepool because it relies on things to coalesce when in shutdown mode | 19:46 |
| anteaya | AJaeger: ah wonderful | 19:47 |
| fungi | yep, though it _does_ remove remaining slaves from jenkins, which someone doing this manually might not think about | 19:47 |
| fungi | re: jenkins03 | 19:47 |
| clarkb | yup it will remove offline slaves once all jobs are done running | 19:47 |
| anteaya | I think both jhesketh and yolanda were caught by not knowing that | 19:47 |
| pabelanger | So we haven't had any new pages since 19:19 UTC | 19:47 |
| pabelanger | and that was a valid user | 19:47 |
| *** achanda has joined #openstack-infra | 19:47 | |
| pabelanger | https://wiki.openstack.org/wiki/Special:NewPages | 19:47 |
| pabelanger | So, the spammers appear to be hitting existing pages / files | 19:48 |
| fungi | i'm deleting 45 leaked floating ips in bluebox now | 19:48 |
| anteaya | fungi: thank you | 19:48 |
| pabelanger | Helen is spamming like a champ | 19:48 |
| anteaya | hopefully she reveals her pattern | 19:49 |
| fungi | pabelanger: a.k.a. "suraj" | 19:49 |
| pabelanger | I haven't blocked anybody today, just deleting pdf files | 19:49 |
| pabelanger | hopefully once our upload patch lands, traffic will slow | 19:49 |
| *** hashar has joined #openstack-infra | 19:49 | |
| fungi | also keep in mind the Special:Nuke Ryan_Lane mentioned earlier, which should undo actions taken by a specific user or possibly also ip address | 19:50 |
| jeblair | do not nuke ryan lane ;) | 19:50 |
| anteaya | agreed | 19:50 |
| *** MarkAtwood1 has joined #openstack-infra | 19:50 | |
| fungi | though i don't see it at https://wiki.openstack.org/wiki/Special:SpecialPages so we may need an extension for it? | 19:50 |
| pabelanger | https://wiki.openstack.org/w/index.php?title=Special:Nuke&target=Terencejames90 | 19:50 |
| *** dalgaaf has joined #openstack-infra | 19:51 | |
| pabelanger | is how I am deleting spam | 19:51 |
| pabelanger | which is nuke I believe | 19:51 |
| fungi | oh, it does exist | 19:51 |
| *** MarkAtwood1 has quit IRC | 19:51 | |
| *** MarkAtwood has quit IRC | 19:51 | |
| fungi | https://wiki.openstack.org/wiki/Special:Nuke | 19:51 |
| pabelanger | Mass delete is what wiki.o.o calls it | 19:51 |
| fungi | yep | 19:51 |
| fungi | so anyway, we'll want a list of known spammer accounts to feed into that | 19:52 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for ceph pure virtual method called bug 1551305 https://review.openstack.org/286140 | 19:52 |
| openstack | bug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed] https://launchpad.net/bugs/1551305 | 19:52 |
| zaro | jeblair: i have run a few tests with regards to http://lists.openstack.org/pipermail/openstack-infra/2016-January/003640.html | 19:55 |
| zaro | jeblair: let me know when you are available to discuss. | 19:55 |
| *** gtmanfred has quit IRC | 19:56 | |
| *** gtmanfred has joined #openstack-infra | 19:57 | |
| *** achanda has quit IRC | 19:57 | |
| fungi | i've finished looking through the other jenkins masters and they seem fine, so once 03 is dealt with we should be back in good shape there | 19:57 |
| anteaya | fungi: thank you | 19:58 |
| *** dims_ has quit IRC | 19:58 | |
| anteaya | the graphs in grafana look as I expect now | 19:58 |
| mrmartin | we have 607 user accounts without passwords, but how will we identify the infected openid ones? | 19:59 |
| *** armax has quit IRC | 19:59 | |
| clarkb | mrmartin: I think you can start with ones that were created recently, then filter through changes they have made to identify which are spamming? | 19:59 |
| fungi | mrmartin: we might need to work backwards from known spam patterns to put together a list of which accounts made/uploaded them | 19:59 |
| mrmartin | yeah | 20:00 |
| *** Guest71383 is now known as redrobot | 20:01 | |
| *** ashleighfarnham has quit IRC | 20:02 | |
| *** baoli has quit IRC | 20:02 | |
| *** amitgandhinz has quit IRC | 20:03 | |
| *** sdake has joined #openstack-infra | 20:04 | |
| *** amitgandhinz has joined #openstack-infra | 20:05 | |
| mrmartin | ok, so we need to find the spam pattern first. | 20:07 |
| mrmartin | are we the only mediawiki users who met with this kind of attack? | 20:07 |
| *** dtardivel has quit IRC | 20:08 | |
| *** cznewt has quit IRC | 20:08 | |
| clarkb | mrmartin: aiui many wikis have come under similar attack recently and not just mediawiki | 20:09 |
| *** dkranz-meeting is now known as dkranz | 20:09 | |
| clarkb | ubuntu for example uses moinmoin and ran into the same thing | 20:09 |
| clarkb | (which makes me think a mediawiki vulnerability is less likely as a source) | 20:09 |
| *** gtmanfred has quit IRC | 20:09 | |
| mrmartin | and do we know what was their solution for this problem? | 20:09 |
| mrmartin | have they found any pattern in the attacks? | 20:10 |
| clarkb | mrmartin: they went read only | 20:10 |
| mrmartin | nice | 20:10 |
| mrmartin | so we should close all the user accounts, and give away write-only access for trusted people only? | 20:11 |
| *** achanda has joined #openstack-infra | 20:11 | |
| *** angdraug has quit IRC | 20:11 | |
| *** dims has joined #openstack-infra | 20:12 | |
| *** david-lyle_ is now known as david-lyle | 20:12 | |
| *** hockeynut_afk is now known as hockeynut | 20:13 | |
| pc_m | Hi all. I did a Jenkins run and one of the tests that produced a log before, is passing, but not providing any log. Any thoughts on what is going on and what I should do? | 20:13 |
| *** cznewt has joined #openstack-infra | 20:14 | |
| *** eharney_ has joined #openstack-infra | 20:15 | |
| clarkb | pc_m: it typically helps to link to the run in question | 20:15 |
| pc_m | clarkb: Sure. Was seeing if anyone would respond. https://review.openstack.org/#/c/282538/5 | 20:16 |
| fungi | yay... "Forbidden launching node id: 8386189 in provider: rax-iad error: [...] Forbidden: Quota exceeded for ram: Requested 8192, but already used 1531904 of 1536000" | 20:17 |
| fungi | so why are we over quota in rax-iad? | 20:17 |
| *** maishsk has joined #openstack-infra | 20:17 | |
| *** gtmanfred has joined #openstack-infra | 20:18 | |
| fungi | 1531904/8192=187 | 20:18 |
| *** yamahata has quit IRC | 20:18 | |
| *** exploreshaifali has joined #openstack-infra | 20:18 | |
| *** eharney has quit IRC | 20:19 | |
| fungi | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n220 says "max-servers: 190" | 20:19 |
| fungi | i wonder if they lowered our quota there? | 20:19 |
| *** _nadya_ has quit IRC | 20:20 | |
| anteaya | :( | 20:20 |
| anteaya | jroll: who would know if infra's rax quota was changed recently? | 20:20 |
| *** ZZelle_ has joined #openstack-infra | 20:20 | |
| fungi | it's been that since https://review.openstack.org/253176 raised it from 85 back in early december | 20:21 |
| jroll | anteaya: I could probably find out | 20:21 |
| anteaya | jroll: you are awesome | 20:21 |
| fungi | it's possible we've had it set too high this entire time | 20:21 |
| jroll | fungi: tenant ID? | 20:21 |
| *** sdake has quit IRC | 20:22 | |
| *** fawadkhaliq has quit IRC | 20:22 | |
| *** sshnaidm has quit IRC | 20:22 | |
| fungi | jroll: OS_PROJECT_ID=637776 | 20:23 |
| pabelanger | fungi: do you mind promoting 286189 for zuul? that is the disable pdf uploads for wiki.o.o | 20:23 |
| pabelanger | fungi: currently 2h15mins in check queue | 20:23 |
| *** doug-fish has quit IRC | 20:24 | |
| fungi | pabelanger: sounds like it just needs to be enqueued, not promoted | 20:24 |
| pabelanger | fungi: ah, didn't know right syntax | 20:24 |
| jroll | fungi: taking a look | 20:25 |
| *** nmagnezi has quit IRC | 20:26 | |
| fungi | pabelanger: it's in the gate now, and getting workers assigned | 20:26 |
| *** kzaitsev_mb has joined #openstack-infra | 20:27 | |
| jroll | fungi: that quota is accurate, I'm not sure where to find logs on quotas though :/ | 20:27 |
| fungi | anyway, i'm inclined to lower the max-servers there to 180 (effective quota of 187 minus 4 for template instances during snapshot image builds, pls a few breathing room) | 20:28 |
| pleia2 | fungi: makes sense to me | 20:29 |
| *** dims has quit IRC | 20:29 | |
| *** doug-fish has joined #openstack-infra | 20:30 | |
| *** _nadya_ has joined #openstack-infra | 20:30 | |
| anteaya | if we avoid the over quota errors I'm fine with lowering it | 20:30 |
| *** annegentle has quit IRC | 20:31 | |
| clarkb | pc_m: which test? | 20:31 |
| *** annegentle has joined #openstack-infra | 20:32 | |
| jpmaxman | Hey guys on wiki.o.o any summary update ? Sorry I had to step away for a bit. | 20:32 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Lower max-servers in rax-iad to 180 https://review.openstack.org/286262 | 20:32 |
| fungi | pleia2: anteaya: ^ | 20:32 |
| openstackgerrit | gordon chung proposed openstack/requirements: bump aodhclient upper-constraints https://review.openstack.org/286263 | 20:32 |
| pc_m | clarkb: networking-cisco-coverage | 20:32 |
| *** doug-fis_ has joined #openstack-infra | 20:32 | |
| jpmaxman | sorry guys and gals :D | 20:32 |
| *** sshnaidm has joined #openstack-infra | 20:32 | |
| fungi | pabelanger: eta 6 minutes to merge on the pdf blocking change | 20:33 |
| pc_m | clarkb: Says "File Not Found". | 20:33 |
| pc_m | clarkb: Produced output on previous versions. | 20:33 |
| *** dizquierdo has joined #openstack-infra | 20:33 | |
| *** tongli has quit IRC | 20:34 | |
| clarkb | pc_m: https://jenkins03.openstack.org/job/networking-cisco-coverage/6/console "ran out of time uploading files" | 20:34 |
| pabelanger | fungi: woot | 20:34 |
| fungi | jpmaxman: so we confirmed there are definitely compromised/throwaway lp accounts involved. we've temporarily disabled new account creation and are working on characterizing some means of making things harder for them (rejecting pdf uploads, url/pattern blacklisting is the current thrust) | 20:34 |
| clarkb | jpmaxman: a change is going in to prevent pdf uploads | 20:34 |
| *** doug-fish has quit IRC | 20:34 | |
| clarkb | jpmaxman: and I think fungi is still working to identify a pattern of users/spamming | 20:34 |
| pc_m | clarkb: Can you point me to info on how I could find the above link (so I don't have to ask)? | 20:35 |
| fungi | yeah, took a break to try to figure out what our current ci node launch errors were all about | 20:35 |
| *** keedya has quit IRC | 20:35 | |
| *** keedya has joined #openstack-infra | 20:35 | |
| clarkb | pc_m: you hvae to go to the individual jenkins masters (01-07) and look at the job logs there | 20:35 |
| *** pots72 has joined #openstack-infra | 20:36 | |
| *** salv-orlando has joined #openstack-infra | 20:36 | |
| clarkb | pc_m: they list what change triggered them and when they ran and from that you can find the one you are interested in typically | 20:36 |
| *** |-paul-| has joined #openstack-infra | 20:36 | |
| anteaya | fungi: thanks for that | 20:36 |
| fungi | 5 jobs remaining before jenkins03 has quiesced and then i can take it back down and clean it up | 20:37 |
| *** doug-fis_ has quit IRC | 20:37 | |
| *** mizar has joined #openstack-infra | 20:37 | |
| *** flongo has quit IRC | 20:37 | |
| anteaya | ack | 20:37 |
| fungi | working now on reversing our lp spammer case study openid url to an actual lp account | 20:37 |
| anteaya | oooohhhhhh | 20:38 |
| *** xyang1 has joined #openstack-infra | 20:38 | |
| *** keedya has quit IRC | 20:38 | |
| *** maishsk has quit IRC | 20:38 | |
| *** keedya has joined #openstack-infra | 20:39 | |
| *** salv-orl_ has quit IRC | 20:39 | |
| *** exploreshaifali has quit IRC | 20:39 | |
| pleia2 | off for some lunch + lunchtime errands, bbiab | 20:40 |
| *** ybathia has joined #openstack-infra | 20:40 | |
| *** Sukhdev has joined #openstack-infra | 20:40 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Disable PDF uploads by users https://review.openstack.org/286189 | 20:40 |
| pc_m | clarkb: Thanks. I'm running coverage test comparison. Similar to Rally project. seeing differences run to run with coverage reporting. :( | 20:40 |
| AJaeger | yeah, check queue is getting shorter - and the Zuul Job Queue is going down as well - so jenkins cleanup efforts have helped. Thanks! | 20:41 |
| jpmaxman | so to me the question still remains if I'm using a comrpomised/throwaway lp account am I succefully guessing / scripting the captcha or are the edits/updates being made some other way (API, etc)? | 20:41 |
| *** ybathia has quit IRC | 20:41 | |
| boris-42 | pc_m: hi | 20:41 |
| pc_m | boris-42: hi | 20:41 |
| clarkb | pc_m: keep in mind that we merge against the target branch for each run | 20:41 |
| *** eharney_ is now known as eharney | 20:41 | |
| clarkb | pc_m: so if the underlying base changes coverage can change | 20:41 |
| pabelanger | jpmaxman: right, but once we do the first round of patches / clean up, it will be easier to see where those POSTs are coming from | 20:42 |
| anteaya | pleia2: enjoy | 20:42 |
| pc_m | clarkb: On this commit, it runs HEAD vs HEAD^ (which would be the change set for this commit and recent, and this commit is only script changes). | 20:42 |
| pabelanger | I haven't see any HTTP logs, but everything appears to just be a POST to know page | 20:42 |
| clarkb | pc_m: right and HEAD^ may change on each run | 20:42 |
| clarkb | pc_m: because zuul merges the change into latest master and uses that ref for testing | 20:43 |
| jpmaxman | right but that POST could be happening as a result of a user/script hitting the submit button and properly answering the captcha - or it could be a direct post bypassing... | 20:43 |
| mtreinish | clarkb, fungi, nibalizer, pleia2: is there anything in the subunit gearman worker logs? There have been 3 results added to the db since clarkb restarted it last week | 20:43 |
| fungi | pabelanger: jpmaxman: right, i haven't been able to adequately distinguish browser-based and api-based calls from the apache logs. for example, both seem to use action=edit | 20:43 |
| jpmaxman | anyway, hopefully this next round of passes will make it more clear :D | 20:43 |
| *** pots has joined #openstack-infra | 20:43 | |
| pc_m | clarkb: See different # lines missing/partial covered. so not a difference from run to run, but on a single run. | 20:44 |
| jpmaxman | what role does disabling the PDF upload play? | 20:44 |
| clarkb | mtreinish: http://paste.openstack.org/show/488632/ | 20:45 |
| pabelanger | For the most part, the spam I am deleting is all the same. Either quickbooks or microsoft care. So, exposing the tools to allow admins to quick blacklist those keywords will also help too | 20:45 |
| pabelanger | right now, we are just flooded with users posting junk | 20:45 |
| *** maishsk has joined #openstack-infra | 20:45 | |
| fungi | pabelanger: jpmaxman: that's on my to do list is to figure whether there's even a difference in the "api" and the web browser urls, though i see evidence that the attacker retrieves the entire page after the post results in a 302, and also pulls any css and images linked in the target confirmation page. even includes a referrer from the post url, making it seem authentically browseresque | 20:45 |
| *** dprince has quit IRC | 20:45 | |
| openstackgerrit | Erno Kuvaja proposed openstack/requirements: Bump up glance_store to 0.12.0 https://review.openstack.org/286272 | 20:45 |
| jpmaxman | is the write API enabled for a specific use? | 20:45 |
| jpmaxman | You can disable all write modules by setting $wgEnableWriteAPI = false; in LocalSettings.php. The write API is enabled by default as of MediaWiki 1.14, and disabled by default in older versions. | 20:45 |
| clarkb | jpmaxman: we have a small numbre of bots that do legit updates to the wiki | 20:46 |
| pabelanger | We've plugged the new users being created, since Helen is the latest user. But I also think any new user via openid is blocked too | 20:46 |
| fungi | well, i don't mind breaking bot writes to the wiki for a little while as a troubleshooting measure | 20:46 |
| fungi | their use is infrequent and informational in nature | 20:46 |
| *** baoli has joined #openstack-infra | 20:47 | |
| fungi | obviously disabling the api indefinitely is probably not something we want, but finding a way to lock api use down to teh bots account group would be cool | 20:47 |
| mtreinish | clarkb: hmm, a None is getting passed into subunit2sql for the stream | 20:47 |
| jpmaxman | again I think any of these efforts would be great troubleshooting steps. | 20:47 |
| mtreinish | clarkb: is that just once or a bunch of them? | 20:47 |
| jpmaxman | disabling write api to see if it does anything | 20:47 |
| jpmaxman | would be telling for sure | 20:48 |
| *** baoli has quit IRC | 20:48 | |
| clarkb | right the question was is it enabled for a reason, that is the reason. I don't care if we lock it down temporarily | 20:48 |
| clarkb | if I hda more time to devote to this I owuld just make the entire wiki read only | 20:48 |
| *** baoli has joined #openstack-infra | 20:48 | |
| clarkb | and undo that in a week and see if they have moved on | 20:48 |
| *** jsavak has quit IRC | 20:48 | |
| jpmaxman | kind of like an impossible to answer captcha question ;) | 20:49 |
| jpmaxman | but they would be back | 20:49 |
| clarkb | a few meeting agendas would suffer but other than that I don't think it would hurt us too badly | 20:49 |
| jpmaxman | IMHO we need to find how they're getting in and plug the hole | 20:49 |
| *** ociuhandu has quit IRC | 20:49 | |
| pabelanger | well, right now it is only existing users that are spamming. Since we stopped new | 20:49 |
| jpmaxman | but I think we need to iterate faster on enabling disabling | 20:49 |
| pabelanger | and I am simply blocking them as they create new pages | 20:49 |
| jpmaxman | the nice part is they are hitting it so hard | 20:49 |
| jpmaxman | you will know in a few minutes if it works :D | 20:50 |
| mrmartin | I'll do some pattern analysis tomorrow if we don't have an ultimate solution until next morning. | 20:50 |
| jpmaxman | I mean here https://wiki.openstack.org/w/index.php?title=Special:RecentChanges&limit=50 | 20:51 |
| jpmaxman | it is literally 20:49 pabeleanger deletes | 20:51 |
| jpmaxman | 20:50 new spam | 20:51 |
| pabelanger | ya, but if you look at their contribs, they are old accounts. So, mass delete is nuking a fair bunch of stuff ATM | 20:52 |
| pabelanger | we still haven't landed the password auth disable | 20:52 |
| pabelanger | that is in the gate | 20:52 |
| jpmaxman | yup | 20:52 |
| pabelanger | we could enqueue that too | 20:52 |
| fungi | pabelanger: project name, change id and patchset number? | 20:52 |
| jpmaxman | was just more commenting on the speed | 20:52 |
| fungi | i can kick it over now | 20:52 |
| pabelanger | fungi: 285669 | 20:53 |
| pabelanger | from what I see, it is file uploads ATM | 20:53 |
| pabelanger | or existing pages | 20:53 |
| pabelanger | new page creation is down | 20:53 |
| pabelanger | 19:19 UTC was last new page, and that was l18n team | 20:53 |
| *** IlyaG has joined #openstack-infra | 20:54 | |
| *** jsavak has joined #openstack-infra | 20:54 | |
| fungi | pabelanger: so openstack-infra/puppet-mediawiki 285669,2 looks like, but we need a second core reviewer | 20:54 |
| pabelanger | Ah, right | 20:54 |
| fungi | pleia2: you had reviewed that one previously, looks like | 20:54 |
| pabelanger | that will break StatusBot I believe | 20:54 |
| pabelanger | but maybe we just notify people of the breakage | 20:55 |
| fungi | pabelanger: break it, or just prevent it from successfully uploading the log on the wiki? | 20:55 |
| *** Sukhdev has quit IRC | 20:55 | |
| pabelanger | fungi: I believe it won't post to the wiki | 20:55 |
| fungi | er, updating the log on the wiki | 20:55 |
| fungi | so i feel like wiki updating features of #status and #success are probably okay to temporarily suspend while we're working on this as long as we're sure to solve for them soon | 20:56 |
| pabelanger | I started looking at the code last night, but didn't figure out how it auth'd. | 20:56 |
| pabelanger | I don't think it would be hard to convert it to openid | 20:57 |
| *** IlyaG has quit IRC | 20:57 | |
| pabelanger | or API key (if that exists) | 20:57 |
| clarkb | I think it auths with an api key like thing today | 20:57 |
| pabelanger | or allow password auth based on user group | 20:57 |
| pabelanger | okay, so then it might still work | 20:57 |
| *** angdraug has joined #openstack-infra | 20:59 | |
| tonyb | lifeless: Are you able to look at the fixtures thing? The neutron gate is still blocked and it'd be good to do or anadon the fixtures path so we can work on alternates if required. | 21:00 |
| tonyb | lifeless: sorry to keep hasseling you .... | 21:00 |
| *** chlong_ has joined #openstack-infra | 21:01 | |
| fungi | okay, bad news. the openid for Helen is somehow blocked from reverse lookup in lp i think. if i pass in my openid url (connecting with our gerrit lp creds on review.o.o) it returns the url to my lp account, but if i pass in Helen's openid url i get back a None object from people.getByOpenIDIdentifier() | 21:02 |
| anteaya | :( | 21:03 |
| fungi | though that may also just mean that you can create a login.launchpad.net account which doesn't map back to a launchpad.net account | 21:03 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of ovs-vsctl show from hosting node https://review.openstack.org/286279 | 21:03 |
| anteaya | possibly | 21:03 |
| fungi | given that they've been described as distinct and unrelated systems | 21:03 |
| clarkb | fungi: could be an ubuntu one only account | 21:03 |
| clarkb | right that | 21:03 |
| *** davideagnello has quit IRC | 21:03 | |
| *** davideagnello has joined #openstack-infra | 21:04 | |
| fungi | though i'll go ahead and push up this lookup utility script anyway for future use | 21:04 |
| clarkb | jenkins came back +1 on https://review.openstack.org/#/c/285473/4 finally, if I could get that approved real soon now that would be great (it is first step in starting to use osic for testing) | 21:05 |
| clarkb | once that is in I will fix security groups and attempt launching a mirror host | 21:05 |
| *** sputnik13 has quit IRC | 21:05 | |
| pabelanger | fungi: interesting, maybe something long term is to expect wiki.o.o users to have created accounts in review.o.o | 21:06 |
| anteaya | pabelanger: or at least foundation accounts | 21:06 |
| pabelanger | anteaya: right | 21:06 |
| clarkb | pabelanger: or maybe need to explicitly ack users befor ethey get write access? | 21:07 |
| *** sputnik13 has joined #openstack-infra | 21:07 | |
| anteaya | as not every person who wants to offer a wiki page/edit understands gerrit | 21:07 |
| pabelanger | I think somebody suggested openstackid.org but unsure what the migration to that looks like | 21:07 |
| clarkb | (that is a lot of work for humans though) | 21:07 |
| anteaya | clarkb: yeah, it would be | 21:07 |
| pabelanger | clarkb: right, I was looking at some sort of moderation extensions, but not sure humans want to do that | 21:07 |
| anteaya | but we could enact it as spam measures | 21:07 |
| pabelanger | https://www.mediawiki.org/wiki/Extension:Moderation | 21:07 |
| clarkb | fungi: we can ask isd-support@canonical.com for help on the canonical/ubuntu one side | 21:08 |
| pabelanger | if we did moderation, then captcha goes away | 21:08 |
| anteaya | so not having a human approve during normal use, but bring it in in times of spam | 21:08 |
| *** salv-orlando has quit IRC | 21:08 | |
| *** amrith is now known as _amrith_ | 21:08 | |
| anteaya | how many wiki users do we have? | 21:08 |
| anteaya | any ideas? | 21:08 |
| mrmartin | if they could filter spam accounts, our case can be much simpler. | 21:08 |
| *** salv-orlando has joined #openstack-infra | 21:09 | |
| anteaya | and what is the rate of creation during non-spam times | 21:09 |
| clarkb | anteaya: if id numbers are an indication ~7k | 21:09 |
| fungi | pabelanger: worth considering, but though i suspect the page update latency might be terrible if we had every edit moderated | 21:09 |
| mrmartin | 7787 | 21:09 |
| anteaya | clarkb: okay, that's not bad | 21:09 |
| anteaya | have we any kind of rate of creation numbers from say, last month? | 21:09 |
| pabelanger | fungi: maybe some promotion system. First edit is moderated, if valid you go into good users and by pass moderation | 21:09 |
| *** baoli_ has joined #openstack-infra | 21:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a simple LP OpenId reverse lookup utility https://review.openstack.org/286284 | 21:10 |
| mrmartin | anteaya: 1351 new user in the last month | 21:10 |
| *** baoli_ has quit IRC | 21:10 | |
| fungi | pabelanger: these also seem like longer-term solutions, since the availability, efficacy and security of existing extensions for this are likely quite different after we upgrade mediawiki, which means after we move it to ubuntu trusty | 21:11 |
| anteaya | mrmartin: yeah okay so, about 6K up until now | 21:11 |
| pabelanger | fungi: agreed | 21:11 |
| anteaya | about the same as gerrit, I think | 21:11 |
| mrmartin | 1351 new | 21:11 |
| anteaya | I wonder if there is anything that would track the rate of new user creation? | 21:12 |
| *** _nadya_ has quit IRC | 21:12 | |
| fungi | we could trivially graph it from the account registration timestamps in the openstack_wiki.user table | 21:13 |
| ianw | where are the rules about what goes on docs.openstack.org ? | 21:13 |
| *** baoli_ has joined #openstack-infra | 21:13 | |
| fungi | ianw: documentation for official (tc-recognized) project-team deliverables | 21:13 |
| fungi | and documentation for other efforts by official project-teams | 21:13 |
| *** baoli has quit IRC | 21:14 | |
| *** ybathia has joined #openstack-infra | 21:14 | |
| fungi | ianw: so basically, no publishing documentation for unofficial teams is about the only rule i guess, and i don't think that's explicitly documented anywhere | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| ianw | fungi: right, so something like "bashate" falls where? and gerrit-dashboard-creator? | 21:15 |
| *** amitgandhinz has quit IRC | 21:15 | |
| ianw | i don't think they're official, but widely used | 21:15 |
| *** bpokorny has quit IRC | 21:16 | |
| *** rfolco_ has quit IRC | 21:16 | |
| openstackgerrit | Tim Buckley proposed openstack-infra/devstack-gate: Install and run StackViz during devstack-gate host cleanup. (WIP) https://review.openstack.org/212207 | 21:16 |
| fungi | ianw: according to http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml bashate is maintained by the qa team | 21:16 |
| mrmartin | I'm leaving now, trying to do something tomorrow that helps the find infected openid accounts. | 21:16 |
| anteaya | mrmartin: thanks for your help today | 21:16 |
| fungi | mrmartin: thanks for your assistance, and have a great evening! | 21:17 |
| *** ayoung has joined #openstack-infra | 21:17 | |
| mrmartin | thnx, bye | 21:17 |
| *** mrmartin has quit IRC | 21:17 | |
| fungi | ianw: i agree that gerrit-dashboard-creator is not listed there | 21:17 |
| openstackgerrit | Tim Buckley proposed openstack-infra/subunit2sql: Add API methods for getting tests by prefix https://review.openstack.org/283334 | 21:17 |
| *** hashar has quit IRC | 21:18 | |
| fungi | ianw: so presumably an unofficial project, though could either ask the tc to make its maintainers an official team or ally with an existing official team | 21:18 |
| anteaya | ianw: if it can't go on docs.o.o it goes to readthedocs: https://readthedocs.org/ | 21:18 |
| fungi | well, rtd is a common solution to the problem, but it's far from the only option | 21:18 |
| anteaya | okay fair | 21:18 |
| anteaya | I don't think I've seen docs go anywhere else | 21:18 |
| *** cloudtrainme has quit IRC | 21:19 | |
| anteaya | gerrit-dash-creator is sdague's https://review.openstack.org/#/admin/groups/362,members | 21:19 |
| fungi | there are plenty of webservers out there which you could use to host documentation | 21:19 |
| *** jsavak has quit IRC | 21:19 | |
| ianw | ok, cool, yeah i started a change to fix dash-creator for rtd | 21:19 |
| *** jsavak has joined #openstack-infra | 21:19 | |
| ianw | that one would be particularly useful to have documentation, because each dashboard actually builds itself into a useful info page | 21:20 |
| fungi | ianw: as anteaya points out, you might want to confer with sdague to see if he has a preference | 21:20 |
| ianw | maybe we can stick it under qa | 21:20 |
| ianw | yeah, will do | 21:20 |
| *** kgiusti has left #openstack-infra | 21:20 | |
| fungi | his unoffical project, so his rules | 21:20 |
| fungi | er, unofficial | 21:21 |
| *** cloudtrainme has joined #openstack-infra | 21:21 | |
| fungi | okay, where was i? | 21:21 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: * Added support for the Jenkins Slack plugin https://review.openstack.org/266970 | 21:22 |
| pabelanger | https://wiki.openstack.org/w/index.php?title=Special:RecentChanges is looking real good ATM | 21:22 |
| pabelanger | only my actions for the last hour really | 21:22 |
| fungi | any other infra-core or infra-puppet-core reviewers in favor of temporarily disabling password authentication for wiki accounts (now that new password-authed ones can't be created by non-admins) while we work through cleaning them up? https://review.openstack.org/285669 | 21:23 |
| *** dims has joined #openstack-infra | 21:23 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of brctl show from hosting node https://review.openstack.org/286279 | 21:23 |
| *** aeng has joined #openstack-infra | 21:23 | |
| *** hashar has joined #openstack-infra | 21:24 | |
| fungi | tripleo, your container jobs take an excessive amount of time to complete | 21:25 |
| clarkb | fungi: looking | 21:25 |
| fungi | "Build has been executing for 2 hr 27 min" | 21:25 |
| clarkb | approved | 21:26 |
| *** kingia has quit IRC | 21:28 | |
| *** lucas-dinner has quit IRC | 21:28 | |
| *** bpokorny has joined #openstack-infra | 21:29 | |
| *** yamahata has joined #openstack-infra | 21:29 | |
| *** davideagnello has quit IRC | 21:32 | |
| pabelanger | fungi: are you okay with me blocking helen or do you want to keep using her account to collect data? | 21:32 |
| *** esker has joined #openstack-infra | 21:32 | |
| fungi | pabelanger: all the info i could possibly collect is not going to be hapmered by blocking that account | 21:32 |
| pabelanger | fungi: roger | 21:32 |
| fungi | so go for it | 21:32 |
| anteaya | my grandmother's name was Helen | 21:32 |
| anteaya | I keep picturing her at her dressing table with a computer hammering away and cackling to herself | 21:33 |
| anteaya | then closing the laptop and offering tea | 21:33 |
| anteaya | oh Helen | 21:33 |
| *** flepied has quit IRC | 21:33 | |
| sc68cal | lol | 21:34 |
| anteaya | sc68cal: tea? | 21:34 |
| *** davideagnello has joined #openstack-infra | 21:34 | |
| sc68cal | anteaya: why thank you :) | 21:34 |
| smarcet | fungi: could u approve https://review.openstack.org/#/c/285475/? i need it to fix openstackid-dev :) | 21:35 |
| *** aysyd has quit IRC | 21:35 | |
| *** ociuhandu has joined #openstack-infra | 21:35 | |
| *** stevebaker has quit IRC | 21:35 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Disable standard password based auth https://review.openstack.org/285669 | 21:36 |
| *** amitgandhinz has joined #openstack-infra | 21:36 | |
| *** lucasagomes has joined #openstack-infra | 21:36 | |
| *** kingia has joined #openstack-infra | 21:36 | |
| *** stevebaker has joined #openstack-infra | 21:36 | |
| *** tpsilva has quit IRC | 21:37 | |
| *** abregman is now known as abregman|afk | 21:37 | |
| anteaya | sc68cal: I've put the kettle on | 21:38 |
| fungi | smarcet: i would have liked to see a second core reviewer on that one, but it's trivial enough that i feel marginally okay single-core approving since you're under a production deadline rush | 21:38 |
| fungi | and it's been sitting there most of the day with just my review | 21:38 |
| *** dims has quit IRC | 21:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Skip dsvm jobs on cinder docs/reno/unit test changes https://review.openstack.org/286180 | 21:40 |
| *** arxcruz has joined #openstack-infra | 21:42 | |
| smarcet | fungi: thx a lot :) | 21:44 |
| *** dims has joined #openstack-infra | 21:44 | |
| *** smarcet has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Update App Configuration https://review.openstack.org/285475 | 21:47 |
| fungi | pabelanger: since 285669 merged a couple minutes ago, do you feel up to writing a message to the -dev ml letting people know we've temporarily broken #success while trying to work through the wiki spam problem, and that we'll announce again once it's back in working order? | 21:48 |
| *** |-paul-| has quit IRC | 21:48 | |
| fungi | though i guess test a #success first to make sure it really is broken | 21:48 |
| *** baoli_ has quit IRC | 21:48 | |
| fungi | once it ends up on the wiki server | 21:48 |
| fungi | okay, jenkins03 finally just went idle, so cleaning it up now | 21:49 |
| pabelanger | fungi: sure | 21:49 |
| jhesketh | Morning | 21:49 |
| anteaya | morning jhesketh | 21:49 |
| pabelanger | #success we are combating the spam on wiki.o.o | 21:49 |
| openstackstatus | pabelanger: Added success to Success page | 21:49 |
| *** maishsk has quit IRC | 21:49 | |
| pabelanger | fungi: https://wiki.openstack.org/wiki/Successes still working | 21:50 |
| pabelanger | will try again in a few minuts | 21:50 |
| pabelanger | minutes* | 21:50 |
| pabelanger | but agree, a ML post would be good | 21:50 |
| *** fawadkhaliq has joined #openstack-infra | 21:50 | |
| anteaya | jhesketh: so you restarted jenkins 03 for us last night, thank you | 21:52 |
| fungi | okay, jenkins03 is cleaned up and on its way back up now | 21:53 |
| jhesketh | anteaya: yes but it appears to be still having trouble? | 21:53 |
| *** armax has joined #openstack-infra | 21:53 | |
| anteaya | jhesketh: so there are some points about restarting a jenkins in order to ensure it removes all its old nodes and uses zmq to communicate with nodepool | 21:53 |
| *** edmondsw has quit IRC | 21:53 | |
| * jhesketh is still catching up | 21:53 | |
| anteaya | well I'm light on details | 21:53 |
| fungi | jhesketh: i'm guessing you didn't know to delete the slaves from its config.xml | 21:53 |
| anteaya | but yes both the server you restarted and the server yolanda restarted need restarting again | 21:53 |
| fungi | the zmq issue with with jenkins02 and i have a feeling that one was because of not making sure the java processes were completely dead | 21:54 |
| jhesketh | fungi: ah no, I thought removing them from nodepool was enough | 21:54 |
| jhesketh | Sorry :-( | 21:54 |
| fungi | no problem. we'll try to get something written up a little more step-by-step | 21:54 |
| fungi | basically if you nodepool delete while jenkins is offline, it never gets cleaned up in jenkins and so it ends up with numerous dead slaves it thinks should be reachable but aren't | 21:55 |
| anteaya | fungi: ah sorry | 21:55 |
| *** hashar has quit IRC | 21:55 | |
| *** jpr has quit IRC | 21:56 | |
| jhesketh | fungi: jenkins02 had trouble too? | 21:56 |
| fungi | looks like jenkins03 already has new workers assigned and running jobs, so should be all set | 21:56 |
| anteaya | yes after you were asleep yolanda addressed 02 and 04 | 21:56 |
| anteaya | asleep/offline | 21:56 |
| pabelanger | fungi: so, pdf block looks good. Just seen a spammer switch to jpeg upload. But that's been the first spammer in a few hours now | 21:56 |
| fungi | pabelanger: fun! | 21:56 |
| jpmaxman | +1 | 21:57 |
| anteaya | pabelanger: do we allow jpeg upload? | 21:57 |
| pabelanger | anteaya: Ya, there is some default IIRC | 21:57 |
| jhesketh | fungi: so why did 02 and 04 need restarting? | 21:57 |
| fungi | jhesketh: thread leak sounded like | 21:57 |
| jhesketh | Right okay, so a coincidence rather than due to me screwing up 03 | 21:58 |
| anteaya | pabelanger: can you see how many spammers we have? or is it still a crowd? | 21:58 |
| anteaya | jhesketh: yeah, doesn't appear related | 21:58 |
| sc68cal | those spammers are persistent bastards | 21:58 |
| fungi | jhesketh: we have a scripted restart of them weekly to keep that under control, but it seems like they are at the point where that's not frequent enough and they go too far out to lunch sich that the restart script can't safely restart them because their apis are too unresponsive already | 21:58 |
| anteaya | I think all the jenkins are independent of each other | 21:58 |
| anteaya | sc68cal: I have no opinion on their geneology | 21:58 |
| *** salv-orl_ has joined #openstack-infra | 21:59 | |
| fungi | anteaya: reasonably independent, but problems on some can spike load onto others and create new problems for them | 21:59 |
| anteaya | ah, thank you | 21:59 |
| anteaya | sc68cal: tea is ready | 21:59 |
| jhesketh | When I left 03 was picking up jobs again. What is the consequence of it trying to connect to dead nodes that weren't removed from config.xml? | 21:59 |
| fungi | like if we were to knock half our jenkins masters offline at peak load, the remaining half with rapidly degrade because they're incapable of handling all the work on their own | 22:00 |
| anteaya | fair enough | 22:00 |
| jhesketh | fungi: right. I had tried to use those plays but they failed | 22:00 |
| fungi | jhesketh: jenkins quickly recognizes they're unreachable and marks them offline on its own, they just never get cleaned up | 22:00 |
| sc68cal | anteaya: ah thank you. used the electric kettle at indy hall, going to buy myself one for home | 22:00 |
| anteaya | nice | 22:01 |
| anteaya | where is indy hall? | 22:01 |
| anteaya | I used an electric kettle | 22:01 |
| anteaya | sunbeam | 22:01 |
| sc68cal | 3rd and market - philly | 22:01 |
| fungi | jhesketh: so jenkins03 had some hundreds of "offline" slaves it was periodically checking to see if they were ever coming back | 22:01 |
| anteaya | has a pretty blue light this one | 22:01 |
| anteaya | sc68cal: nice | 22:01 |
| anteaya | jenkins03 was so hopeful | 22:01 |
| sc68cal | the one at indy hall is pretty advanced - Bonavita variable temperature | 22:01 |
| fungi | i've now reset its expectations to mostly match reality | 22:02 |
| *** salv-orlando has quit IRC | 22:02 | |
| *** annegentle has quit IRC | 22:02 | |
| anteaya | best anyone can hope for | 22:02 |
| jhesketh | fungi: ah so it just ties 03 up. Did it cause further gate issues (ie were the other masters unable to keep up? ) | 22:02 |
| sc68cal | it's got like 10 buttons on it. It's like openstack | 22:02 |
| sc68cal | sorry - 6 | 22:02 |
| anteaya | sc68cal: wow | 22:02 |
| fungi | jhesketh: it wasn't disruptive to the overall system that i could see, no | 22:03 |
| anteaya | can you convince it to boil water? | 22:03 |
| fungi | so no real harm done | 22:03 |
| jhesketh | Okay that's good. | 22:03 |
| *** kencjohnston has quit IRC | 22:03 | |
| sc68cal | anteaya: I kid you not, I just pressed all the buttons until something happened | 22:03 |
| anteaya | sc68cal: sounds like openstack | 22:03 |
| sc68cal | exactly | 22:03 |
| anteaya | ha ha ha | 22:03 |
| * anteaya sips her tea | 22:03 | |
| anteaya | one button, I have full confidence in it | 22:04 |
| fungi | the issue with jenkins02 was more troublesome since it was causing lots of in-use and offline nodes to look like they were satisfying ready node demand, and also sucking up quota because nodepool didn't know to clean them up | 22:04 |
| jhesketh | fungi: the real harm from yesterday was done by me though sorry :-(. In fixing why ansible wasn't running I inadvertently caused nodepool to cease working | 22:04 |
| fungi | jhesketh: yep, saw that, but you fixed it | 22:04 |
| fungi | so thanks for noticing and correcting | 22:05 |
| jhesketh | I had removed Infra cloud from the clouds.yaml which nodepool didn't like. | 22:05 |
| jhesketh | Yeah I should have noticed it much sooner though :-( | 22:05 |
| fungi | anyway, since i've gone all day too busy to eat something, i'm going to disappear and grab a meal, then hopefully back for more fun in an hour or so | 22:05 |
| *** sdake has joined #openstack-infra | 22:05 | |
| jhesketh | (And it was AJaeger that pointed it out ) | 22:05 |
| anteaya | fungi: enjoy food | 22:05 |
| anteaya | jhesketh: he is so helpful that way | 22:06 |
| *** doug-fish has joined #openstack-infra | 22:06 | |
| jhesketh | fungi: okay. Thanks for all you do! | 22:06 |
| anteaya | and supportive as you fix it, I'm guessing too | 22:06 |
| jhesketh | Let me know if I can help | 22:06 |
| jhesketh | anteaya: yeah he's pretty awesome | 22:06 |
| anteaya | indeed | 22:06 |
| anteaya | jhesketh: current issue is wiki spam | 22:06 |
| anteaya | there is an etherpad: https://etherpad.openstack.org/p/wiki.openstack.org | 22:07 |
| anteaya | pabelanger: does the etherpad reflect the latest reality? | 22:07 |
| pabelanger | okay, stepping away from desk. All this wiki stuff is getting to me. But good news, the amount of new spam is almost zero in the last few hours | 22:07 |
| pabelanger | anteaya: no, I will update shortly after walk | 22:07 |
| anteaya | okay thanks, enjoy your walk | 22:08 |
| *** sarob has quit IRC | 22:08 | |
| ianw | fungi: so i think why i got myself confused is that the jobs look all correct, but no bueno http://docs.openstack.org/developer/bashate ... i bet it has to do with "openstack-dev/" | 22:08 |
| ianw | cause devstack we use custom jobs | 22:08 |
| anteaya | ianw: yes I think you are correct | 22:08 |
| anteaya | you can set the path in a specific variable | 22:09 |
| *** dims has quit IRC | 22:09 | |
| *** baoli has joined #openstack-infra | 22:10 | |
| jhesketh | anteaya: okay. Sounds like pabelanger has made some great progress. I'll wait for his updates to the pad | 22:10 |
| anteaya | makes sense to me | 22:11 |
| anteaya | thanks | 22:11 |
| *** samueldmq has joined #openstack-infra | 22:13 | |
| *** dkranz has quit IRC | 22:14 | |
| *** _ody__ is now known as _ody | 22:14 | |
| *** phschwartz_ is now known as phschwartz | 22:14 | |
| *** sdake has quit IRC | 22:14 | |
| samueldmq | hi, there is an approved keystone change in which all gates are presenting 'skipped' status | 22:15 |
| samueldmq | it's patch 231289 | 22:15 |
| *** jsavak has quit IRC | 22:15 | |
| samueldmq | what does that mean? | 22:15 |
| *** jsavak has joined #openstack-infra | 22:15 | |
| openstackgerrit | Yuriy Taraday proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests https://review.openstack.org/285620 | 22:16 |
| *** sdague has quit IRC | 22:18 | |
| anteaya | samueldmq: okay it is in the gate pipeline and has a black dot beside it | 22:18 |
| anteaya | samueldmq: hover over the black dot | 22:18 |
| *** sdake has joined #openstack-infra | 22:18 | |
| anteaya | it tells me it has failed because it has a merge conflict | 22:18 |
| *** yamahata has quit IRC | 22:18 | |
| samueldmq | anteaya: perfect! merge conflict | 22:19 |
| anteaya | so somewhere above it since it passed check a merge conflict got in | 22:19 |
| *** alivigni has quit IRC | 22:19 | |
| anteaya | zuul saw it and pulled it out of the gate queue since it can't merge | 22:19 |
| *** yamahata has joined #openstack-infra | 22:19 | |
| anteaya | but leave it for now since if the gate resets it will get retested | 22:19 |
| *** armax has quit IRC | 22:19 | |
| anteaya | and it is possible the patch with the merge conflict may not merge | 22:19 |
| anteaya | we don't know yet | 22:20 |
| samueldmq | anteaya: makes sense, we know of other patch that was merging and would cause the merge conflict | 22:20 |
| anteaya | ah | 22:20 |
| anteaya | that would be it then | 22:20 |
| samueldmq | anteaya: so that's clear now for us why this is giving the error | 22:20 |
| *** cloudtrainme has quit IRC | 22:20 | |
| anteaya | great | 22:20 |
| mtreinish | clarkb: so was that error in the log repeating itself or was it just a one off? | 22:21 |
| samueldmq | anteaya: thanks for your help; appreciated | 22:21 |
| mtreinish | clarkb: oh, actually I just found the bug in the script | 22:21 |
| openstackgerrit | Yuriy Taraday proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests https://review.openstack.org/285620 | 22:21 |
| anteaya | samueldmq: pleasure, thanks for the continued great questions | 22:21 |
| EmilienM | fungi: just fyi https://review.openstack.org/#/c/285542/ | 22:21 |
| samueldmq | anteaya: always trying my best :) | 22:21 |
| anteaya | you are doing a great job | 22:21 |
| crinkle | jeblair: clarkb when you have a moment today could you give your feedback on the infracloud network requirements thread? | 22:22 |
| greghaynes | crinkle: that LGTM | 22:23 |
| *** claudiub has quit IRC | 22:23 | |
| crinkle | greghaynes: go away and take care of your child :P | 22:24 |
| greghaynes | crinkle: oh, actually, one detail - the 'management network' needs to be reachable from the nodes we have | 22:24 |
| greghaynes | crinkle: crap, right! | 22:24 |
| *** sarob has joined #openstack-infra | 22:25 | |
| *** rhallisey has quit IRC | 22:25 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Fix bug on missing subunit2sql data https://review.openstack.org/286304 | 22:26 |
| mtreinish | clarkb, fungi: ^^^ | 22:26 |
| *** armax has joined #openstack-infra | 22:26 | |
| mtreinish | that should fix the subunit gearman worker | 22:26 |
| *** eharney has quit IRC | 22:26 | |
| *** abregman|afk has quit IRC | 22:29 | |
| keedya | clarkb is this review https://review.openstack.org/#/c/284827/ need any more modifications ? | 22:29 |
| *** flepied has joined #openstack-infra | 22:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Publish bashate docs https://review.openstack.org/286305 | 22:30 |
| ianw | anteaya: ^ upon further, further investigation | 22:30 |
| * anteaya looks at 286305 | 22:31 | |
| anteaya | the repo has docs.openstack.org as its publish to site: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n314 | 22:33 |
| prometheanfire | ianw: can you +w this? https://review.openstack.org/#/c/281960/ | 22:34 |
| ianw | anteaya: yep, but {name}-docs is the job that does the publishing, which you have to pull in via "openstack-*-publish-jobs" | 22:34 |
| prometheanfire | ianw: it has two +2 (one from you :P) | 22:34 |
| mmedvede | jhesketh: around? I have a question on zuul swift upload | 22:34 |
| *** reed_ has joined #openstack-infra | 22:35 | |
| jhesketh | mmedvede: sure | 22:35 |
| ianw | SpamapS: any reason you didn't workflow 281960 ^ ? | 22:35 |
| *** doug-fis_ has joined #openstack-infra | 22:35 | |
| anteaya | ianw: I'd argue for infra-publish-jobs: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n484 | 22:35 |
| anteaya | ianw: as that gives you check, gate and post | 22:35 |
| *** thorst has quit IRC | 22:35 | |
| anteaya | whereas client just gives you release | 22:35 |
| jeblair | crinkle: that lgtm | 22:35 |
| anteaya | does bashate release? | 22:36 |
| prometheanfire | he did ask that question, but dunno why that would bar a workflow | 22:36 |
| crinkle | jeblair: ty | 22:36 |
| mmedvede | jhesketh: so I do not think swift formpost middleware does support setting content-encoding. I was wondering if you thought how to handle that, or maybe there is already solution. | 22:36 |
| ianw | anteaya: that puts things under "docs.openstack.org/infra" | 22:37 |
| mmedvede | jhesketh: I am actively using zuul swift upload, but have to hack around to set content-encoding for compressed text, so that browser opens it properly. | 22:37 |
| *** doug-fish has quit IRC | 22:37 | |
| *** Scalefab has joined #openstack-infra | 22:37 | |
| anteaya | ianw: oh, hmmm | 22:38 |
| anteaya | okay well if bashate releases then I guess client could work | 22:38 |
| ianw | anteaya: yeah, we just tag new releases. i started down this path because i thought i should probably add release notes :) | 22:39 |
| *** julim has quit IRC | 22:39 | |
| jhesketh | mmedvede: hmm not sure if the middleware supports it or not (haven't looked). We haven't ran into it because we use os_loganalyze and can set our own headers there | 22:39 |
| jhesketh | mmedvede: what are you serving the content from | 22:39 |
| anteaya | ianw: okay then looks like it should work | 22:39 |
| anteaya | yay release notes | 22:39 |
| anteaya | thanks for your patience with me | 22:40 |
| mmedvede | jhesketh: we serve it from swift container itself | 22:40 |
| jhesketh | mmedvede: so with some kind of cdn? | 22:40 |
| keedya | anteaya: would you be able to review my pull request : https://review.openstack.org/#/c/284827/ | 22:40 |
| *** _amrith_ is now known as amrith | 22:41 | |
| *** baoli has quit IRC | 22:41 | |
| *** thorst has joined #openstack-infra | 22:42 | |
| mmedvede | jhesketh: I do not think there is cdn involved. But your answer is good enough. I thought maybe I was missing something obvious. | 22:42 |
| *** rockyg has quit IRC | 22:42 | |
| *** cj has joined #openstack-infra | 22:43 | |
| cj | hey folks | 22:43 |
| cj | I'm having a heck of a time with gerritbot | 22:43 |
| cj | mostly because I can't get the test target to run | 22:44 |
| *** ybathia has quit IRC | 22:44 | |
| cj | which in turn is because of pbr throwing the «TypeError: dist must be a Distribution instance» error | 22:44 |
| *** rockyg has joined #openstack-infra | 22:45 | |
| openstackgerrit | Dmitry Borodaenko proposed openstack-infra/project-config: Added new repository for fuel-plugin-murano https://review.openstack.org/269567 | 22:45 |
| *** regXboi has quit IRC | 22:46 | |
| *** thorst has quit IRC | 22:46 | |
| *** sdake has quit IRC | 22:46 | |
| *** doug-fis_ has quit IRC | 22:48 | |
| *** doug-fish has joined #openstack-infra | 22:48 | |
| anteaya | cj: what are you trying to do with gerritbot? | 22:49 |
| cj | anteaya: get the tests to pass, get one running for the fdio channels, replace openstackgerrit, etc. | 22:49 |
| cj | They currently crash daily and leave no log messages | 22:50 |
| *** tiswanso has quit IRC | 22:50 | |
| anteaya | I'm sorry I'm still lacking context | 22:50 |
| anteaya | you are trying to run your own instance of gerritbot? | 22:50 |
| *** jsavak has quit IRC | 22:50 | |
| cj | anteaya: | 22:50 |
| cj | (gerritbot)cjac@build0:/usr/src/git/gerritbot$ pwd | 22:50 |
| cj | /usr/src/git/gerritbot | 22:50 |
| cj | (gerritbot)cjac@build0:/usr/src/git/gerritbot$ git describe | 22:50 |
| cj | 0.2.0-11-gd48b1bc | 22:50 |
| cj | (gerritbot)cjac@build0:/usr/src/git/gerritbot$ python setup.py test -v > /dev/null 2>&1 ; echo $? | 22:50 |
| cj | 1 | 22:50 |
| anteaya | cj: please use a paste service | 22:50 |
| cj | anteaya: for some value of "my own" | 22:51 |
| cj | where tha t value is "your own" | 22:51 |
| anteaya | sorry I'm still lacking context | 22:51 |
| anteaya | and please use a paste service for logs and traces | 22:51 |
| *** dizquierdo has quit IRC | 22:52 | |
| cj | okay. I'm on the LF release engineering team. The gerrit bots are hard to maintain, do not print errors to logs and fail silently. | 22:52 |
| cj | because of this, we need to re-start the bot daily | 22:52 |
| *** Scalefab has quit IRC | 22:52 | |
| *** doug-fish has quit IRC | 22:53 | |
| cj | I want to patch it up and improve the test suite so that we can identify the problem, fix it and ensure that it stays fixed. | 22:53 |
| anteaya | what is the LF release? | 22:53 |
| *** doug-fish has joined #openstack-infra | 22:53 | |
| cj | anteaya: I'm not sure. I am a member of the Release Engineering team at The Linux Foundation | 22:54 |
| cj | I maintain services for projects hosted with The Linux Foundation. | 22:54 |
| cj | Including gerritbot instances | 22:54 |
| *** achanda has quit IRC | 22:54 | |
| cj | such as openstackgerrit, for instance | 22:55 |
| *** amitgandhinz has quit IRC | 22:55 | |
| anteaya | oh the linux foundation | 22:55 |
| anteaya | okay great | 22:55 |
| anteaya | well we aren't all that happy with our bots at the moment either | 22:55 |
| anteaya | for many of the same reasons you outline | 22:55 |
| cj | Great, so there is some empitus to get it resolved. | 22:56 |
| pleia2 | cj: has the problem been happening since you started running the bots, or just recently? | 22:56 |
| jeblair | cj: i don't think gerritbot has any tests... | 22:56 |
| jeblair | cj: but if it did, "tox -e py27" would be the way to run them | 22:56 |
| cj | pleia2: for quite a while, from what I hear. I just started 3 weeks ago, and I've been tasked with standing one up. It won't start, so the ticket is open until I fix that problem, specifically. | 22:56 |
| pleia2 | cj: ah, gotcha | 22:57 |
| pleia2 | cj: I met one of your colleagues at linux.conf.au recently, trying to convince you all to open source more of your ops ;) | 22:57 |
| *** baoli has joined #openstack-infra | 22:57 | |
| pleia2 | (helpfully I completely forgot his name) | 22:58 |
| cj | that won't take much convincing. Who did you speak with? | 22:58 |
| cj | ha. | 22:58 |
| cj | Andrew Grimberg perhaps? | 22:58 |
| cj | pleia2: did you see http://git.linuxfoundation.org/ ? | 22:58 |
| pleia2 | I don't think so | 22:58 |
| cj | Konstantin maybe? :-) | 22:58 |
| anteaya | the gerrit bot tests look like it does pep8 and flake8 at best: http://git.openstack.org/cgit/openstack-infra/gerritbot/tree/ | 22:58 |
| pleia2 | cj: I did see that <3 | 22:58 |
| anteaya | no unit tests | 22:59 |
| cj | super! I guess that means there's low hanging fruit. | 22:59 |
| pleia2 | cj: could be, but I am also bad with faces \o/ | 22:59 |
| anteaya | very much so | 22:59 |
| cj | batting one thousand! | 22:59 |
| pleia2 | I think this is why I like irc | 23:00 |
| cj | anteaya: okay, can you tell me how to launch this bot? | 23:00 |
| anteaya | beyond what is in the installation file, no I have not launched this myself | 23:00 |
| pabelanger | anteaya: jhesketh: I have added updates to the current status of wiki.o.o | 23:00 |
| anteaya | pabelanger: thank you | 23:00 |
| anteaya | pabelanger: did you have a nice walk? | 23:00 |
| *** thorst has joined #openstack-infra | 23:00 | |
| pabelanger | we haven't had any new spam for the last few hours | 23:00 |
| anteaya | pabelanger: yay no new spam | 23:01 |
| pabelanger | anteaya: not long enough | 23:01 |
| anteaya | I feel the same way about my walks | 23:01 |
| anteaya | very windy outside roads are bad | 23:01 |
| anteaya | heard of someone totaling their car today :( | 23:01 |
| mordred | cj: btw - «TypeError: dist must be a Distribution instance» usually means your setuptools is way too old, iirc | 23:01 |
| aeng | hi, wondering is there an outage at the moment? https://openstackid.org/ | 23:01 |
| pleia2 | cj: there's an init script for it that you should have if you're using our puppet module https://git.openstack.org/cgit/openstack-infra/puppet-gerritbot/tree/files/gerritbot.init | 23:02 |
| anteaya | aeng: what are you seeing? | 23:02 |
| jeblair | cj: you may also be interested in https://review.openstack.org/253238 | 23:02 |
| aeng | "Whoops, looks like something went wrong." | 23:02 |
| anteaya | aeng: we are tighening things up with the wiki | 23:02 |
| anteaya | aeng: what were you trying to do that you saw that? | 23:02 |
| aeng | anteaya, when I click login | 23:02 |
| aeng | "The OpenID authentication failed." | 23:02 |
| jeblair | cj: and then as a first step, i'd recommend adding a "-d" option to it so that you can easily run it in the foreground. | 23:03 |
| mordred | jeblair: do we have the bot spec written up? | 23:03 |
| jeblair | mordred: no | 23:03 |
| anteaya | aeng: so you naviigate to openstackid.org, click login, and get "whoops, looks like something went wrong" is that accurate? | 23:03 |
| *** sdake has joined #openstack-infra | 23:03 | |
| jeblair | mordred: we're still at the impasse wrt interactive and yaml file configuration | 23:03 |
| aeng | anteaya, navigate to that page, click login then "The OpenID authentication failed." | 23:04 |
| *** baoli has quit IRC | 23:04 | |
| anteaya | aeng: okay thanks | 23:04 |
| anteaya | who has access to the openid server to take a peek at some logs? | 23:04 |
| jamesmcarthur | aeng: there is a problem with OpenStackID smarcet: should be popping in shortly to request an update | 23:04 |
| aeng | anteaya, actually,,, 1) Go to https://www.openstack.org/profile/, 2) click login, "The OpenID authentication failed." | 23:05 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/infra-manual: Update translation setup instructions https://review.openstack.org/284047 | 23:05 |
| *** cloudtrainme has joined #openstack-infra | 23:05 | |
| anteaya | aeng: ah thank you, that was the part I was missing | 23:05 |
| aeng | and if navigate to "https://openstackid.org/", message, Whoops, looks like something went wrong. | 23:05 |
| *** thorst has quit IRC | 23:05 | |
| anteaya | jamesmcarthur: thanks for the update | 23:05 |
| mordred | jeblair: oh right | 23:05 |
| aeng | jamesmcarthur, thanks for the update. Is someone looking at it now? | 23:06 |
| pleia2 | oh good, smarcet is on it | 23:06 |
| jamesmcarthur | I think he might need an assist from someone on Infra, but I’d expect him to pop on in a moment | 23:07 |
| pleia2 | I'm around as needed | 23:07 |
| *** pots has quit IRC | 23:07 | |
| cj | oh, hey mordred. good to see you here. | 23:08 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenstackId relase 1.0.12 https://review.openstack.org/286315 | 23:08 |
| *** dingyichen has joined #openstack-infra | 23:08 | |
| *** achanda has joined #openstack-infra | 23:08 | |
| cj | mordred: huh. I did a pip install setuptools and removed the system python-setuptools and still get that exception | 23:08 |
| mordred | hrm. | 23:08 |
| openstackgerrit | Merged openstack-infra/project-config: Lower max-servers in rax-iad to 180 https://review.openstack.org/286262 | 23:09 |
| mordred | cj: I do not get that error :) | 23:09 |
| cj | jeblair: is there a way I can check that review out from gerrit? I'm still getting used to the gerrit environment. | 23:09 |
| mordred | cj: in your git repo, do "git review -d 253238" | 23:10 |
| cj | mordred: heh. using virtualenv ? | 23:10 |
| mordred | cj: assuming you have git-review installed of course | 23:10 |
| mordred | cj: yes - I ran "tox -epep8 -r" ... -r says "please recreate the virtualenv | 23:10 |
| jeblair | (and if you don't have git-review installed, you might want to do so, it will make some things much easier) | 23:10 |
| *** yamamoto_ has joined #openstack-infra | 23:11 | |
| *** sarob has quit IRC | 23:11 | |
| *** smarcet has joined #openstack-infra | 23:11 | |
| jamesmcarthur | Can anyone approve this patch to fix OpenStackID? https://review.openstack.org/286315 | 23:11 |
| *** fbo has quit IRC | 23:11 | |
| smarcet | fungi: yolanda: mordred: could u please review and approve this one https://review.openstack.org/#/c/286315/1 | 23:11 |
| *** sarob has joined #openstack-infra | 23:11 | |
| jeblair | jamesmcarthur, smarcet: how did it break? | 23:11 |
| smarcet | current production version is missing one class that is declared on the app.php.erb template | 23:12 |
| smarcet | so basically is throwing a missing class exception bc manifest | 23:12 |
| cj | pkg_resources.DistributionNotFound: The 'python-daemon<2.1.0,>=2.0.4' distribution was not found and is required by the application | 23:12 |
| smarcet | that is bc the app.php.erb is dependant on version | 23:12 |
| mordred | wfm ... jeblair easy review above | 23:12 |
| *** yamahata has quit IRC | 23:12 | |
| cj | yeah, installing git-review was one of the first things my boss had me do | 23:12 |
| *** akscram has quit IRC | 23:12 | |
| mordred | cj: good boss! | 23:12 |
| jeblair | mordred: well, that's the change i'm asking questions about :) | 23:12 |
| *** dimtruck is now known as zz_dimtruck | 23:12 | |
| mordred | jeblair: neat | 23:12 |
| *** sbadia has quit IRC | 23:13 | |
| jeblair | mordred, smarcet, jamesmcarthur: i'm wondering how we could be in a position where we need an emergency upgrade to fix something which didn't break because of an upgrade | 23:13 |
| *** akscram has joined #openstack-infra | 23:13 | |
| smarcet | bc the app.php.erb is dependant on version | 23:13 |
| *** sbadia has joined #openstack-infra | 23:14 | |
| smarcet | i updated for dev | 23:14 |
| *** amrith is now known as _amrith_ | 23:14 | |
| smarcet | but that classs that i added on the manifest for dev | 23:14 |
| * aeng head twisting from jeblair comments | 23:14 | |
| smarcet | does not exist on current production release | 23:14 |
| smarcet | that is why is breaking | 23:14 |
| smarcet | problem is | 23:14 |
| *** fbo has joined #openstack-infra | 23:14 | |
| anteaya | smarcet: but if you added it to -dev how is it breaking in production? | 23:14 |
| smarcet | the puppet manifest has no tagging | 23:14 |
| *** dims has joined #openstack-infra | 23:14 | |
| smarcet | bc dev and production are not the same | 23:15 |
| cj | mordred: you going to be at LFNW this year? It occurs to me that the last time I saw you was at one of those. | 23:15 |
| smarcet | problem is puppet manifest is the same for both | 23:15 |
| *** Sukhdev has joined #openstack-infra | 23:15 | |
| smarcet | dev and production | 23:15 |
| smarcet | and app.php.erb is dependant on release | 23:15 |
| pleia2 | smarcet: so a change was made in puppet that broke prod? | 23:15 |
| anteaya | so the puppet manifest ran on production | 23:15 |
| *** salv-orl_ has quit IRC | 23:15 | |
| anteaya | and broke production | 23:15 |
| *** yamamoto_ has quit IRC | 23:16 | |
| smarcet | anteaya: aye | 23:16 |
| jeblair | this could have been fixed with a revert of the puppet change, right? | 23:19 |
| *** salv-orlando has joined #openstack-infra | 23:19 | |
| jamesmcarthur | jeblair: sure, but it could also be fixed with this latest merge | 23:19 |
| anteaya | or we could be breaking all new things | 23:19 |
| anteaya | we don't know | 23:19 |
| jamesmcarthur | well, we do know because it’s runnign fine on dev | 23:20 |
| jeblair | tbh, i would have preferred a simple revert of the puppet change that broke this, followed by a correction so that we could roll into production gracefully | 23:20 |
| jeblair | rather than an emergency change to production because something was broken in dev | 23:20 |
| jamesmcarthur | It’s either an emergency rollback or an emergency change. Again, this is a simple issue of a bad version# in config. | 23:21 |
| *** smarcet has quit IRC | 23:23 | |
| jeblair | i'm happy to help with true emergencies, but not every error made needs to rise to the level of asking folks to drop what they're doing to perform emergency production changes. | 23:24 |
| jeblair | i've approved that change | 23:24 |
| jeblair | however, i will not be around if something goes wrong with it | 23:24 |
| openstackgerrit | Petr Malik proposed openstack/requirements: Add xmltodict to global-requirements https://review.openstack.org/286316 | 23:24 |
| anteaya | jeblair: ack | 23:25 |
| jeblair | i will probably ask harder questions of similar changes in the future | 23:25 |
| jamesmcarthur | jeblair: we’re the only people that know how to update and work on that code. We’re a bit hamstrung by this as we rely on you guys to push through changes for an environment that you don’t normally deal with. | 23:26 |
| jamesmcarthur | jeblair: I get that it’s unusual situation, and I don’t think it’s really ideal for either party. | 23:26 |
| *** Daisy has joined #openstack-infra | 23:28 | |
| jamesmcarthur | Now we’ve had a 30 minute conversation about why this is or isn’t an emergency. Very similar to what’s been happening with the wiki. | 23:28 |
| jeblair | jamesmcarthur: i'm sorry you feel hamstrung. the entire line of my questions was directed to try to understand why you thought this merited an emergency change like that rather than being able to fix it yourself via a change to puppet. | 23:28 |
| jamesmcarthur | It’s an emergency because it needed to be fixed. We either needed you to roll it back, or push the fix. | 23:29 |
| jamesmcarthur | It’s as simple as that. | 23:29 |
| jeblair | jamesmcarthur: right. rolling back would have been the right thing to do. | 23:29 |
| jamesmcarthur | We did fix it ourselves. We just needed it to be merged. | 23:29 |
| *** Qiming has joined #openstack-infra | 23:30 | |
| jamesmcarthur | jeblair: I’m having a hard time understanding your stance. You’re not familiar with the environment, the code, or the change. | 23:30 |
| jamesmcarthur | I don’t understand the pushback. | 23:30 |
| nibalizer | jamesmcarthur: I think he doesn't like the interruption | 23:31 |
| *** Daisy has quit IRC | 23:31 | |
| jamesmcarthur | nibalizer: we would have needed to interrupt him in either case. That’s my point. | 23:31 |
| jeblair | jamesmcarthur: no, the puppet rollback would not need to have been escalated to the group that the roll-forward did | 23:31 |
| *** Daisy has joined #openstack-infra | 23:32 | |
| jeblair | the puppet rollback could have been accomplished by folks who are more familiar with the systems | 23:32 |
| jeblair | by going with the emergency roll- | 23:32 |
| *** pradk has quit IRC | 23:32 | |
| *** rossella_s has quit IRC | 23:32 | |
| fungi | okay, back and catching up | 23:32 |
| jeblair | forward solution, people who are unfamiliar were called upon to make a decision about something that they are not familiar with, in a crisis mode | 23:32 |
| jeblair | we're happy to help in crisis situations | 23:32 |
| *** rossella_s has joined #openstack-infra | 23:33 | |
| jeblair | but we should avoid making them when simpler solutions are available | 23:33 |
| *** sridhar_ram has quit IRC | 23:33 | |
| jamesmcarthur | jeblair: nobody could access OpenStackID, including Track Chairs on the last day of review. That was teh emergency. I appreciate your help, but we needed something immediate and it was a quick fix. | 23:33 |
| *** sc68cal has quit IRC | 23:34 | |
| jeblair | (and no information about the situation was provided in the commit message, it was basically just 'merge this or everything stays broken') | 23:34 |
| cj | okay, this is nuts: | 23:34 |
| cj | http://paste2.org/e73t0W1Y | 23:34 |
| cj | so first it downloads the version of python-daemon that is specified | 23:34 |
| cj | then it downloads one that is outside of the specification | 23:34 |
| cj | then it errors because the one it uses is not the one it specifies | 23:34 |
| cj | and then it throws an exception. | 23:34 |
| *** baoli has joined #openstack-infra | 23:35 | |
| cj | is this maybe because I'm using pbr and it's broken? | 23:35 |
| *** flepied has quit IRC | 23:36 | |
| *** Daisy has quit IRC | 23:36 | |
| *** andymaier has joined #openstack-infra | 23:37 | |
| anteaya | cj: that is a distinct possibility | 23:38 |
| cj | jeblair: so what is this -d option that you speak of? To which command do I pass it? | 23:38 |
| jeblair | cj: it doesn't exist. i was saying that adding an option to gerritbot to run in the foreground might be useful for debugging | 23:39 |
| cj | oh, I see! | 23:39 |
| jpmaxman | Gerritbot doesn't update the wiki does it (just random shot in the dark question in case it is related to wiki activity earlier). | 23:39 |
| jeblair | jpmaxman: no; only statusbot | 23:40 |
| EmilienM | fungi: can I continue to migrate our last ubuntu jobs? | 23:40 |
| jamesmcarthur | jeblair: “the puppet rollback could have been accomplished by folks who are more familiar with the systems” — Those of us that are familiar with OpenStackID aren’t able to push anything to production or rollback, AFAIK. | 23:40 |
| jeblair | jamesmcarthur: who merged the puppet change that broke it? | 23:40 |
| pabelanger | cj: jeblair could be gerritbot is broken on latest python-daemons too. Like zuul and nodepool where | 23:40 |
| jeblair | pabelanger: yes, the change i linked cj to attempts to correct that | 23:41 |
| pabelanger | Ah, missed that backscroll | 23:41 |
| jamesmcarthur | jeblair: smarcet asked fungi to merge it, which he did. | 23:41 |
| jamesmcarthur | Again, this was a minor config error with an easy fix. | 23:41 |
| jeblair | jamesmcarthur: a larger group reviews puppet changes than does system-config changes | 23:41 |
| jamesmcarthur | It was explained pretty clearly in IRC. I am still not clear on the pushback. | 23:41 |
| jamesmcarthur | jeblair: but none of them are familiar with this environment | 23:42 |
| jeblair | jamesmcarthur: it wasn't explained _at all_ in the commit message for the change you asked me to approve | 23:42 |
| jeblair | jamesmcarthur: i just approved it because you asked me to | 23:42 |
| jamesmcarthur | jeblair: no, but it was explained to you here in IRC and to all the other folks that brought the error to our attention | 23:42 |
| *** yamamoto_ has joined #openstack-infra | 23:42 | |
| jeblair | jamesmcarthur: you do know that i approved the change a long time ago, right? | 23:43 |
| jamesmcarthur | Yes. I do. | 23:43 |
| jeblair | ok, that's good :) | 23:43 |
| pabelanger | #success #succes still works | 23:46 |
| openstackstatus | pabelanger: Added success to Success page | 23:46 |
| *** xyang1 has quit IRC | 23:46 | |
| jeblair | jamesmcarthur: i am a very conservative system administrator. i don't like production to be broken, and when it does break, i like to find the safest way to fix it. for me, that generally means rolling backward, not forward, and identifying how a production break happened and working to avoid it in the future. | 23:46 |
| pabelanger | minus the typo, success bot is still working | 23:46 |
| nibalizer | pabelanger: we learned to spell in the same city I think | 23:46 |
| aeng | +1 jeblair | 23:46 |
| jeblair | jamesmcarthur: so i just want to figure out how to avoid this in the future | 23:47 |
| jeblair | jamesmcarthur: perhaps the puppet module should switch on versions, or perhaps it's a matter of setting defaults in a certain way to make them more future proof. both things are done elsewhere in our puppet modules, so there may be some patterns to work from there. | 23:49 |
| *** jtomasek_ has quit IRC | 23:50 | |
| jpmaxman | jeblair: that's understandable but also you do have to trust the people that are touching this code every single day :D Just an observation. Also jamesmcarthur seems legitimately confused about how to do a rollback without interrupting the same people that would need to push a fix. So maybe that's a good place to start. | 23:50 |
| *** gordc has quit IRC | 23:50 | |
| *** matrohon has quit IRC | 23:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch puppet-lint to ubuntu-trusty https://review.openstack.org/285542 | 23:50 |
| nibalizer | the following machines are disabled in the emergency file: logstash-worker20.openstack.org afstest.openstack.org cacti.openstack.org controller00.hpuseast.ic.openstack.org | 23:51 |
| jamesmcarthur | jeblair: right. I’m all for conservative, but there is nobody that deals with this code more than smarcet | 23:51 |
| nibalizer | i'd say we caan put cacti back into rotation | 23:51 |
| *** hichihara has joined #openstack-infra | 23:51 | |
| SpamapS | ianw: I didn't +A because I wanted more opinions to land on it. | 23:51 |
| jamesmcarthur | jeblair: This is the first time since we’ve been operating OpenStackID that it went down. So is it a common thing? No. Is it something we’ll be looking at, absolutely. | 23:52 |
| *** rockyg has quit IRC | 23:52 | |
| mordred | jpmaxman: it's not really about trust or not trust - we're all able to keep working on the same system by proactively communicating about what's going on and being diligent in providing each other with the information we all need to understand what's going on | 23:52 |
| jamesmcarthur | mordred: we did provide all the necessary information. That’s why I’m a bit frustrated here. | 23:52 |
| nibalizer | jamesmcarthur: did these changes go to opensatckid-dev first? | 23:52 |
| jamesmcarthur | We went directly to IRC when there was a problem and we proposed a fix. | 23:52 |
| *** smarcet has joined #openstack-infra | 23:53 | |
| jamesmcarthur | nibalizer: yes | 23:53 |
| nibalizer | and we still broke the production one? | 23:53 |
| fungi | jeblair: jamesmcarthur: granted, when i solo-approved 285475 i should have pressed harder asking whether that was going to be safe in production, but it _did_ at least provide a default value for the parameter it was adding | 23:53 |
| jamesmcarthur | They were already changed there. There was a minor issue with teh versionID in the config. | 23:53 |
| fungi | and i approved it on the grounds that it was urgent and we were stretched thin on puppet reviewers | 23:53 |
| fungi | but i agree that it's unfortunate for an urgent now-now-now change to result in an equally urgent now-now-now fix | 23:54 |
| jamesmcarthur | fungi: smarcet didn’t ask for an “urgent now-now-now” change, he asked if you could merge. | 23:55 |
| jamesmcarthur | We don’t have approval to merge, thus the request to fungi in IRC to merge. | 23:55 |
| jamesmcarthur | And yes, turns out, it broke something. Not ideal, but it’s not as though things don’t get broken. So we proposed an urgent fix. | 23:55 |
| *** Swami has quit IRC | 23:55 | |
| *** thorst has joined #openstack-infra | 23:56 | |
| mordred | jamesmcarthur: totally. so - for next time - putting in a why is really helpful to people. "Change to dev accidentally half-rolled in to production. Attempting to revert the half-change from production is likely more disruptive than just rolling forward since the change in question has been vetter in dev" - in the commit message allows for everyone involved to get a quick bullet understanding of | 23:56 |
| mordred | what's going on, why it's important and why this particular course of action was chosen | 23:56 |
| jamesmcarthur | mordred: 100% fair. | 23:56 |
| *** ybathia has joined #openstack-infra | 23:56 | |
| openstackgerrit | Colleen Murphy proposed openstack/diskimage-builder: Add --version option to disk-image-create https://review.openstack.org/277658 | 23:56 |
| jamesmcarthur | mordred: And agreed on the commit message. We will make it a point to be better | 23:56 |
| mordred | jamesmcarthur: cool. and I honestly just mean it as a "it helps us out to understand what's going on" and not as any sort of finger pointing | 23:57 |
| *** baoli has quit IRC | 23:57 | |
| openstackgerrit | Colleen Murphy proposed openstack/diskimage-builder: Add --version option to disk-image-create https://review.openstack.org/277658 | 23:57 |
| jamesmcarthur | mordred: yes sir, it’s perfectly reasonable | 23:57 |
| *** thorst_ has joined #openstack-infra | 23:58 | |
| smarcet | problem is , we cant change puppet app.php.erb | 23:58 |
| fungi | jamesmcarthur: yep, sorry, i misinterpreted smarcet's "this need to be reviewed asap please" to imply it was urgent and blocking a production deployment deadline | 23:58 |
| smarcet | without breaking something | 23:58 |
| smarcet | bc puppet is deployed on both servers | 23:59 |
| doug-fish | Are the issues @ https://openstackid.org/ under discussion here? (I can tell *something* broken is under discussion, but not quite sure what) | 23:59 |
| smarcet | there is any way that production server works with a labelled puppet config ? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!