| *** lukebrowning has joined #openstack-infra | 00:00 | |
| *** salv-orlando has quit IRC | 00:00 | |
| *** bobh has joined #openstack-infra | 00:02 | |
| *** kzaitsev_pi has quit IRC | 00:03 | |
| *** lukebrowning has quit IRC | 00:05 | |
| *** derekh has joined #openstack-infra | 00:06 | |
| *** lukebrowning has joined #openstack-infra | 00:07 | |
| *** kzaitsev_pi has joined #openstack-infra | 00:09 | |
| *** HeOS has quit IRC | 00:09 | |
| *** lukebrowning has quit IRC | 00:12 | |
| *** david-lyle has quit IRC | 00:12 | |
| *** david-lyle has joined #openstack-infra | 00:12 | |
| *** lukebrowning has joined #openstack-infra | 00:14 | |
| *** rlandy|ruck has quit IRC | 00:15 | |
| *** lukebrowning has quit IRC | 00:18 | |
| *** shu-mutou has quit IRC | 00:19 | |
| *** shu-mutou has joined #openstack-infra | 00:19 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for UNREACHABLE tasks https://review.openstack.org/509241 | 00:20 |
|---|---|---|
| *** lukebrowning has joined #openstack-infra | 00:20 | |
| *** harlowja has quit IRC | 00:23 | |
| *** lukebrowning has quit IRC | 00:25 | |
| *** lukebrowning has joined #openstack-infra | 00:26 | |
| *** jamesmcarthur has joined #openstack-infra | 00:26 | |
| *** derekh has quit IRC | 00:27 | |
| *** raissa has quit IRC | 00:28 | |
| fungi | SamYaple: that was jeblair's patch | 00:28 |
| fungi | ended up being a redirect rule typo | 00:28 |
| SamYaple | ah cool. thanks jeblair for the quick turn around | 00:28 |
| SamYaple | yea jeblair helped last night, but had to leave after seeing a stacktrace | 00:28 |
| fungi | we had \1 (regex style backreference) instead of $1 so apache wasn't honoring it | 00:29 |
| SamYaple | ah got it | 00:29 |
| SamYaple | also pulled in the updated secrets fetching. was still using the original script | 00:29 |
| SamYaple | worked well | 00:29 |
| SamYaple | no more weird "why do i have to spell out 'gerrit'" haha | 00:29 |
| fungi | i ask myself that all the time | 00:30 |
| SamYaple | i work on openstack so 'g' autocompletes as gerrit, not google | 00:30 |
| SamYaple | yay | 00:30 |
| *** jamesmcarthur has quit IRC | 00:31 | |
| *** lukebrowning has quit IRC | 00:31 | |
| *** derekh has joined #openstack-infra | 00:32 | |
| *** lukebrowning has joined #openstack-infra | 00:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zone-zuul-ci.org master: Make zuul-ci.org self-hosting https://review.openstack.org/529723 | 00:33 |
| fungi | and not github either | 00:34 |
| jeblair | infra-root: the .org registry has refused to delegate zuul-ci.org to ns[12].openstack.org for some reason (i have a useless error code and that's it). my best guess, after much debugging, is that there's either something about the way the openstack.org domain itself is registered, or the .org registry has some implicit requirements about glue records for delegation (even when they should not theoretically be needed). | 00:36 |
| fungi | this in no way surprises me | 00:36 |
| jeblair | infra-root: i propose that rather than continuing to try to delegate to ns[12].openstack.org, we simply delegate zuul-ci.org to itself, via ns[12].zuul-ci.org. i've added glue records for this and confirmed it works. | 00:37 |
| fungi | eventually we'll have a neutral domain under our control to self-host dns for central resources including our nameservers anyway | 00:37 |
| jeblair | infra-root: yes, that ^ -- once we settle on a name for that, we cane make *it* self-hosting and delegate zuul-ci.org to it. | 00:38 |
| fungi | so as a temporary measure, having zuul-ci.org self-host its nameservers seems like a fine compromise | 00:38 |
| *** lukebrowning has quit IRC | 00:38 | |
| *** dizquierdo has quit IRC | 00:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Revert "Add ns3.openstack.org" https://review.openstack.org/529724 | 00:39 |
| *** lukebrowning has joined #openstack-infra | 00:39 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for UNREACHABLE tasks https://review.openstack.org/509241 | 00:40 |
| jeblair | fungi: how does https://review.openstack.org/529723 look ? | 00:41 |
| fungi | like we want a gerritbot entry for that repo ;) | 00:42 |
| jeblair | 00:33 < openstackgerrit> James E. Blair proposed openstack-infra/zone-zuul-ci.org master: Make zuul-ci.org self-hosting https://review.openstack.org/529723 | 00:42 |
| fungi | oh, i guess we already have one and totally missed it in scrollbak! | 00:42 |
| jeblair | it's phase shifted or something i guess | 00:43 |
| fungi | i started in on the new holiday spiced rum from our local distillery, so going to blame that | 00:43 |
| *** lukebrowning has quit IRC | 00:44 | |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Make zuul-ci.org self-hosting https://review.openstack.org/529723 | 00:44 |
| fungi | ip addresses match, should work fine | 00:44 |
| fungi | also, wow that merges quickly when there are no jobs to be concerned with ;) | 00:45 |
| *** lukebrowning has joined #openstack-infra | 00:46 | |
| *** jamesmcarthur has joined #openstack-infra | 00:48 | |
| *** gongysh has joined #openstack-infra | 00:49 | |
| *** lukebrowning has quit IRC | 00:50 | |
| *** lukebrowning has joined #openstack-infra | 00:52 | |
| *** edmondsw has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:53 | |
| *** gongysh has quit IRC | 00:55 | |
| *** salv-orlando has joined #openstack-infra | 00:56 | |
| *** lukebrowning has quit IRC | 00:57 | |
| *** edmondsw has quit IRC | 00:57 | |
| *** jamesmcarthur has joined #openstack-infra | 00:57 | |
| *** markvoelker has quit IRC | 00:58 | |
| *** jamesmcarthur has quit IRC | 01:02 | |
| *** salv-orlando has quit IRC | 01:02 | |
| *** claudiub has quit IRC | 01:02 | |
| *** cuongnv has joined #openstack-infra | 01:03 | |
| *** camunoz has quit IRC | 01:06 | |
| *** derekh has quit IRC | 01:07 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Replace marked.js with markdown-it.js https://review.openstack.org/518625 | 01:09 |
| *** markvoelker has joined #openstack-infra | 01:10 | |
| jeblair | fungi: https://dnssec-debugger.verisignlabs.com/ns1.zuul-ci.org | 01:13 |
| jeblair | fungi: there are lots of green things and no yellow or red things | 01:13 |
| jeblair | however, it looks like we're using nsec and not nsec3 | 01:14 |
| jeblair | i suspect that "auto-dnssec: maintain" only does nsec and not nsec3 | 01:14 |
| jeblair | it's not a big deal for us (i mean, the zone file is in git!) but still would be nice to have on principle | 01:15 |
| jeblair | i mean, public git :) | 01:16 |
| *** daidv has quit IRC | 01:25 | |
| *** daidv has joined #openstack-infra | 01:25 | |
| *** Apoorva has quit IRC | 01:29 | |
| *** lukebrowning has joined #openstack-infra | 01:30 | |
| *** psachin has joined #openstack-infra | 01:31 | |
| *** lukebrowning has quit IRC | 01:35 | |
| *** lukebrowning has joined #openstack-infra | 01:36 | |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Add ns3.openstack.org" https://review.openstack.org/529724 | 01:37 |
| *** marst has quit IRC | 01:38 | |
| *** liujiong has joined #openstack-infra | 01:38 | |
| *** lukebrowning_ has joined #openstack-infra | 01:40 | |
| *** slaweq_ has joined #openstack-infra | 01:41 | |
| *** lukebrowning has quit IRC | 01:41 | |
| *** liujiong has quit IRC | 01:41 | |
| *** liujiong has joined #openstack-infra | 01:42 | |
| *** lukebrowning_ has quit IRC | 01:45 | |
| *** slaweq_ has quit IRC | 01:46 | |
| *** lukebrowning has joined #openstack-infra | 01:46 | |
| *** gyee has quit IRC | 01:48 | |
| *** jascott1 has quit IRC | 01:49 | |
| *** jascott1 has joined #openstack-infra | 01:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add OpenAPI documentation https://review.openstack.org/529013 | 01:50 |
| *** lukebrowning has quit IRC | 01:51 | |
| *** lukebrowning has joined #openstack-infra | 01:53 | |
| *** jascott1 has quit IRC | 01:54 | |
| *** SyntaxASI has quit IRC | 01:55 | |
| *** lukebrowning has quit IRC | 01:58 | |
| *** salv-orlando has joined #openstack-infra | 01:58 | |
| *** lukebrowning has joined #openstack-infra | 01:59 | |
| *** salv-orlando has quit IRC | 02:03 | |
| *** lukebrowning has quit IRC | 02:04 | |
| *** pbourke_ has quit IRC | 02:05 | |
| *** pbourke_ has joined #openstack-infra | 02:07 | |
| *** gongysh has joined #openstack-infra | 02:09 | |
| *** zhenguo has joined #openstack-infra | 02:12 | |
| *** kiennt26 has joined #openstack-infra | 02:14 | |
| *** bobh has quit IRC | 02:17 | |
| *** liujiong has quit IRC | 02:18 | |
| *** bobh has joined #openstack-infra | 02:19 | |
| *** linkmark has quit IRC | 02:19 | |
| *** liujiong has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has joined #openstack-infra | 02:20 | |
| *** bobh has quit IRC | 02:21 | |
| *** Apoorva has joined #openstack-infra | 02:30 | |
| *** lukebrowning has joined #openstack-infra | 02:34 | |
| *** Apoorva has quit IRC | 02:35 | |
| *** lukebrowning has quit IRC | 02:39 | |
| *** psachin has quit IRC | 02:40 | |
| *** lukebrowning has joined #openstack-infra | 02:40 | |
| *** mriedem has quit IRC | 02:40 | |
| *** SumitNaiksatam has quit IRC | 02:44 | |
| *** lukebrowning has quit IRC | 02:45 | |
| *** lukebrowning has joined #openstack-infra | 02:51 | |
| *** iyamahat has quit IRC | 02:53 | |
| *** yamahata has quit IRC | 02:53 | |
| *** psachin has joined #openstack-infra | 02:54 | |
| *** lukebrowning has quit IRC | 02:56 | |
| *** lukebrowning has joined #openstack-infra | 02:57 | |
| *** salv-orlando has joined #openstack-infra | 02:59 | |
| *** lukebrowning has quit IRC | 03:02 | |
| *** salv-orlando has quit IRC | 03:03 | |
| *** lukebrowning has joined #openstack-infra | 03:04 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** lukebrowning has quit IRC | 03:09 | |
| *** zhurong has joined #openstack-infra | 03:09 | |
| *** lukebrowning has joined #openstack-infra | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:13 | |
| *** jamesmcarthur has quit IRC | 03:13 | |
| *** jamesmcarthur has joined #openstack-infra | 03:13 | |
| *** lukebrowning has quit IRC | 03:15 | |
| *** lukebrowning has joined #openstack-infra | 03:16 | |
| *** xinliang has quit IRC | 03:19 | |
| *** ramishra has quit IRC | 03:19 | |
| *** lukebrowning has quit IRC | 03:21 | |
| *** lukebrowning has joined #openstack-infra | 03:23 | |
| *** coolsvap has joined #openstack-infra | 03:26 | |
| *** lukebrowning has quit IRC | 03:28 | |
| *** lukebrowning has joined #openstack-infra | 03:29 | |
| *** yamahata has joined #openstack-infra | 03:34 | |
| *** liujiong has quit IRC | 03:34 | |
| *** lukebrowning has quit IRC | 03:34 | |
| *** lukebrowning has joined #openstack-infra | 03:35 | |
| openstackgerrit | megan guiney proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 03:37 |
| *** liujiong has joined #openstack-infra | 03:37 | |
| *** Goneri has quit IRC | 03:37 | |
| *** lukebrowning has quit IRC | 03:40 | |
| *** lukebrowning has joined #openstack-infra | 03:41 | |
| *** lukebrowning has quit IRC | 03:46 | |
| *** lukebrowning has joined #openstack-infra | 03:48 | |
| *** lukebrowning has quit IRC | 03:53 | |
| *** lukebrowning has joined #openstack-infra | 03:54 | |
| *** ykarel has joined #openstack-infra | 03:59 | |
| *** lukebrowning has quit IRC | 03:59 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| mwhahaha | getting a proxy error on http://zuulv3.openstack.org/ | 03:59 |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** salv-orlando has joined #openstack-infra | 04:00 | |
| *** lukebrowning has joined #openstack-infra | 04:00 | |
| *** ykarel_ has joined #openstack-infra | 04:01 | |
| *** slaweq_ has joined #openstack-infra | 04:02 | |
| *** ykarel has quit IRC | 04:03 | |
| *** salv-orlando has quit IRC | 04:04 | |
| *** lukebrowning has quit IRC | 04:05 | |
| *** lukebrowning has joined #openstack-infra | 04:07 | |
| *** slaweq_ has quit IRC | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** lukebrowning has quit IRC | 04:11 | |
| *** lukebrowning has joined #openstack-infra | 04:13 | |
| *** markvoelker has quit IRC | 04:13 | |
| *** lukebrowning has quit IRC | 04:18 | |
| *** lukebrowning has joined #openstack-infra | 04:19 | |
| *** lukebrowning has quit IRC | 04:24 | |
| *** jamesmcarthur has joined #openstack-infra | 04:25 | |
| *** dbecker has quit IRC | 04:25 | |
| *** lukebrowning has joined #openstack-infra | 04:25 | |
| *** rcernin has quit IRC | 04:26 | |
| *** rcernin has joined #openstack-infra | 04:27 | |
| *** edmondsw has joined #openstack-infra | 04:28 | |
| *** threestrands_ has quit IRC | 04:29 | |
| *** ykarel__ has joined #openstack-infra | 04:29 | |
| *** jamesmcarthur has quit IRC | 04:29 | |
| *** lukebrowning has quit IRC | 04:30 | |
| *** ykarel_ has quit IRC | 04:31 | |
| *** zhurong has quit IRC | 04:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add OpenAPI documentation https://review.openstack.org/529013 | 04:33 |
| *** edmondsw has quit IRC | 04:33 | |
| *** lukebrowning has joined #openstack-infra | 04:34 | |
| *** dbecker has joined #openstack-infra | 04:39 | |
| *** yamamoto has joined #openstack-infra | 04:39 | |
| *** lukebrowning has quit IRC | 04:39 | |
| *** udesale has joined #openstack-infra | 04:39 | |
| *** lukebrowning has joined #openstack-infra | 04:41 | |
| *** links has joined #openstack-infra | 04:43 | |
| *** lukebrowning has quit IRC | 04:46 | |
| *** gmann has joined #openstack-infra | 04:46 | |
| *** lukebrowning has joined #openstack-infra | 04:47 | |
| *** rcernin has quit IRC | 04:47 | |
| *** gongysh has quit IRC | 04:48 | |
| *** rcernin has joined #openstack-infra | 04:49 | |
| *** lukebrowning has quit IRC | 04:52 | |
| *** lukebrowning has joined #openstack-infra | 04:53 | |
| *** Apoorva has joined #openstack-infra | 04:56 | |
| *** ykarel_ has joined #openstack-infra | 04:57 | |
| *** salv-orlando has joined #openstack-infra | 04:58 | |
| *** lukebrowning has quit IRC | 04:58 | |
| *** pgadiya has joined #openstack-infra | 04:58 | |
| *** lbragstad has quit IRC | 04:59 | |
| *** ykarel__ has quit IRC | 04:59 | |
| *** lukebrowning has joined #openstack-infra | 05:00 | |
| *** ramishra has joined #openstack-infra | 05:03 | |
| *** lukebrowning has quit IRC | 05:04 | |
| *** lukebrowning has joined #openstack-infra | 05:06 | |
| *** lukebrowning has quit IRC | 05:10 | |
| *** lukebrowning has joined #openstack-infra | 05:12 | |
| *** lukebrowning has quit IRC | 05:17 | |
| *** lukebrowning has joined #openstack-infra | 05:18 | |
| *** lukebrowning has quit IRC | 05:23 | |
| *** lukebrowning has joined #openstack-infra | 05:25 | |
| *** daidv has quit IRC | 05:26 | |
| *** cuongnv has quit IRC | 05:26 | |
| *** daidv has joined #openstack-infra | 05:26 | |
| *** cuongnv has joined #openstack-infra | 05:27 | |
| *** ykarel__ has joined #openstack-infra | 05:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use the integration-gate job-template https://review.openstack.org/529417 | 05:29 |
| *** lukebrowning has quit IRC | 05:29 | |
| *** ykarel_ has quit IRC | 05:30 | |
| *** janki has joined #openstack-infra | 05:34 | |
| *** ykarel_ has joined #openstack-infra | 05:36 | |
| *** ykarel__ has quit IRC | 05:39 | |
| *** Apoorva has quit IRC | 05:43 | |
| *** gcb has quit IRC | 05:44 | |
| *** ykarel_ has quit IRC | 05:44 | |
| *** lukebrowning has joined #openstack-infra | 05:45 | |
| *** lukebrowning has quit IRC | 05:50 | |
| *** armax has quit IRC | 05:54 | |
| *** vivsoni has joined #openstack-infra | 05:56 | |
| *** gongysh has joined #openstack-infra | 05:57 | |
| *** dhajare has joined #openstack-infra | 06:03 | |
| *** markvoelker has joined #openstack-infra | 06:14 | |
| *** edmondsw has joined #openstack-infra | 06:16 | |
| *** edmondsw has quit IRC | 06:20 | |
| *** lukebrowning has joined #openstack-infra | 06:24 | |
| AJaeger | Zuul is busy - "Queue lengths: 2995 events, 0 management events, 693 results."... | 06:27 |
| AJaeger | huge spike at 5:00 and now calculating ;( | 06:27 |
| *** lukebrowning has quit IRC | 06:29 | |
| *** ykarel has joined #openstack-infra | 06:29 | |
| *** alexchadin has joined #openstack-infra | 06:31 | |
| vivsoni | AJaeger: On uploaded patch, zuul jobs timed out | 06:32 |
| *** felipemonteiro__ has joined #openstack-infra | 06:35 | |
| vivsoni | AJaeger: something wrong with zuul... are you aware of it ? | 06:35 |
| AJaeger | vivsoni: I'm not a root, so can only see what's up there. | 06:36 |
| AJaeger | infra-root, if any of you is still/already up, could you check zuul, please? ^ | 06:36 |
| AJaeger | zuul is busy since 3:00 with a large spike at 5:00 according to cacti.openstack.org | 06:38 |
| AJaeger | infra-root, we're swapping ;( | 06:39 |
| AJaeger | I have a guess, I see 46 changes from EmilienM like https://review.openstack.org/#/c/529748/ - all depends-on openstack/puppet-openstack-integration with updating zuul.yaml in-repo. | 06:40 |
| AJaeger | infra-root, We currently use 8 GB swap ;( | 06:41 |
| AJaeger | #status notice Zuul.openstack.org is currently under heavy load and not starting new jobs. We're waiting for an admin to restart Zuul. | 06:42 |
| openstackstatus | AJaeger: sending notice | 06:42 |
| AJaeger | vivsoni: that's all I can do myself | 06:42 |
| vivsoni | AJaeger: ok... thanks.. will wait | 06:43 |
| -openstackstatus- NOTICE: Zuul.openstack.org is currently under heavy load and not starting new jobs. We're waiting for an admin to restart Zuul. | 06:43 | |
| openstackstatus | AJaeger: finished sending notice | 06:45 |
| *** heyongli has joined #openstack-infra | 06:48 | |
| *** markvoelker has quit IRC | 06:49 | |
| *** jascott1 has joined #openstack-infra | 06:50 | |
| *** heyongli has quit IRC | 06:53 | |
| *** heyongli has joined #openstack-infra | 06:54 | |
| *** hemna_ has quit IRC | 06:56 | |
| *** lukebrowning has joined #openstack-infra | 06:58 | |
| *** jascott1 has quit IRC | 07:02 | |
| *** liujiong has quit IRC | 07:02 | |
| *** lukebrowning has quit IRC | 07:02 | |
| *** jascott1 has joined #openstack-infra | 07:02 | |
| *** liujiong has joined #openstack-infra | 07:02 | |
| *** liujiong has quit IRC | 07:02 | |
| *** heyongli has quit IRC | 07:04 | |
| *** lukebrowning has joined #openstack-infra | 07:04 | |
| *** heyongli has joined #openstack-infra | 07:04 | |
| *** rcernin has quit IRC | 07:04 | |
| *** jascott1 has quit IRC | 07:07 | |
| *** lukebrowning has quit IRC | 07:09 | |
| *** lukebrowning has joined #openstack-infra | 07:10 | |
| *** heyongli has quit IRC | 07:14 | |
| *** askb has quit IRC | 07:14 | |
| *** heyongli has joined #openstack-infra | 07:14 | |
| *** ykarel has quit IRC | 07:15 | |
| *** lukebrowning has quit IRC | 07:15 | |
| *** lukebrowning has joined #openstack-infra | 07:16 | |
| AJaeger | EmilienM: looks like you ran by accident a DoS attack on Zuul. Please do not send that much zuul.yaml changes at once. | 07:16 |
| *** pblaho has joined #openstack-infra | 07:17 | |
| *** pblaho has quit IRC | 07:18 | |
| *** pblaho has joined #openstack-infra | 07:18 | |
| *** slaweq_ has joined #openstack-infra | 07:19 | |
| *** abelur_ has quit IRC | 07:19 | |
| *** lukebrowning has quit IRC | 07:21 | |
| *** lukebrowning has joined #openstack-infra | 07:23 | |
| *** slaweq_ has quit IRC | 07:23 | |
| *** heyongli has quit IRC | 07:24 | |
| *** heyongli has joined #openstack-infra | 07:24 | |
| *** lukebrowning has quit IRC | 07:33 | |
| *** heyongli has quit IRC | 07:34 | |
| *** heyongli has joined #openstack-infra | 07:35 | |
| *** claudiub has joined #openstack-infra | 07:40 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add implicit project name matching https://review.openstack.org/529801 | 07:41 |
| *** d0ugal has quit IRC | 07:43 | |
| *** d0ugal has joined #openstack-infra | 07:44 | |
| *** d0ugal has quit IRC | 07:44 | |
| *** d0ugal has joined #openstack-infra | 07:44 | |
| *** heyongli has quit IRC | 07:45 | |
| *** heyongli has joined #openstack-infra | 07:45 | |
| frickler | AJaeger: I think I could in theory restart zuul, but I'm not sure whether waiting to see whether it resolves itself would be more useful. or giving other admins the chance to debug this further. I also wouldn't know how to save+restore the queues | 07:48 |
| *** armaan has joined #openstack-infra | 07:54 | |
| *** heyongli has quit IRC | 07:55 | |
| *** heyongli has joined #openstack-infra | 07:55 | |
| *** lukebrowning has joined #openstack-infra | 07:58 | |
| Roamer` | BTW, should this Zuul out-of-memory problem explain the failure of the centos7 runs on https://review.openstack.org/#/c/529726/ or is it something else? test_network_basic_ops failed with "Cannot allocate memory"; should I recheck it? (after the Zuul DoS is resolved) | 08:00 |
| *** jascott1 has joined #openstack-infra | 08:01 | |
| *** lukebrowning has quit IRC | 08:03 | |
| ttx | In doubt, let's all blame EmilienM | 08:03 |
| *** edmondsw has joined #openstack-infra | 08:04 | |
| *** lukebrowning has joined #openstack-infra | 08:04 | |
| *** heyongli has quit IRC | 08:05 | |
| *** heyongli has joined #openstack-infra | 08:05 | |
| *** edmondsw has quit IRC | 08:09 | |
| *** lukebrowning has quit IRC | 08:09 | |
| d0ugal | +1 | 08:10 |
| *** lukebrowning has joined #openstack-infra | 08:11 | |
| *** amoralej|off is now known as amoralej | 08:13 | |
| *** hashar has joined #openstack-infra | 08:14 | |
| *** heyongli has quit IRC | 08:15 | |
| *** lukebrowning has quit IRC | 08:16 | |
| *** heyongli has joined #openstack-infra | 08:16 | |
| *** lukebrowning has joined #openstack-infra | 08:17 | |
| *** sshnaidm is now known as sshnaidm|off | 08:18 | |
| *** stakeda has joined #openstack-infra | 08:20 | |
| *** daidv has quit IRC | 08:22 | |
| *** cuongnv has quit IRC | 08:22 | |
| *** lukebrowning has quit IRC | 08:22 | |
| *** daidv has joined #openstack-infra | 08:22 | |
| *** cuongnv has joined #openstack-infra | 08:22 | |
| *** lukebrowning has joined #openstack-infra | 08:23 | |
| *** pcaruana has joined #openstack-infra | 08:24 | |
| *** jamesmcarthur has joined #openstack-infra | 08:25 | |
| *** heyongli has quit IRC | 08:26 | |
| *** kiennt26 has quit IRC | 08:26 | |
| *** heyongli has joined #openstack-infra | 08:26 | |
| *** kiennt26 has joined #openstack-infra | 08:26 | |
| *** lukebrowning has quit IRC | 08:28 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| *** heyongli has quit IRC | 08:36 | |
| *** heyongli has joined #openstack-infra | 08:36 | |
| *** yamamoto has quit IRC | 08:36 | |
| *** makowals has quit IRC | 08:36 | |
| AJaeger | frickler, infra-root, I would *not* restore the check queue - it still contains those 46 zuul.yaml changes... | 08:42 |
| AJaeger | frickler: in the past, once it started swapping heavily, it would not recover. | 08:43 |
| AJaeger | It's the last day before Christmas, let's take the day off ;) | 08:44 |
| * AJaeger rans further Christmas errands... | 08:44 | |
| *** jpena|off is now known as jpena | 08:45 | |
| *** markvoelker has joined #openstack-infra | 08:46 | |
| *** heyongli has quit IRC | 08:46 | |
| *** heyongli has joined #openstack-infra | 08:46 | |
| *** yamamoto has joined #openstack-infra | 08:47 | |
| *** yamamoto has quit IRC | 08:48 | |
| *** yamamoto has joined #openstack-infra | 08:49 | |
| *** slaweq_ has joined #openstack-infra | 08:51 | |
| *** ccamacho has quit IRC | 08:54 | |
| *** zhurong has joined #openstack-infra | 08:55 | |
| *** heyongli has quit IRC | 08:56 | |
| *** heyongli has joined #openstack-infra | 08:57 | |
| *** slaweq_ has quit IRC | 08:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add OpenAPI documentation https://review.openstack.org/529013 | 09:01 |
| *** HeOS has joined #openstack-infra | 09:01 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** lukebrowning has joined #openstack-infra | 09:02 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Really fix canonical/non-canonical project merge https://review.openstack.org/529810 | 09:05 |
| *** heyongli has quit IRC | 09:07 | |
| *** heyongli has joined #openstack-infra | 09:07 | |
| *** lukebrowning has quit IRC | 09:07 | |
| *** ccamacho has joined #openstack-infra | 09:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Really fix canonical/non-canonical project merge https://review.openstack.org/529810 | 09:08 |
| *** lukebrowning has joined #openstack-infra | 09:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add implicit project name matching https://review.openstack.org/529801 | 09:09 |
| *** ykarel has joined #openstack-infra | 09:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Really fix canonical/non-canonical project merge https://review.openstack.org/529810 | 09:10 |
| *** lukebrowning has quit IRC | 09:13 | |
| *** lukebrowning has joined #openstack-infra | 09:15 | |
| *** heyongli has quit IRC | 09:17 | |
| *** heyongli has joined #openstack-infra | 09:17 | |
| *** derekh has joined #openstack-infra | 09:18 | |
| *** markvoelker has quit IRC | 09:19 | |
| *** lukebrowning has quit IRC | 09:19 | |
| *** felipemonteiro__ has quit IRC | 09:21 | |
| *** lukebrowning has joined #openstack-infra | 09:21 | |
| *** makowals has joined #openstack-infra | 09:23 | |
| *** lukebrowning has quit IRC | 09:26 | |
| ttx | frickler: yeah I would just restart it, and not restore jobs. The root cause is known, so I don't think it's worth blocking everyone until someone else is up | 09:26 |
| *** heyongli has quit IRC | 09:27 | |
| *** heyongli has joined #openstack-infra | 09:27 | |
| *** lukebrowning has joined #openstack-infra | 09:28 | |
| *** lucas-afk is now known as lucasagomes | 09:31 | |
| *** heyongli has quit IRC | 09:37 | |
| *** heyongli has joined #openstack-infra | 09:38 | |
| *** lukebrowning has quit IRC | 09:38 | |
| *** kiennt26 has quit IRC | 09:40 | |
| *** lukebrowning has joined #openstack-infra | 09:41 | |
| *** ralonsoh has joined #openstack-infra | 09:45 | |
| *** lukebrowning has quit IRC | 09:46 | |
| *** lukebrowning has joined #openstack-infra | 09:47 | |
| *** heyongli has quit IRC | 09:48 | |
| *** heyongli has joined #openstack-infra | 09:48 | |
| *** florianf has joined #openstack-infra | 09:49 | |
| *** lukebrowning has quit IRC | 09:52 | |
| *** edmondsw has joined #openstack-infra | 09:52 | |
| *** lukebrowning has joined #openstack-infra | 09:54 | |
| frickler | ttx: o.k., I'll give it a try in a moment | 09:56 |
| *** edmondsw has quit IRC | 09:57 | |
| frickler | infra-root: I have received complaints about an account doing lots of +1 reviews following +W from others, behaving almost bot-like. not sure what to do about it, also not sure whether to share the account name in public, please advise | 09:57 |
| frickler | ttx: might be a tc issue, too ^^ | 09:57 |
| *** heyongli has quit IRC | 09:58 | |
| *** heyongli has joined #openstack-infra | 09:58 | |
| *** lukebrowning has quit IRC | 09:59 | |
| *** owalsh_ is now known as owalsh | 10:00 | |
| *** lukebrowning has joined #openstack-infra | 10:00 | |
| *** zhurong has quit IRC | 10:04 | |
| *** e0ne has joined #openstack-infra | 10:04 | |
| *** lukebrowning has quit IRC | 10:05 | |
| *** lukebrowning has joined #openstack-infra | 10:06 | |
| AJaeger | frickler: that's a social problem, so somebody should reach out to that person privately. | 10:07 |
| frickler | infra-root: not sure whether this is a known issue, zuul systemd scripts could need improvement, shouldn't stick in "active (exited)" IMO. http://paste.openstack.org/show/629630/ also failed to remove the pid file while stopping | 10:08 |
| *** heyongli has quit IRC | 10:08 | |
| frickler | AJaeger: yes, that's why on second thought I guessed the TC would like to take action | 10:08 |
| *** heyongli has joined #openstack-infra | 10:08 | |
| AJaeger | frickler: I don't think that's the role of TC... | 10:09 |
| * AJaeger sees an empty Zuul ;) | 10:10 | |
| *** gongysh has quit IRC | 10:10 | |
| *** daidv has quit IRC | 10:10 | |
| cmurphy | what's the harm in an extra +1 review? | 10:10 |
| frickler | AJaeger: yes, just restarted. how about this: #notice zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 10:11 |
| frickler | cmurphy: it could be considered spam if it always happens within 15 minutes after the patch received a +W | 10:11 |
| *** lukebrowning has quit IRC | 10:11 | |
| *** e0ne has quit IRC | 10:11 | |
| AJaeger | cmurphy: if it comes constantly after a +W, it looks like somebody looking for stackalytics numbers - and nothing else... It does not harm. | 10:11 |
| *** psachin has quit IRC | 10:12 | |
| cmurphy | AJaeger: frickler sure it doesn't look great but it's not creating extra work for anyone :) | 10:12 |
| *** lukebrowning has joined #openstack-infra | 10:13 | |
| AJaeger | frickler: let me reach out to that person, just had one myself. | 10:13 |
| AJaeger | cmurphy: example https://review.openstack.org/#/c/529579/ | 10:13 |
| *** e0ne has joined #openstack-infra | 10:13 | |
| cmurphy | yes I saw that :) | 10:13 |
| frickler | cmurphy: if you receive reviews via email, like I do for some projects, it does create extra work. and unneccessary when the patch has already been approved | 10:13 |
| *** cuongnv has quit IRC | 10:14 | |
| cmurphy | that's true and fair | 10:14 |
| frickler | #status notice zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 10:15 |
| openstackstatus | frickler: sending notice | 10:15 |
| *** shardy has joined #openstack-infra | 10:15 | |
| -openstackstatus- NOTICE: zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 10:16 | |
| *** markvoelker has joined #openstack-infra | 10:16 | |
| *** lukebrowning has quit IRC | 10:17 | |
| openstackstatus | frickler: finished sending notice | 10:18 |
| *** heyongli has quit IRC | 10:18 | |
| *** ykarel is now known as ykarel|afk | 10:18 | |
| *** heyongli has joined #openstack-infra | 10:19 | |
| *** lukebrowning has joined #openstack-infra | 10:19 | |
| *** lukebrowning has quit IRC | 10:24 | |
| *** lukebrowning has joined #openstack-infra | 10:25 | |
| *** heyongli has quit IRC | 10:29 | |
| *** heyongli has joined #openstack-infra | 10:29 | |
| *** lukebrowning has quit IRC | 10:30 | |
| AJaeger | frickler: could you review https://review.openstack.org/528644 to remove some jobs, please? https://etherpad.openstack.org/p/Nvt3ovbn5x contains some more infra-changes if you have time... | 10:32 |
| *** stakeda has quit IRC | 10:34 | |
| *** ramishra has quit IRC | 10:34 | |
| frickler | AJaeger: +2 to the first one, not sure I'll get to do more reviews today | 10:37 |
| *** lukebrowning has joined #openstack-infra | 10:38 | |
| AJaeger | frickler: thanks | 10:38 |
| *** heyongli has quit IRC | 10:39 | |
| *** heyongli has joined #openstack-infra | 10:39 | |
| *** lukebrowning has quit IRC | 10:43 | |
| *** lukebrowning has joined #openstack-infra | 10:44 | |
| *** lukebrowning has quit IRC | 10:49 | |
| *** heyongli has quit IRC | 10:49 | |
| *** heyongli has joined #openstack-infra | 10:49 | |
| *** markvoelker has quit IRC | 10:50 | |
| *** lukebrowning has joined #openstack-infra | 10:51 | |
| *** slaweq_ has joined #openstack-infra | 10:55 | |
| *** lukebrowning has quit IRC | 10:55 | |
| *** dsariel has joined #openstack-infra | 10:55 | |
| *** lukebrowning has joined #openstack-infra | 10:57 | |
| *** heyongli has quit IRC | 10:59 | |
| *** heyongli has joined #openstack-infra | 11:00 | |
| *** udesale has quit IRC | 11:00 | |
| *** lukebrowning has quit IRC | 11:01 | |
| AJaeger | frickler: from the one user you noticed, I got a ""550 User suspended: XXX@XXX" back - so no way to reach out ;( | 11:02 |
| frickler | AJaeger: infra-root: so for me that would seem to be a valid reason to suspend the associated gerrit account for spamming, too. interested to hear other's opinions | 11:03 |
| *** lukebrowning has joined #openstack-infra | 11:03 | |
| *** armaan has quit IRC | 11:05 | |
| *** armaan has joined #openstack-infra | 11:06 | |
| *** lukebrowning has quit IRC | 11:08 | |
| *** lukebrowning has joined #openstack-infra | 11:09 | |
| *** heyongli has quit IRC | 11:10 | |
| *** heyongli has joined #openstack-infra | 11:10 | |
| *** lukebrowning has quit IRC | 11:14 | |
| *** lukebrowning has joined #openstack-infra | 11:16 | |
| *** slaweq_ has quit IRC | 11:19 | |
| *** heyongli has quit IRC | 11:20 | |
| *** heyongli has joined #openstack-infra | 11:20 | |
| *** lukebrowning has quit IRC | 11:21 | |
| *** lukebrowning has joined #openstack-infra | 11:22 | |
| *** lukebrowning has quit IRC | 11:27 | |
| *** lukebrowning has joined #openstack-infra | 11:28 | |
| *** heyongli has quit IRC | 11:30 | |
| *** heyongli has joined #openstack-infra | 11:30 | |
| *** pgadiya has quit IRC | 11:31 | |
| *** lukebrowning has quit IRC | 11:33 | |
| *** xinliang has joined #openstack-infra | 11:33 | |
| *** xinliang has quit IRC | 11:33 | |
| *** xinliang has joined #openstack-infra | 11:33 | |
| *** tosky has joined #openstack-infra | 11:35 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add winrm certificate handling https://review.openstack.org/529821 | 11:40 |
| *** heyongli has quit IRC | 11:40 | |
| *** edmondsw has joined #openstack-infra | 11:41 | |
| *** heyongli has joined #openstack-infra | 11:41 | |
| *** janki has quit IRC | 11:41 | |
| *** lukebrowning has joined #openstack-infra | 11:42 | |
| *** edmondsw has quit IRC | 11:45 | |
| *** lukebrowning has quit IRC | 11:46 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Add new designate scenario job to neutron grafana https://review.openstack.org/529822 | 11:47 |
| *** markvoelker has joined #openstack-infra | 11:47 | |
| *** lukebrowning has joined #openstack-infra | 11:48 | |
| *** heyongli has quit IRC | 11:51 | |
| *** heyongli has joined #openstack-infra | 11:51 | |
| smcginnis | So... I've noticed a certain user that always - and only - every +1's patches minutes after they have been +W. | 11:52 |
| smcginnis | Should we just disable this account? Or does anyone think it's actually worth trying to contact this person? | 11:52 |
| *** lukebrowning has quit IRC | 11:53 | |
| frickler | smcginnis: we already tried, see above. assuming that is the same incident | 11:53 |
| smcginnis | frickler: Oh, guess I should have read backscroll. Let me see if it is. | 11:54 |
| *** lukebrowning has joined #openstack-infra | 11:55 | |
| frickler | AJaeger: mordred: not sure whether this is a new issue with sphinx docs or known and pending a fix: http://logs.openstack.org/85/524185/13/check/build-openstack-sphinx-docs/831806f/job-output.txt.gz#_2017-12-22_10_53_02_991400 | 11:55 |
| smcginnis | frickler: Sure enough, looks to be the same person. | 11:55 |
| smcginnis | I do not think it's harmless. It generates email and further games the stackalytics numbers that, whether we like or not (not!), is used by a lot of companies. | 11:56 |
| *** wolverineav has joined #openstack-infra | 11:57 | |
| *** alexchadin has quit IRC | 11:58 | |
| *** lukebrowning has quit IRC | 11:59 | |
| *** lukebrowning has joined #openstack-infra | 12:01 | |
| *** heyongli has quit IRC | 12:01 | |
| *** heyongli has joined #openstack-infra | 12:01 | |
| *** askb has joined #openstack-infra | 12:01 | |
| *** alexchad_ has joined #openstack-infra | 12:02 | |
| *** lukebrowning has quit IRC | 12:06 | |
| *** askb has quit IRC | 12:06 | |
| *** alexchad_ has quit IRC | 12:06 | |
| *** lukebrowning has joined #openstack-infra | 12:07 | |
| *** alexchadin has joined #openstack-infra | 12:08 | |
| *** smatzek has joined #openstack-infra | 12:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add winrm certificate handling https://review.openstack.org/529821 | 12:11 |
| *** heyongli has quit IRC | 12:11 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** heyongli has joined #openstack-infra | 12:11 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Really fix canonical/non-canonical project merge https://review.openstack.org/529810 | 12:12 |
| *** lukebrowning has quit IRC | 12:12 | |
| Roamer` | hmm, could somebody take a look at http://logs.openstack.org/22/529722/3/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/c66bcb1/job-output.txt.gz#_2017-12-22_11_50_08_309355 - I don't think these failures are my fault... :) | 12:12 |
| Roamer` | (problems with yum and puppet installing the needed packages) | 12:12 |
| *** alexchadin has quit IRC | 12:13 | |
| *** lukebrowning has joined #openstack-infra | 12:13 | |
| *** lukebrowning has quit IRC | 12:18 | |
| *** yamamoto has joined #openstack-infra | 12:19 | |
| *** lukebrowning has joined #openstack-infra | 12:20 | |
| *** ramishra has joined #openstack-infra | 12:20 | |
| *** markvoelker has quit IRC | 12:20 | |
| *** heyongli has quit IRC | 12:21 | |
| *** heyongli has joined #openstack-infra | 12:22 | |
| frickler | Roamer`: to me that looks like an issue with puppet, so I guess you should ask some puppet folks first. #puppet-openstack can probably be more helpful | 12:23 |
| AJaeger | smcginnis: I reached out to three people and got from one a 505 - account deactivated. Do you still get this from the last one or two hours? | 12:23 |
| Roamer` | frickler, thanks, will do | 12:23 |
| smcginnis | AJaeger: I think the last I saw was several hours ago. | 12:24 |
| *** rosmaita has joined #openstack-infra | 12:24 | |
| AJaeger | frickler: https://review.openstack.org/#/c/529006/ should fix cinder | 12:24 |
| *** wolverineav has quit IRC | 12:24 | |
| *** lukebrowning has quit IRC | 12:24 | |
| AJaeger | frickler: so, yes, it's a fallout... | 12:25 |
| *** jamesmcarthur has joined #openstack-infra | 12:26 | |
| *** lukebrowning has joined #openstack-infra | 12:26 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** lukebrowning has quit IRC | 12:31 | |
| *** heyongli has quit IRC | 12:32 | |
| *** heyongli has joined #openstack-infra | 12:32 | |
| *** gmann has quit IRC | 12:33 | |
| *** rosmaita has quit IRC | 12:37 | |
| *** lucasagomes is now known as lucas-hungry | 12:38 | |
| *** lukebrowning has joined #openstack-infra | 12:38 | |
| *** jamesmcarthur has joined #openstack-infra | 12:39 | |
| *** witek has quit IRC | 12:40 | |
| *** heyongli has quit IRC | 12:42 | |
| *** heyongli has joined #openstack-infra | 12:42 | |
| *** lukebrowning has quit IRC | 12:42 | |
| *** jamesmcarthur has quit IRC | 12:43 | |
| *** armaan_ has joined #openstack-infra | 12:45 | |
| *** lukebrowning has joined #openstack-infra | 12:46 | |
| *** armaan has quit IRC | 12:47 | |
| *** linkmark has joined #openstack-infra | 12:47 | |
| *** lukebrowning has quit IRC | 12:51 | |
| *** heyongli has quit IRC | 12:52 | |
| *** lukebrowning has joined #openstack-infra | 12:52 | |
| *** heyongli has joined #openstack-infra | 12:52 | |
| *** jpena is now known as jpena|lunch | 12:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard-webclient master: Migrate js-draft job to be zuul v3 native https://review.openstack.org/528481 | 12:54 |
| *** dave-mccowan has joined #openstack-infra | 12:57 | |
| *** lukebrowning has quit IRC | 12:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard-webclient master: Fix zuul job names https://review.openstack.org/528586 | 12:57 |
| *** lukebrowning has joined #openstack-infra | 12:59 | |
| *** HeOS has quit IRC | 13:01 | |
| *** heyongli has quit IRC | 13:02 | |
| *** heyongli has joined #openstack-infra | 13:03 | |
| *** lukebrowning has quit IRC | 13:03 | |
| *** lukebrowning has joined #openstack-infra | 13:05 | |
| *** coolsvap has quit IRC | 13:06 | |
| *** jamesmcarthur has joined #openstack-infra | 13:06 | |
| *** armaan_ has quit IRC | 13:08 | |
| *** armaan has joined #openstack-infra | 13:09 | |
| *** lukebrowning has quit IRC | 13:09 | |
| *** jamesmcarthur has quit IRC | 13:10 | |
| *** jaypipes is now known as leakypipes | 13:13 | |
| *** heyongli has quit IRC | 13:13 | |
| *** heyongli has joined #openstack-infra | 13:13 | |
| *** lukebrowning has joined #openstack-infra | 13:15 | |
| *** dave-mcc_ has joined #openstack-infra | 13:16 | |
| *** markvoelker has joined #openstack-infra | 13:17 | |
| *** dave-mccowan has quit IRC | 13:19 | |
| *** lukebrowning has quit IRC | 13:20 | |
| *** ykarel|afk is now known as ykarel | 13:21 | |
| *** lukebrowning has joined #openstack-infra | 13:21 | |
| *** owalsh is now known as owalsh_afk | 13:23 | |
| *** heyongli has quit IRC | 13:23 | |
| *** heyongli has joined #openstack-infra | 13:23 | |
| *** lukebrowning has quit IRC | 13:26 | |
| *** lukebrowning has joined #openstack-infra | 13:27 | |
| *** bauzas is now known as bauwser | 13:28 | |
| *** lyarwood has joined #openstack-infra | 13:30 | |
| *** ralonsoh has quit IRC | 13:30 | |
| *** lukebrowning has quit IRC | 13:32 | |
| lyarwood | Would anyone be able to point me at the `tempest-full` job definition as introduced to nova by https://review.openstack.org/#/c/529417/4/zuul.d/projects.yaml and currently failing on stable/pike http://logs.openstack.org/84/529384/1/check/tempest-full/9d165b9/job-output.txt.gz | 13:32 |
| *** heyongli has quit IRC | 13:33 | |
| *** heyongli has joined #openstack-infra | 13:33 | |
| *** rpittau has quit IRC | 13:34 | |
| mnaser | hi lyarwood, you can look at this file - https://github.com/openstack/tempest/blob/master/.zuul.yaml | 13:35 |
| frickler | fungi: do we also want to remove jenkins from its prominent position in the CI results tab? seems it still shows up there even after you removed the votes, see https://review.openstack.org/#/c/419761/ for an example | 13:35 |
| mnaser | you'll notice tempest-full has no 'run' key, but the parent of it 'devstack-tempest' has a run of 'playbooks/devstack-tempest.yaml' | 13:35 |
| mnaser | https://github.com/openstack/tempest/blob/master/playbooks/devstack-tempest.yaml | 13:36 |
| mnaser | (which is using the roles here https://github.com/openstack/tempest/tree/master/roles ) | 13:36 |
| lyarwood | mnaser: ack thanks! | 13:36 |
| mnaser | np :) | 13:37 |
| frickler | lyarwood: also in general this shows you the inheritance for the job that is being run under "zuul: inheritance_path": http://logs.openstack.org/84/529384/1/check/tempest-full/9d165b9/zuul-info/inventory.yaml | 13:37 |
| *** mriedem has joined #openstack-infra | 13:38 | |
| lyarwood | nice, thanks frickler, that's really useful | 13:38 |
| *** markvoelker has quit IRC | 13:40 | |
| *** markvoelker has joined #openstack-infra | 13:40 | |
| fungi | frickler: good idea... i think it'll take editing the ciRegex var near the top of hideci.js (in system-config) to drop special treatment of that username, i'll get a patch up | 13:41 |
| *** stephenfin is now known as finucannot | 13:42 | |
| mriedem | clarkb: booya http://status.openstack.org/elastic-recheck/data/integrated_gate.html | 13:42 |
| mriedem | Overall Categorization Rate: 67.7% | 13:43 |
| mriedem | back up to a semi-passing grade | 13:43 |
| mriedem | D- is passing yes? | 13:43 |
| *** heyongli has quit IRC | 13:43 | |
| *** heyongli has joined #openstack-infra | 13:44 | |
| tristanC | mriedem: are the crm114 report useful for categorization? | 13:45 |
| *** links has quit IRC | 13:45 | |
| mriedem | i never use them | 13:46 |
| mriedem | don't know what they are | 13:46 |
| *** jpena|lunch is now known as jpena | 13:46 | |
| AJaeger | lyarwood: in genreal: Use codesearch.openstack.org to find anything ;) | 13:46 |
| fungi | bayesian analysis of lines in the logs, highlighting uncommon patterns of text | 13:46 |
| tristanC | they should point at anomalies in logs (the pr_error score), though i couldn't find very useful report yet | 13:47 |
| *** dtantsur|afk is now known as dtantsur | 13:47 | |
| *** lucas-hungry is now known as lucasagomes | 13:51 | |
| fungi | frickler: thinking further through the ci results table and how hideci.js works, i'm unconvinced removing jenkins there is wise. first, it means that when looking at old merged (pre-zuulv3) reviews, you'll no longer see the jenkins job entries in the ci table... but also, we'll stop hiding the review comments from jenkins so they'll always end up displaying in the comment list even with ci comments | 13:51 |
| fungi | toggled to hidden | 13:51 |
| fungi | though i guess dropping it from th firstPartyCI var would help | 13:53 |
| fungi | since it would de-prioritize the results from jenkins in the table | 13:53 |
| fungi | but still, it'll take effect when looking at old reviews too, not just new ones | 13:53 |
| *** lukebrowning has joined #openstack-infra | 13:53 | |
| fungi | so may still not be the best solution | 13:53 |
| *** heyongli has quit IRC | 13:54 | |
| *** heyongli has joined #openstack-infra | 13:54 | |
| fungi | deleting the negative verify votes served a fairly distinct purpose, since they would shadow positive test results when looking at the changes list. i don't think the situation with the ci results table on individual changes is nearly so confusing | 13:55 |
| *** ykarel is now known as ykarel|away | 13:56 | |
| *** lukebrowning has quit IRC | 13:58 | |
| *** lukebrowning has joined #openstack-infra | 14:00 | |
| *** ianychoi has quit IRC | 14:01 | |
| *** kiennt26 has joined #openstack-infra | 14:02 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Status branch protection checking for github https://review.openstack.org/471175 | 14:02 |
| *** ykarel|away has quit IRC | 14:02 | |
| *** heyongli has quit IRC | 14:04 | |
| *** heyongli has joined #openstack-infra | 14:04 | |
| *** lukebrowning has quit IRC | 14:04 | |
| *** ianychoi has joined #openstack-infra | 14:05 | |
| *** lukebrowning has joined #openstack-infra | 14:06 | |
| *** Goneri has joined #openstack-infra | 14:07 | |
| *** shardy has quit IRC | 14:08 | |
| *** amoralej is now known as amoralej|lunch | 14:10 | |
| *** lukebrowning has quit IRC | 14:11 | |
| *** shardy has joined #openstack-infra | 14:11 | |
| *** jamesmcarthur has joined #openstack-infra | 14:12 | |
| *** Goneri has quit IRC | 14:12 | |
| *** lukebrowning has joined #openstack-infra | 14:13 | |
| *** heyongli has quit IRC | 14:14 | |
| *** pblaho has quit IRC | 14:14 | |
| *** heyongli has joined #openstack-infra | 14:14 | |
| *** lukebrowning has quit IRC | 14:17 | |
| *** jamesmcarthur has quit IRC | 14:18 | |
| *** lukebrowning has joined #openstack-infra | 14:19 | |
| *** makowals has quit IRC | 14:19 | |
| *** dhajare has quit IRC | 14:20 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Migrate js-draft job to be zuul v3 native https://review.openstack.org/528481 | 14:21 |
| *** heyongli has quit IRC | 14:24 | |
| *** heyongli has joined #openstack-infra | 14:25 | |
| Roamer` | frickler, thanks for posting another recheck on 529725 (the timed-out neutron test), I wasn't quite sure if that was the right thing to do | 14:27 |
| Roamer` | BTW will Zuul notice that there is a +W on this one and send it to the gate once it has passed the check pipeline? | 14:28 |
| frickler | fungi: I agree with the argument about old reviews to leave the status quo, that is a side of the issue I hadn't considered | 14:28 |
| Roamer` | (the +W was placed just before this morning's Zuul trouble) | 14:28 |
| *** smatzek has quit IRC | 14:29 | |
| *** lukebrowning has quit IRC | 14:29 | |
| *** smatzek has joined #openstack-infra | 14:30 | |
| frickler | Roamer`: looking for the "slow node" messages like this one is mostly a good indication for a recheck condition: http://logs.openstack.org/25/529725/1/check/legacy-tempest-dsvm-neutron-full-ocata/95bfb1d/job-output.txt.gz#_2017-12-22_12_09_05_327852 | 14:30 |
| frickler | Roamer`: also yes, the patch will enter gate once zuul has verified it +1 | 14:30 |
| *** smatzek has quit IRC | 14:30 | |
| *** lukebrowning has joined #openstack-infra | 14:31 | |
| *** HeOS has joined #openstack-infra | 14:33 | |
| *** heyongli has quit IRC | 14:35 | |
| *** heyongli has joined #openstack-infra | 14:35 | |
| *** lukebrowning has quit IRC | 14:35 | |
| *** rosmaita has joined #openstack-infra | 14:36 | |
| EmilienM | AJaeger: lol | 14:37 |
| EmilienM | is that serious? | 14:38 |
| EmilienM | did I really ddos zuul? | 14:38 |
| AJaeger | frickler, fungi, could I get a quick review on a job removal, please? https://review.openstack.org/#/c/528482 | 14:38 |
| EmilienM | ok first I'm sorry | 14:38 |
| frickler | andreaf: the new tempest-full job seems to be failing pretty often, if you don't have a fast fix, I guess we will need for revert https://review.openstack.org/529417 for now | 14:38 |
| AJaeger | EmilienM: Yes, you did - it was unusable from 6:00 UTC until frickler restarted it after 11:00 | 14:38 |
| EmilienM | omg | 14:38 |
| AJaeger | EmilienM: better discuss with jeblair how not to DoS Zuul... | 14:38 |
| *** lukebrowning has joined #openstack-infra | 14:39 | |
| AJaeger | EmilienM: little traffic, so perfect timing ;) No worries... | 14:39 |
| EmilienM | you know I didn't do it on purporse | 14:39 |
| EmilienM | purpose* | 14:39 |
| EmilienM | but what if someone does it on purpose? it sounds like we aren't ready for that | 14:39 |
| AJaeger | EmilienM: now you know what to do ;) | 14:40 |
| AJaeger | EmilienM: jeblair is the best to discuss that later... | 14:40 |
| AJaeger | thanks, frickler for quick review | 14:40 |
| EmilienM | https://review.openstack.org/#/q/topic:zuul/beaker | 14:40 |
| EmilienM | I was trying to migrate jobs | 14:41 |
| AJaeger | frickler, fungi please also review this small change in the same stack https://review.openstack.org/#/c/529498 | 14:43 |
| *** lukebrowning has quit IRC | 14:43 | |
| *** shardy is now known as shardy_afk | 14:43 | |
| *** lukebrowning has joined #openstack-infra | 14:45 | |
| *** heyongli has quit IRC | 14:45 | |
| *** heyongli has joined #openstack-infra | 14:45 | |
| frickler | AJaeger: +2 | 14:46 |
| frickler | andreaf: I'm confused, tempest-full is non-voting for tempest itself, but voting for other projects, is that really intended? | 14:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy draft job from storyboard-webclient https://review.openstack.org/528482 | 14:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Add job for publishing built javascript content https://review.openstack.org/528483 | 14:49 |
| *** lukebrowning has quit IRC | 14:49 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Update location automatically https://review.openstack.org/529838 | 14:49 |
| fungi | EmilienM: yeah, we have a known risk right now in that zuul has to create complete speculative configurations for proposed configuration changes being tested, and each full copy of the global configuration takes a good chunk of memory. we've talked about some solutions down the road where zuul could indefinitely queue new configuration changes until it's below a certain memory threshold or something | 14:49 |
| fungi | so it doesn't try to load them all into memory at the same time, but we also figure the further we get from v3 migration the less often we'll see large batches of job configuration changes proposed togetheer | 14:50 |
| *** lukebrowning has joined #openstack-infra | 14:51 | |
| EmilienM | ttx: you can blame me for working :-P | 14:51 |
| ttx | EmilienM: I work too! Coding on the ptgbot might not count as work though | 14:52 |
| pabelanger | good bot | 14:52 |
| EmilienM | what happen if we have someone with bad intent writing a script that keep patching zuul.yaml in some projects and doesn't stop? | 14:55 |
| *** heyongli has quit IRC | 14:55 | |
| *** heyongli has joined #openstack-infra | 14:55 | |
| *** lukebrowning has quit IRC | 14:56 | |
| *** edmondsw has joined #openstack-infra | 14:56 | |
| *** edmondsw has quit IRC | 14:57 | |
| *** lukebrowning has joined #openstack-infra | 14:57 | |
| *** amoralej|lunch is now known as amoralej | 14:58 | |
| fungi | EmilienM: we disable their account | 14:58 |
| EmilienM | fungi: if sending 30 patches against zuul.yaml breaks zuul, how many should I "recheck" in a row, now? 10? 5 ? | 14:59 |
| EmilienM | or 2-3 at a time? | 14:59 |
| fungi | EmilienM: 5 at a time is probably safe | 14:59 |
| EmilienM | k | 14:59 |
| fungi | really depends on how much else is also going on at the same time | 14:59 |
| *** smatzek has joined #openstack-infra | 15:00 | |
| fungi | EmilienM: this graph is helpful too http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 15:00 |
| fungi | (zuul scheduler memory consumption) | 15:00 |
| *** lukebrowning has quit IRC | 15:02 | |
| *** shardy_afk is now known as shardy | 15:03 | |
| * EmilienM bookmarks this url | 15:04 | |
| fungi | sane memory usage seems to stabilize somewhere around 10gb in use | 15:05 |
| *** heyongli has quit IRC | 15:05 | |
| *** heyongli has joined #openstack-infra | 15:06 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Update location automatically https://review.openstack.org/529838 | 15:09 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Add command to clean up at start of day https://review.openstack.org/529842 | 15:09 |
| *** heyongli has quit IRC | 15:16 | |
| *** heyongli has joined #openstack-infra | 15:16 | |
| *** marst has joined #openstack-infra | 15:17 | |
| *** armax has joined #openstack-infra | 15:18 | |
| *** sambetts|afk is now known as sambetts | 15:20 | |
| *** mriedem has quit IRC | 15:21 | |
| *** fultonj has joined #openstack-infra | 15:23 | |
| *** fultonj has quit IRC | 15:25 | |
| *** heyongli has quit IRC | 15:26 | |
| *** heyongli has joined #openstack-infra | 15:26 | |
| jeblair | mpeterson, andreaf: it looks like the free-spacing regex approach i tested in https://review.openstack.org/529633 works. mpeterson, why don't you try that in the job you're working on? | 15:28 |
| *** lukebrowning has joined #openstack-infra | 15:29 | |
| fungi | jeblair: not sure if you saw me mention yesterday, but i think i performed a fairly accurate test of whether we can just cname zuul.o.o to zuulv3. since it has only one vhost, that seems to work fine based on testing with /etc/hosts entries on my client | 15:31 |
| fungi | so i don't expect we need to bother adding a redirect there unless we just really want one | 15:31 |
| *** dsariel has quit IRC | 15:33 | |
| *** lukebrowning has quit IRC | 15:34 | |
| jeblair | fungi: ah ok, cool. yeah, i suppose in that case the hostname doesn't really matter... and i guess we can start the 'deprecation' period zuulv3.o.o as soon as we do... | 15:34 |
| *** dsariel has joined #openstack-infra | 15:35 | |
| *** heyongli has quit IRC | 15:36 | |
| *** heyongli has joined #openstack-infra | 15:36 | |
| jeblair | fungi: i was thinking today would be a good day to merge the switch-lists.o.o to multisite change... | 15:37 |
| jeblair | EmilienM: with the zuul restart, the project stanza debug option is live :) | 15:37 |
| EmilienM | jeblair: good | 15:38 |
| *** mriedem has joined #openstack-infra | 15:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test debug flag https://review.openstack.org/529846 | 15:39 |
| * jeblair checks to see if he's going to eat crow for breakfast | 15:40 | |
| fungi | jeblair: yeah, today is a good day for lists.o.o | 15:42 |
| *** armaan has quit IRC | 15:44 | |
| *** armaan has joined #openstack-infra | 15:44 | |
| *** xarses_ has quit IRC | 15:45 | |
| *** heyongli has quit IRC | 15:46 | |
| *** heyongli has joined #openstack-infra | 15:47 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Reject some commands if a track is not scheduled https://review.openstack.org/529847 | 15:47 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Also find room for tracks in additional rooms https://review.openstack.org/529848 | 15:47 |
| *** kiennt26 has quit IRC | 15:52 | |
| *** xarses_ has joined #openstack-infra | 15:56 | |
| *** heyongli has quit IRC | 15:57 | |
| *** heyongli has joined #openstack-infra | 15:57 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Make legacy-osc-placement-dsvm-functional voting and gating https://review.openstack.org/529850 | 15:57 |
| AJaeger | config-core, two small project-config changes for review, please: https://review.openstack.org/#/c/529498 and https://review.openstack.org/#/c/528644/ (both have one +2 by frickler already) | 15:58 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Make legacy-osc-placement-dsvm-functional voting and gating https://review.openstack.org/529850 | 16:00 |
| *** lukebrowning has joined #openstack-infra | 16:02 | |
| *** lukebrowning has quit IRC | 16:07 | |
| *** heyongli has quit IRC | 16:07 | |
| *** heyongli has joined #openstack-infra | 16:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update jobs for merlin and vinz-webclient https://review.openstack.org/529498 | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config master: consolidate interop jobs https://review.openstack.org/528644 | 16:08 |
| AJaeger | thanks, fungi! | 16:12 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Status branch protection checking for github https://review.openstack.org/471175 | 16:15 |
| *** armax has quit IRC | 16:16 | |
| *** heyongli has quit IRC | 16:17 | |
| *** heyongli has joined #openstack-infra | 16:17 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Fix zuul job names https://review.openstack.org/528586 | 16:21 |
| *** derekh has quit IRC | 16:25 | |
| *** heyongli has quit IRC | 16:27 | |
| AJaeger | config-core, we have quite a few openstack-zuul-jobs changes to remove obsolete jobs. any time for what should be a quick review? https://review.openstack.org/#/c/528650/ swift ; https://review.openstack.org/#/c/522102/ novaclient ; https://review.openstack.org/#/c/522471/ networking-midonet are the three largest; rest at https://etherpad.openstack.org/p/Nvt3ovbn5x which is shrinking nicely | 16:27 |
| *** heyongli has joined #openstack-infra | 16:28 | |
| *** jascott1 has quit IRC | 16:28 | |
| *** jascott1 has joined #openstack-infra | 16:28 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Rewrite documentation in more user-friendly way https://review.openstack.org/529853 | 16:28 |
| clarkb | mriedem: was that mostly just adding queries for current issues that brought up categorization? | 16:29 |
| mriedem | clarkb: yeah | 16:30 |
| *** xarses_ has quit IRC | 16:32 | |
| *** jascott1 has quit IRC | 16:33 | |
| *** lukebrowning has joined #openstack-infra | 16:34 | |
| *** heyongli has quit IRC | 16:38 | |
| *** heyongli has joined #openstack-infra | 16:38 | |
| *** dsariel has quit IRC | 16:38 | |
| *** lukebrowning has quit IRC | 16:39 | |
| *** lukebrowning has joined #openstack-infra | 16:41 | |
| *** sambetts is now known as sambetts|afk | 16:42 | |
| *** ramishra has quit IRC | 16:44 | |
| *** Apoorva has joined #openstack-infra | 16:44 | |
| *** lukebrowning has quit IRC | 16:45 | |
| *** jpich has quit IRC | 16:46 | |
| *** lukebrowning has joined #openstack-infra | 16:47 | |
| *** xarses_ has joined #openstack-infra | 16:47 | |
| *** heyongli has quit IRC | 16:48 | |
| *** heyongli has joined #openstack-infra | 16:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove integrated-gate-nova-net https://review.openstack.org/529858 | 16:49 |
| *** lukebrowning has quit IRC | 16:52 | |
| *** armaan has quit IRC | 16:55 | |
| *** fultonj has joined #openstack-infra | 16:56 | |
| *** armaan has joined #openstack-infra | 16:56 | |
| *** heyongli has quit IRC | 16:58 | |
| *** heyongli has joined #openstack-infra | 16:58 | |
| *** fultonj has quit IRC | 16:59 | |
| *** fultonj has joined #openstack-infra | 16:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Make lists.openstack.org multisite https://review.openstack.org/527499 | 17:00 |
| jeblair | clarkb, fungi: ^ can you +2 that, but leave the +W for me? | 17:01 |
| *** lukebrowning has joined #openstack-infra | 17:02 | |
| *** dtantsur is now known as dtantsur|afk | 17:02 | |
| *** lukebrowning has quit IRC | 17:06 | |
| *** lukebrowning has joined #openstack-infra | 17:08 | |
| *** heyongli has quit IRC | 17:08 | |
| *** heyongli has joined #openstack-infra | 17:09 | |
| clarkb | jeblair: +2'd | 17:09 |
| jeblair | config-core: you may be interested in taking a look at https://review.openstack.org/529846 to see a demonstration of a new debugging feature in zuul | 17:10 |
| jeblair | EmilienM: ^ | 17:10 |
| *** lukebrowning has quit IRC | 17:12 | |
| AJaeger | jeblair: cool - let's give a hint to "Toggle CI" ;) | 17:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix indentation on debug statement https://review.openstack.org/529864 | 17:13 |
| *** yamamoto has quit IRC | 17:13 | |
| jeblair | AJaeger: i'm not touching toggle ci, sorry. no idea how that stuff works. | 17:13 |
| *** iyamahat has joined #openstack-infra | 17:13 | |
| jeblair | AJaeger: i would hope that if you knew to request that, you'd know to hit the button to look at it. | 17:14 |
| *** lukebrowning has joined #openstack-infra | 17:14 | |
| jeblair | i spotted a minor error, fixed in 529864 | 17:15 |
| jeblair | (you can still infer the correct info though) | 17:15 |
| *** Goneri has joined #openstack-infra | 17:15 | |
| jeblair | config-core: we should not encourage people to actually land changes with 'debug: true' in project-pipeline configs -- i think it's best used in un-merged changes to find out why something isn't running in check, and then removed in a later patchset. we might make an exception for the periodic pipeline, however. | 17:17 |
| fungi | jeblair: was edge-computing supposed to be edge-computing@openstack? | 17:17 |
| jeblair | fungi: yes thanks, revising. | 17:17 |
| fungi | otherwise, lgtm | 17:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Make lists.openstack.org multisite https://review.openstack.org/527499 | 17:18 |
| jeblair | clarkb, fungi: ^ | 17:18 |
| EmilienM | mnaser, mwhahaha: I suggest we bookmark https://review.openstack.org/#/c/529846/1/.zuul.yaml next time we have issues with zuul | 17:18 |
| *** dizquierdo has joined #openstack-infra | 17:18 | |
| *** heyongli has quit IRC | 17:19 | |
| *** lukebrowning has quit IRC | 17:19 | |
| EmilienM | jeblair: have you ever thought of a "recheck" shortcut in gertty? | 17:19 |
| *** heyongli has joined #openstack-infra | 17:19 | |
| *** yamamoto has joined #openstack-infra | 17:20 | |
| EmilienM | jeblair: sorry, where is the debug log on https://review.openstack.org/529846 ? please show me a link | 17:20 |
| *** lukebrowning has joined #openstack-infra | 17:20 | |
| clarkb | fungi: was that supposed to be +W | 17:21 |
| clarkb | jeblair: ^ you might want to -W if nit | 17:21 |
| fungi | clarkb: no, i realized after rereading scrollback | 17:21 |
| fungi | un-did approval per earlier request | 17:22 |
| jeblair | cool, i also wipd | 17:22 |
| *** rmcall has joined #openstack-infra | 17:22 | |
| jeblair | EmilienM: it's in the message zuul leaves on the change | 17:22 |
| EmilienM | ok | 17:23 |
| EmilienM | jeblair: indeed, super useful | 17:23 |
| jeblair | EmilienM: i bet adding a recheck comment could be added to the 'reviewkeys' feature pretty easily | 17:23 |
| *** lucasagomes is now known as lucas-afk | 17:24 | |
| fungi | jeblair: it merits pointing out, lists.o.o is still in the nergency disable list on puppetmaster anyway, to avoid reverting the selective black-holing in /etc/aliases | 17:25 |
| *** yamamoto has quit IRC | 17:25 | |
| *** lukebrowning has quit IRC | 17:25 | |
| fungi | s/nergency/emergency/ | 17:25 |
| jeblair | fungi: thanks for pointing that out! | 17:25 |
| jeblair | fungi: what do we need to clear that? | 17:25 |
| fungi | i just backed up aliases and exim.conf to my homedir in case | 17:25 |
| *** rmcall has quit IRC | 17:26 | |
| fungi | jeblair: i was hoping the spamming of owner addresses would subside and we could just let puppet revert it at that point, but months later and we're still under constant flood | 17:26 |
| fungi | not sure if that's a great long-term solution to the problem or whether we need to put in something more general at this point | 17:26 |
| *** Apoorva has quit IRC | 17:26 | |
| fungi | i have saved some links to solutions from other communities with similar issues, particularly mediawiki, who seem to be doing some combination of subject and sender domain black-listing for delivery to owner addresses | 17:27 |
| jeblair | fungi: well, it seems like we ought to at least encode the status-quo in cfgmgt so we can operate as normal, yeah? | 17:28 |
| fungi | yeah, i can whip up something for managing /etc/aliases in puppet | 17:28 |
| *** gouthamr has joined #openstack-infra | 17:28 | |
| * fungi looks into that real quick | 17:28 | |
| jeblair | fungi: looks like puppet-exim manages that file | 17:29 |
| *** heyongli has quit IRC | 17:29 | |
| fungi | yeah | 17:29 |
| jeblair | and it's just a really old template | 17:29 |
| fungi | seeing that | 17:29 |
| *** heyongli has joined #openstack-infra | 17:29 | |
| jeblair | fungi: we should probably add a hash argument to the module for additional aliases | 17:29 |
| jeblair | (then we can drop 'gerrit2' and 'jenkins' as special cases :) | 17:30 |
| fungi | yup | 17:30 |
| fungi | would also give us the ability to block a few extra things on lists.o.o (particularly, security@l.o.o gets a fair amount of spam too) | 17:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix indentation on debug statement https://review.openstack.org/529864 | 17:34 |
| *** lukebrowning has joined #openstack-infra | 17:35 | |
| fungi | should have something in a sec, just making sure i get the hash iteration syntax for erb correct | 17:35 |
| *** ijw has joined #openstack-infra | 17:38 | |
| *** heyongli has quit IRC | 17:39 | |
| *** heyongli has joined #openstack-infra | 17:39 | |
| *** lukebrowning has quit IRC | 17:40 | |
| *** bauwser is now known as bauzas | 17:40 | |
| *** bauzas is now known as bauzas|pto | 17:40 | |
| *** mriedem is now known as bauzas | 17:40 | |
| *** lukebrowning has joined #openstack-infra | 17:41 | |
| *** bauzas is now known as mriedem | 17:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-exim master: Make /etc/aliases extensible https://review.openstack.org/529868 | 17:42 |
| fungi | working on related system-config changes for that ^ now | 17:42 |
| *** lukebrowning has quit IRC | 17:46 | |
| clarkb | the jenkins one can be dropped completely ya? | 17:48 |
| fungi | yep | 17:48 |
| clarkb | fungi: comment on the aliases change | 17:49 |
| fungi | thanks | 17:49 |
| clarkb | I think I got that correct but have only recently caffeinated so may not have counted correctly | 17:49 |
| *** heyongli has quit IRC | 17:49 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Set gerrit2 as an alias to root on Gerrit servers https://review.openstack.org/529870 | 17:50 |
| fungi | that's ^ putting the gerrit2 alias back for us | 17:50 |
| *** heyongli has joined #openstack-infra | 17:50 | |
| fungi | will work up the one with all the current *-owner black-holes in a moment | 17:50 |
| fungi | clarkb: yep, good eye. fixing now | 17:51 |
| clarkb | re logstash upgrade I've noticed that we don't need the proper jenkins log client anymore because we don'thvae to listen to 0mq events. I think I may update the puppetry to just run a geard instead then upgrade (this was we don't forget to do that) | 17:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-exim master: Make /etc/aliases extensible https://review.openstack.org/529868 | 17:52 |
| fungi | clarkb: ^ fixed | 17:52 |
| *** lukebrowning has joined #openstack-infra | 17:52 | |
| jeblair | clarkb: ++ | 17:52 |
| *** hashar is now known as hasharAway | 17:53 | |
| clarkb | fungi: I've +2'd but not approved both changes as the system-config one -1'd on the depends on being updated (I rechecked it though). Thinking we want to merge those roughly at the same time to avoid removing the alias then adding it back again | 17:55 |
| clarkb | fungi: so +W after everything has a V+1 | 17:55 |
| *** lukebrowning has quit IRC | 17:56 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Blackhole E-mail for some list owner addresses https://review.openstack.org/529871 | 17:58 |
| fungi | cool, and there's ^ the spam blocking change | 17:58 |
| mnaser | EmilienM: cool stuff re: debug | 17:58 |
| mnaser | ~the more you learn~ | 17:58 |
| *** lukebrowning has joined #openstack-infra | 17:58 | |
| *** heyongli has quit IRC | 18:00 | |
| *** heyongli has joined #openstack-infra | 18:00 | |
| jeblair | fungi: oh, actually i think that last one needs a change | 18:01 |
| *** HeOS has quit IRC | 18:01 | |
| jeblair | fungi: left commont on 529871 | 18:01 |
| fungi | thanks | 18:03 |
| *** lukebrowning has quit IRC | 18:03 | |
| openstackgerrit | Emilien Macchi proposed openstack/gertty master: reviewkeys: add 'message' parameter https://review.openstack.org/529872 | 18:04 |
| EmilienM | jeblair: ^ tested, works fine | 18:04 |
| *** lukebrowning has joined #openstack-infra | 18:05 | |
| openstackgerrit | Emilien Macchi proposed openstack/gertty master: reviewkeys: add 'message' parameter https://review.openstack.org/529872 | 18:05 |
| fungi | jeblair: should i just move that block from the global site manifest into openstack_project::lists? | 18:05 |
| jeblair | EmilienM: heh, you got one of my review comments, but i also left a second one | 18:06 |
| jeblair | fungi: the extra_aliases block, or the oppenstack_project::server block? | 18:08 |
| *** xarses has joined #openstack-infra | 18:08 | |
| jeblair | fungi: i think just move extra_aliases to the exim block in openstack_project::lists... | 18:08 |
| openstackgerrit | Emilien Macchi proposed openstack/gertty master: reviewkeys: add 'message' parameter https://review.openstack.org/529872 | 18:09 |
| *** lukebrowning has quit IRC | 18:09 | |
| EmilienM | jeblair: your second comment doesn't work for me | 18:09 |
| fungi | yeah, that's what i was asking. seemed like the thing to do anyway | 18:09 |
| EmilienM | jeblair: nevermind, I found | 18:10 |
| *** heyongli has quit IRC | 18:10 | |
| jeblair | EmilienM: oh sorry, reviewkey not review | 18:10 |
| *** heyongli has joined #openstack-infra | 18:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/gertty master: reviewkeys: add 'message' parameter https://review.openstack.org/529872 | 18:10 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Blackhole E-mail for some list owner addresses https://review.openstack.org/529871 | 18:10 |
| EmilienM | jeblair: done^ :-) thanks for the quick review, works like a charm | 18:10 |
| fungi | jeblair: ^ like that? | 18:10 |
| *** lukebrowning has joined #openstack-infra | 18:11 | |
| jeblair | fungi: ++ | 18:11 |
| openstackgerrit | Emilien Macchi proposed openstack/gertty master: Adding files/dirs in gitignore https://review.openstack.org/529873 | 18:12 |
| *** dhill_ has quit IRC | 18:12 | |
| *** dhill_ has joined #openstack-infra | 18:12 | |
| jeblair | EmilienM: cool. i think in the future it would be nice to support a reviewkey with no approvals which just left a message without changing the vote, but that's more complicated. and we can add that later without breaking compatability with this. | 18:14 |
| openstackgerrit | Merged openstack/gertty master: reviewkeys: add 'message' parameter https://review.openstack.org/529872 | 18:15 |
| openstackgerrit | Merged openstack/gertty master: Adding files/dirs in gitignore https://review.openstack.org/529873 | 18:15 |
| fungi | now i just need to debug the failing jobs for those changes. the puppet-exim change looks like it's passing, but the system-config changes are not getting through puppet apply yet | 18:15 |
| EmilienM | jeblair: thanks! | 18:16 |
| *** lukebrowning has quit IRC | 18:16 | |
| jeblair | fungi: that's pretty weird -- the system-config change is failing on hosts that it doesn't change. that suggests something is wrong with the apply test running on puppet-exim | 18:17 |
| fungi | Detail: undefined method `each' for "extra_aliases":String at /etc/puppet/modules/exim/manifests/init.pp:93 | 18:17 |
| *** lukebrowning has joined #openstack-infra | 18:17 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor master: Add support for running a standalone geard https://review.openstack.org/529874 | 18:17 |
| *** claudiub has quit IRC | 18:19 | |
| fungi | i wonder why it's not getting the {} default there and thinks there's a strong | 18:20 |
| *** heyongli has quit IRC | 18:20 | |
| fungi | oh, actually the one i'm looking at is the review.o.o entry which passed in an array | 18:20 |
| *** heyongli has joined #openstack-infra | 18:20 | |
| fungi | http://logs.openstack.org/70/529870/1/check/legacy-infra-puppet-apply-3-ubuntu-trusty/95c5f01/openstack-infra/system-config/applytest/puppetapplytest01.final.out.FAILED | 18:21 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor master: Add support for running a standalone geard https://review.openstack.org/529874 | 18:21 |
| fungi | but this one should he simply gotten the default {} and it's also claiming to be iterating over a string? http://logs.openstack.org/70/529870/1/check/legacy-infra-puppet-apply-3-ubuntu-trusty/95c5f01/openstack-infra/system-config/applytest/puppetapplytest03.final.out.FAILED | 18:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run standalone geard for log processing https://review.openstack.org/529875 | 18:22 |
| *** lukebrowning has quit IRC | 18:22 | |
| fungi | wait, is the template saying it's being asked to iterate over the string "extra_aliases"? | 18:22 |
| fungi | hah! i see the problem | 18:23 |
| * fungi sighs | 18:23 | |
| *** lukebrowning has joined #openstack-infra | 18:23 | |
| clarkb | missing $ | 18:24 |
| jeblair | fungi: oh, is the problem in the openstack_project::server application? | 18:24 |
| clarkb | https://review.openstack.org/#/c/529870/1/modules/openstack_project/manifests/server.pp line 186 needs $ | 18:25 |
| jeblair | that explains why it's affecting lots of hosts but only in the system-config change | 18:25 |
| jeblair | ya | 18:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Set gerrit2 as an alias to root on Gerrit servers https://review.openstack.org/529870 | 18:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Blackhole E-mail for some list owner addresses https://review.openstack.org/529871 | 18:25 |
| jeblair | okay, so tests working as expected :) | 18:25 |
| fungi | yep! | 18:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Make lists.openstack.org multisite https://review.openstack.org/527499 | 18:27 |
| jeblair | fungi, clarkb: ^ and there's the conflict resolution i was expecting :) | 18:28 |
| *** jpena is now known as jpena|off | 18:28 | |
| *** lukebrowning has quit IRC | 18:28 | |
| *** heyongli has quit IRC | 18:30 | |
| *** heyongli has joined #openstack-infra | 18:31 | |
| *** jgriffith is now known as jgriffith-merryC | 18:34 | |
| *** jgriffith-merryC is now known as jgriffith | 18:34 | |
| *** iyamahat has quit IRC | 18:35 | |
| clarkb | if zuul comes back happy with the stack I think it is ready | 18:35 |
| *** gyee has joined #openstack-infra | 18:37 | |
| fungi | yep, checking now | 18:38 |
| fungi | or i would, except my gertty seems to be taking forever to sync it | 18:40 |
| *** heyongli has quit IRC | 18:41 | |
| *** armax has joined #openstack-infra | 18:41 | |
| *** heyongli has joined #openstack-infra | 18:41 | |
| *** dave-mcc_ has quit IRC | 18:41 | |
| fungi | i guess at some point we may want domain-specific aliases support, but lgtm for now | 18:42 |
| jeblair | yeah, i think this will be okay even for the multisite lists | 18:43 |
| clarkb | its unlikely we'll have a name collision and not want to block things I'm guessing | 18:43 |
| *** jamesmca_ has joined #openstack-infra | 18:43 | |
| jeblair | right. the moment we have two different "foobar-owner" addresses and only want to :fail: one of them, we can split this out to multiple files :) | 18:44 |
| *** lukebrowning has joined #openstack-infra | 18:44 | |
| *** rmcall has joined #openstack-infra | 18:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Status branch protection checking for github https://review.openstack.org/471175 | 18:46 |
| *** lukebrowning has quit IRC | 18:48 | |
| AJaeger | clarkb, fungi, jeblair, everybody: Merry Christmas! I'm disappearing for a bit now... | 18:48 |
| *** felipemonteiro__ has joined #openstack-infra | 18:48 | |
| *** AJaeger is now known as AJaeger_ | 18:49 | |
| clarkb | AJaeger: thanks! | 18:49 |
| fungi | thanks AJaeger_! | 18:49 |
| jeblair | AJaeger: thanks, you too! | 18:49 |
| fungi | enjoy the holidays | 18:49 |
| AJaeger_ | thanks - enjoy the holidays as well... | 18:50 |
| fungi | my next several bits of holiday enjoyment involve last-minute cleaning before our first round of guests appear tomorrow | 18:50 |
| *** heyongli has quit IRC | 18:51 | |
| *** heyongli has joined #openstack-infra | 18:51 | |
| jeblair | fungi, clarkb: looks like the stack is all green | 18:52 |
| jeblair | i +3d all 3 | 18:53 |
| jeblair | i'm going to take a break while that lands and rolls out. i will probably make the lists.o.o switch after lunch. | 18:53 |
| jeblair | (i mean the multisite switch -- the exim aliases are approved and should land now) | 18:53 |
| clarkb | jeblair: roger | 18:54 |
| openstackgerrit | Gage Hugo proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Keystone https://review.openstack.org/524237 | 18:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Enable direct use of github driver in debug tool https://review.openstack.org/529606 | 18:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Status branch protection checking for github https://review.openstack.org/471175 | 18:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Status branch protection checking for github https://review.openstack.org/471175 | 18:59 |
| *** HeOS has joined #openstack-infra | 19:00 | |
| *** harlowja has joined #openstack-infra | 19:01 | |
| *** heyongli has quit IRC | 19:01 | |
| *** heyongli has joined #openstack-infra | 19:01 | |
| *** harlowja has quit IRC | 19:04 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Enable direct use of github driver in debug tool https://review.openstack.org/529606 | 19:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Status branch protection checking for github https://review.openstack.org/471175 | 19:04 |
| *** xarses has quit IRC | 19:09 | |
| openstackgerrit | Merged openstack-infra/puppet-exim master: Make /etc/aliases extensible https://review.openstack.org/529868 | 19:11 |
| *** heyongli has quit IRC | 19:11 | |
| *** poothia has quit IRC | 19:11 | |
| *** heyongli has joined #openstack-infra | 19:12 | |
| openstackgerrit | Merged openstack-infra/system-config master: Set gerrit2 as an alias to root on Gerrit servers https://review.openstack.org/529870 | 19:12 |
| openstackgerrit | Merged openstack-infra/system-config master: Blackhole E-mail for some list owner addresses https://review.openstack.org/529871 | 19:12 |
| fungi | i can take lists.o.o out of the emergency file and get those ^ applied since i have backups of the aliases file and can fix it back up if things don't go quite right | 19:16 |
| *** lukebrowning has joined #openstack-infra | 19:20 | |
| *** gouthamr has quit IRC | 19:20 | |
| fungi | #status log lists.openstack.org has been taken out of the emergency disable list now that the spam blackholing in /etc/aliases is managed by puppet | 19:21 |
| openstackstatus | fungi: finished logging | 19:21 |
| fungi | infra-root: ^ i'm watching the server now to make sure puppet updates it correctly, and will follow up with quick fixes if necessary | 19:21 |
| *** heyongli has quit IRC | 19:22 | |
| *** heyongli has joined #openstack-infra | 19:22 | |
| *** yamamoto has joined #openstack-infra | 19:24 | |
| pabelanger | okay, able to assist if needed too | 19:24 |
| clarkb | should check review.o.o too | 19:24 |
| fungi | yep, though i'm less worried about catching that one right away | 19:24 |
| fungi | if the aliases entries on lists.o.o get undone, a lot of people are going to suddenly get bombarded by thousands of chinese spam | 19:25 |
| *** poothia has joined #openstack-infra | 19:25 | |
| *** lukebrowning has quit IRC | 19:25 | |
| *** florianf has quit IRC | 19:25 | |
| *** fultonj has quit IRC | 19:26 | |
| *** lukebrowning has joined #openstack-infra | 19:26 | |
| *** felipemonteiro__ has quit IRC | 19:27 | |
| *** amoralej is now known as amoralej|off | 19:30 | |
| *** jascott1 has joined #openstack-infra | 19:31 | |
| *** lukebrowning has quit IRC | 19:31 | |
| *** heyongli has quit IRC | 19:32 | |
| *** heyongli has joined #openstack-infra | 19:32 | |
| *** armax has quit IRC | 19:32 | |
| *** lukebrowning has joined #openstack-infra | 19:33 | |
| *** jamesmca_ has quit IRC | 19:33 | |
| *** jascott1 has quit IRC | 19:34 | |
| *** jascott1 has joined #openstack-infra | 19:35 | |
| *** yamamoto has quit IRC | 19:35 | |
| *** lukebrowning has quit IRC | 19:38 | |
| fungi | looks like lists.o.o puppet is about to get applied now | 19:39 |
| *** dave-mccowan has joined #openstack-infra | 19:40 | |
| *** gouthamr has joined #openstack-infra | 19:40 | |
| *** lukebrowning has joined #openstack-infra | 19:41 | |
| jeblair | fungi: i'm around | 19:41 |
| fungi | cool | 19:42 |
| fungi | i'm basically staring at a filtered tail -f of the syslog on that server | 19:42 |
| *** heyongli has quit IRC | 19:42 | |
| fungi | ansible has copied modules and hiera data onto the server | 19:42 |
| *** heyongli has joined #openstack-infra | 19:42 | |
| fungi | so just waiting for the config stage | 19:42 |
| fungi | config just now getting applied | 19:45 |
| fungi | finished. no errors, checking results | 19:45 |
| fungi | lgtm! | 19:45 |
| fungi | i' | 19:45 |
| jeblair | also we created some new users | 19:45 |
| jeblair | dmsimard, frickler: congrats you're mailing list admins now :) | 19:45 |
| fungi | m a little surprised it worked as intended, frankly ;) | 19:45 |
| *** lukebrowning has quit IRC | 19:46 | |
| jeblair | fungi: i checked the exim4.conf change against your backup -- only diff is the keep_environment setting | 19:47 |
| jeblair | so that's as expected / good | 19:47 |
| fungi | yep, i just did the same | 19:47 |
| *** fultonj has joined #openstack-infra | 19:47 | |
| fungi | and for /etc/aliases, only the gerrit2 and jenkins removals | 19:47 |
| *** lukebrowning has joined #openstack-infra | 19:47 | |
| fungi | so, wow, that won't need debugging | 19:47 |
| fungi | checking review.o.o aliases next | 19:48 |
| jeblair | i think we're good until i break everything after lunch :) | 19:48 |
| fungi | that's gonna be some fun times! | 19:48 |
| * fungi 's gonna sing the doom song now! | 19:48 | |
| fungi | /etc/aliases on review.o.o looks good too | 19:49 |
| fungi | the gerrit2 alias bubbled up to the top, but that's expected | 19:49 |
| fungi | now i need to take a break from all this fun to do some cleaning... back soon as i can be | 19:50 |
| *** lukebrowning has quit IRC | 19:52 | |
| *** heyongli has quit IRC | 19:52 | |
| *** heyongli has joined #openstack-infra | 19:53 | |
| *** lukebrowning has joined #openstack-infra | 19:53 | |
| *** salv-orlando has quit IRC | 19:54 | |
| *** salv-orlando has joined #openstack-infra | 19:55 | |
| *** tushar has quit IRC | 19:56 | |
| *** vivsoni has quit IRC | 19:56 | |
| *** fultonj has quit IRC | 19:57 | |
| *** lukebrowning has quit IRC | 19:58 | |
| *** jmccrory has quit IRC | 19:59 | |
| *** jmccrory has joined #openstack-infra | 19:59 | |
| *** salv-orlando has quit IRC | 19:59 | |
| *** lukebrowning has joined #openstack-infra | 20:00 | |
| *** gouthamr has quit IRC | 20:00 | |
| *** heyongli has quit IRC | 20:03 | |
| *** heyongli has joined #openstack-infra | 20:03 | |
| *** jamesmcarthur has joined #openstack-infra | 20:04 | |
| *** lukebrowning has quit IRC | 20:05 | |
| *** lukebrowning has joined #openstack-infra | 20:06 | |
| *** lukebrowning has quit IRC | 20:11 | |
| *** gaurangt has quit IRC | 20:13 | |
| *** heyongli has quit IRC | 20:13 | |
| *** heyongli has joined #openstack-infra | 20:13 | |
| *** pcaruana has quit IRC | 20:13 | |
| *** e0ne has quit IRC | 20:16 | |
| *** lukebrowning has joined #openstack-infra | 20:17 | |
| *** gaurangt has joined #openstack-infra | 20:19 | |
| *** lukebrowning has quit IRC | 20:21 | |
| *** fanzhang has quit IRC | 20:21 | |
| *** fanzhang has joined #openstack-infra | 20:21 | |
| *** lukebrowning has joined #openstack-infra | 20:23 | |
| *** heyongli has quit IRC | 20:23 | |
| *** heyongli has joined #openstack-infra | 20:23 | |
| *** edmondsw has joined #openstack-infra | 20:25 | |
| *** lukebrowning has quit IRC | 20:27 | |
| *** lukebrowning has joined #openstack-infra | 20:29 | |
| *** heyongli has quit IRC | 20:33 | |
| *** heyongli has joined #openstack-infra | 20:34 | |
| *** lukebrowning has quit IRC | 20:34 | |
| *** lukebrowning has joined #openstack-infra | 20:36 | |
| *** lukebrowning has quit IRC | 20:40 | |
| *** rosmaita has quit IRC | 20:41 | |
| *** dizquierdo has quit IRC | 20:43 | |
| *** heyongli has quit IRC | 20:44 | |
| *** heyongli has joined #openstack-infra | 20:44 | |
| *** dsariel has joined #openstack-infra | 20:47 | |
| *** hasharAway has quit IRC | 20:49 | |
| *** heyongli has quit IRC | 20:54 | |
| *** heyongli has joined #openstack-infra | 20:54 | |
| jeblair | i've re-added lists.openstack.org to the emergency file and approved 527499 | 20:54 |
| jeblair | once it lands, i plan on stopping exim and apache on lists.o.o, moving the lists to their new filesystem location, then running puppet | 20:55 |
| *** lukebrowning has joined #openstack-infra | 20:55 | |
| *** hashar has joined #openstack-infra | 20:55 | |
| *** salv-orlando has joined #openstack-infra | 20:55 | |
| *** edmondsw has quit IRC | 20:58 | |
| *** lukebrowning has quit IRC | 21:00 | |
| *** salv-orlando has quit IRC | 21:00 | |
| *** lukebrowning has joined #openstack-infra | 21:02 | |
| *** heyongli has quit IRC | 21:04 | |
| *** heyongli has joined #openstack-infra | 21:04 | |
| fungi | sounds great | 21:05 |
| *** dsariel has quit IRC | 21:05 | |
| *** lukebrowning has quit IRC | 21:07 | |
| *** lukebrowning has joined #openstack-infra | 21:09 | |
| *** xarses has joined #openstack-infra | 21:11 | |
| openstackgerrit | Merged openstack-infra/system-config master: Make lists.openstack.org multisite https://review.openstack.org/527499 | 21:12 |
| *** e0ne has joined #openstack-infra | 21:13 | |
| *** andreww has joined #openstack-infra | 21:14 | |
| *** lukebrowning has quit IRC | 21:14 | |
| *** heyongli has quit IRC | 21:14 | |
| *** heyongli has joined #openstack-infra | 21:15 | |
| *** lukebrowning has joined #openstack-infra | 21:15 | |
| *** xarses has quit IRC | 21:16 | |
| jeblair | okay, i'm going to stop exim and apache on lists now | 21:17 |
| jeblair | and also mailman :) | 21:17 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for tempest-full pike token 404 failure bug 1739829 https://review.openstack.org/529888 | 21:19 |
| openstack | bug 1739829 in tempest "tempest-full job failing in stable/pike with 404 from keystone during tempest verify-config" [Undecided,New] https://launchpad.net/bugs/1739829 | 21:19 |
| *** markvoelker has quit IRC | 21:20 | |
| fungi | k, i'm around if something goes sideways | 21:20 |
| jeblair | cool | 21:20 |
| *** lukebrowning has quit IRC | 21:21 | |
| jeblair | i'm rsyncing now | 21:21 |
| jeblair | i could move them, but i'd feel better having a copy | 21:21 |
| jeblair | i think i'll move the old copies to a backup location though, so it's clear | 21:21 |
| *** HeOS has quit IRC | 21:22 | |
| *** lukebrowning has joined #openstack-infra | 21:22 | |
| fungi | sure | 21:22 |
| fungi | makes sense | 21:23 |
| *** smatzek has quit IRC | 21:23 | |
| *** smatzek has joined #openstack-infra | 21:24 | |
| *** heyongli has quit IRC | 21:25 | |
| *** heyongli has joined #openstack-infra | 21:25 | |
| *** gyee has quit IRC | 21:25 | |
| *** lukebrowning has quit IRC | 21:27 | |
| jeblair | okay rsyncing the archives is taking a while. i guess i should have seen that coming. | 21:28 |
| *** lukebrowning has joined #openstack-infra | 21:28 | |
| *** smatzek has quit IRC | 21:29 | |
| fungi | at least the lists have been pretty dead today, so likely nobody will notice the lag | 21:29 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for tempest-full pike token 404 failure bug 1739829 https://review.openstack.org/529888 | 21:30 |
| openstack | bug 1739829 in tempest "tempest-full job failing in stable/pike with 404 from keystone during tempest verify-config" [Undecided,New] https://launchpad.net/bugs/1739829 | 21:30 |
| clarkb | lots of little files? | 21:30 |
| jeblair | ayup | 21:30 |
| *** lukebrowning has quit IRC | 21:33 | |
| *** lukebrowning has joined #openstack-infra | 21:34 | |
| *** heyongli has quit IRC | 21:35 | |
| andreaf | frickler: err, I need to fix that, nice catch | 21:35 |
| *** heyongli has joined #openstack-infra | 21:35 | |
| jeblair | i think this server might be slow :( | 21:39 |
| inc0 | hey guys, quick zuul question, how can I force rebase on Kolla when it has Depends-On: << some change in kolla-ansible >>? | 21:39 |
| inc0 | kolla-ansible change happened and I want kolla to pick it up | 21:39 |
| *** lukebrowning has quit IRC | 21:39 | |
| fungi | inc0: just recheck | 21:39 |
| fungi | zuul will always use the latest patchset | 21:40 |
| *** HeOS has joined #openstack-infra | 21:40 | |
| fungi | and will always merge proposed changes to the latest target branch state when testing them | 21:40 |
| inc0 | allright thanks fungi | 21:41 |
| *** jamesmcarthur has quit IRC | 21:41 | |
| *** lukebrowning has joined #openstack-infra | 21:41 | |
| *** e0ne has quit IRC | 21:43 | |
| inc0 | btw full pipeline of build -> test on multiple scenarios -> publish seems to be coming on nicely:) | 21:43 |
| fungi | excellent | 21:43 |
| jeblair | inc0: \o/ | 21:43 |
| jeblair | we're up to 2015 on the dev list | 21:44 |
| inc0 | might be a bit harsh on network traffic of nodepools, but we can iterate from there | 21:44 |
| fungi | jeblair: maybe slow disk? iowait is pretty high | 21:44 |
| jeblair | fungi: ya | 21:44 |
| jeblair | i don't know if anyone's mentioned this, but that list has got a lot of messages | 21:44 |
| fungi | people do sometimes send messages to it, yes | 21:45 |
| *** heyongli has quit IRC | 21:45 | |
| *** heyongli has joined #openstack-infra | 21:45 | |
| *** e0ne has joined #openstack-infra | 21:46 | |
| *** lukebrowning has quit IRC | 21:47 | |
| jeblair | perhaps i should stop this and just move the directories | 21:47 |
| jeblair | we don't have any puppetry to delete things, and we have backups.... | 21:47 |
| fungi | are they staying on the same fs at least? | 21:48 |
| jeblair | ya | 21:48 |
| fungi | so should be atomic moves | 21:48 |
| *** lukebrowning has joined #openstack-infra | 21:48 | |
| fungi | when we rebuild or upgrade, would be nice to move the archives onto a cinder volume | 21:48 |
| jeblair | fungi: yeah, with the new layout, we can just put the entire /srv/mailman on it | 21:49 |
| fungi | if we plan ahead, can do background rsyncs and then one final with services down to catch it up at the end | 21:49 |
| jeblair | that's all the operational data, anything else is on / and dpkg managed | 21:49 |
| fungi | so however long the majority takes to sync is fine | 21:49 |
| jeblair | okay, i think i'm going to abort this and switch to moves | 21:50 |
| fungi | wfm | 21:50 |
| *** salv-orlando has joined #openstack-infra | 21:50 | |
| clarkb | ok | 21:51 |
| *** lukebrowning has quit IRC | 21:53 | |
| *** ijw has quit IRC | 21:53 | |
| *** e0ne has quit IRC | 21:54 | |
| jeblair | okay that's done, and i reconstructed the symlinks for the public archives | 21:54 |
| jeblair | i think i'm ready to run puppet now | 21:54 |
| fungi | standing by, bracing for impact | 21:55 |
| clarkb | I too am at a keyboard | 21:55 |
| *** heyongli has quit IRC | 21:55 | |
| jeblair | running it now (using kick.sh -- it's still in emergency) | 21:55 |
| * fungi is watching syslog | 21:56 | |
| *** heyongli has joined #openstack-infra | 21:56 | |
| fungi | there it goes | 21:56 |
| *** ijw has joined #openstack-infra | 21:57 | |
| jeblair | there are still some directories in /srv/mailman/openstack it will need to create, but the important ones should be in place already | 21:57 |
| jeblair | (like logs, run, lock, ...) | 21:57 |
| fungi | and the manifest is getting applied now | 21:58 |
| fungi | and completed | 21:58 |
| *** ijw has quit IRC | 21:58 | |
| jeblair | oh, the www-data ownership changes may have been an oversight, we may need to fix that | 21:58 |
| *** lukebrowning has joined #openstack-infra | 21:59 | |
| fungi | owner changed 'www-data' to 'list' | 21:59 |
| fungi | those? | 21:59 |
| jeblair | ya | 21:59 |
| jeblair | on the archives | 21:59 |
| *** ijw has joined #openstack-infra | 22:00 | |
| fungi | agreed, otherwise logged output lgtm | 22:00 |
| jeblair | other than that, things look generally pretty good... main thing is: it didn't create all the lists again :) | 22:00 |
| fungi | right ;) | 22:00 |
| fungi | i was slightly worried we'd missed something that might cause that to happen | 22:00 |
| fungi | should i send something to the infra ml? | 22:01 |
| fungi | 2017-12-22 22:02:17 H=(apnxaxco.org) [183.140.91.124] F=<1424570768@qq.com> rejected RCPT <foundation-owner@lists.openstack.org>: delivery temporarily disabled due to ongoing spam flood | 22:02 |
| fungi | that's still working at least | 22:02 |
| *** lukebrowning has quit IRC | 22:03 | |
| jeblair | we should still have all the services stoppe | 22:04 |
| jeblair | stopeed | 22:04 |
| *** jamesmcarthur has joined #openstack-infra | 22:04 | |
| fungi | oh, well exim is running | 22:04 |
| jeblair | oops. i stopped it again. | 22:04 |
| *** ijw has quit IRC | 22:04 | |
| jeblair | i guess puppet helpfully has that set to running | 22:04 |
| fungi | or the config change cause it to notify via service restart? | 22:05 |
| *** lukebrowning has joined #openstack-infra | 22:05 | |
| fungi | nope, the former | 22:05 |
| fungi | Dec 22 21:58:07 lists puppet-user[24755]: (/Stage[main]/Exim/Service[exim4]/ensure) ensure changed 'stopped' to 'running' | 22:05 |
| *** heyongli has quit IRC | 22:06 | |
| *** heyongli has joined #openstack-infra | 22:06 | |
| fungi | i see you have accidentally broken your mta, here let me fix it for you | 22:06 |
| clarkb | helpful robot | 22:06 |
| fungi | like one of my cats noticing my shoelaces are knotted, and helpfully untangling them for me | 22:07 |
| jeblair | i'll start apache now | 22:08 |
| fungi | k | 22:08 |
| jeblair | or rather, in just a min... | 22:08 |
| jeblair | i think i reconstructed the public archive symlinks wrong | 22:10 |
| jeblair | i'm going to redo that | 22:10 |
| *** lukebrowning has quit IRC | 22:10 | |
| *** lukebrowning has joined #openstack-infra | 22:11 | |
| jeblair | okay started apache | 22:12 |
| *** edmondsw has joined #openstack-infra | 22:12 | |
| fungi | http://lists.openstack.org/pipermail/release-job-failures/2017-December/000677.html used to be a valid archived message | 22:13 |
| fungi | 401? must be the perms/ownership still | 22:13 |
| jeblair | probably the www-data perms issue | 22:13 |
| fungi | yeah | 22:13 |
| jeblair | though the error at http://lists.openstack.org/cgi-bin/mailman/listinfo is perhaps more fundamental | 22:14 |
| jeblair | (or maybe it's the same?) | 22:14 |
| jeblair | i can't find a log entry for it | 22:14 |
| fungi | meant 403 not 401 | 22:14 |
| jeblair | ah found it | 22:15 |
| jeblair | ImportError: No module named mm_cfg_local | 22:15 |
| fungi | oh! | 22:15 |
| jeblair | SetEnv MAILMAN_SITE_DIR /srv/mailman/lists.openstack.org | 22:15 |
| jeblair | that's wrong :/ | 22:15 |
| fungi | should be /srv/mailman/openstack? | 22:16 |
| *** heyongli has quit IRC | 22:16 | |
| jeblair | ya | 22:16 |
| *** lukebrowning has quit IRC | 22:16 | |
| *** heyongli has joined #openstack-infra | 22:16 | |
| jeblair | ok, manually fixed that and http://lists.openstack.org/cgi-bin/mailman/listinfo is back | 22:16 |
| *** edmondsw has quit IRC | 22:17 | |
| *** lukebrowning has joined #openstack-infra | 22:17 | |
| clarkb | thats a bug in writing out the config in puppet? | 22:17 |
| jeblair | yep | 22:18 |
| fungi | 2001:470:e0d6:0:96de:80ff:feec:f9e7 - - [22/Dec/2017:22:17:11 +0000] "GET /pipermail/release-job-failures/2017-December/000677.html HTTP/1.1" 403 558 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" | 22:18 |
| jeblair | looks like i also missed a change to the archive directory | 22:18 |
| jeblair | <Directory /var/lib/mailman/archives/public/> | 22:18 |
| fungi | oh, yep | 22:18 |
| jeblair | that should point to srv/mailman/openstack as well | 22:18 |
| fungi | which likely 'splains the 403 | 22:18 |
| *** efried_cya_jan has quit IRC | 22:19 | |
| jeblair | and Alias /pipermail/ /var/lib/mailman/archives/public/ | 22:19 |
| jeblair | ok, manually fixed those 3 things, how's it look now? | 22:20 |
| clarkb | I can view archives over http | 22:21 |
| fungi | my archive sample looks better | 22:21 |
| fungi | list info pages look good | 22:21 |
| jeblair | http://lists.openstack.org/pipermail/foundation-board-confidential/ is correctly a 404 | 22:21 |
| fungi | i'm able to get to the admin and moderation interface for the infra ml | 22:22 |
| *** lukebrowning has quit IRC | 22:22 | |
| jeblair | should i start up the mailman queue runner now? | 22:23 |
| fungi | and just successfully discarded a bunch of spam posts from the moderation queue | 22:23 |
| fungi | yeah, things are looking good, so should be safe | 22:23 |
| jeblair | KeyError: 'MAILMAN_SITE_DIR' | 22:23 |
| jeblair | missed something there too i guess | 22:24 |
| *** lukebrowning has joined #openstack-infra | 22:24 | |
| *** heyongli has quit IRC | 22:26 | |
| *** heyongli has joined #openstack-infra | 22:26 | |
| jeblair | maybe i need an export in the init script | 22:27 |
| jeblair | that did it | 22:28 |
| jeblair | okay, i'll start exim now and send a test post to openstack-infra | 22:29 |
| *** lukebrowning has quit IRC | 22:29 | |
| fungi | sounds good | 22:29 |
| jeblair | i've stopped exim because i got a bounce of my test message | 22:31 |
| jeblair | 1eSVp9-0007HV-Vq was my test message | 22:31 |
| jeblair | 2017-12-22 22:29:52 1eSVp9-0007HV-Vq ** openstack-infra@lists.openstack.org R=mailman_router T=mailman_transport: Child process of mailman_transport transport returned 1 from command: /var/lib/mail | 22:32 |
| jeblair | man/mail/mailman | 22:32 |
| jeblair | we say this every time, but we should really create an actual test list :) | 22:33 |
| *** dave-mccowan has quit IRC | 22:35 | |
| clarkb | as far as how MAILMAN_SITE_DIR in apache getting set to the lists.openstack.org path I'm somewhat stumped. We use @name for the mailman::site define which should be 'openstack' in system-config | 22:36 |
| *** lukebrowning has joined #openstack-infra | 22:36 | |
| *** heyongli has quit IRC | 22:36 | |
| jeblair | clarkb: are we somehow swapping in the vhost @name rather than the mailman::site @name? | 22:36 |
| clarkb | oh that could be | 22:37 |
| clarkb | since we evaluate the vhost template in the context of the vhost define not the mailman::site itself | 22:37 |
| *** heyongli has joined #openstack-infra | 22:37 | |
| clarkb | we probably want to alias to mailman_site_name and then use that in the vhost. I'll put up a patch for that | 22:37 |
| fungi | i vaguely recall having to work around that in other places | 22:38 |
| *** salv-orlando has quit IRC | 22:40 | |
| fungi | ahh, what i'm remembering is explicitly seting vhost_name in the vhost resource and then referencing vhost_name in the config instead of relying on name | 22:41 |
| fungi | because of scoping issues between the vhost resource and the calling class | 22:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-mailman master: Alias $name in mailman site for vhost templating https://review.openstack.org/529893 | 22:41 |
| clarkb | something like that maybe | 22:41 |
| *** lukebrowning has quit IRC | 22:41 | |
| *** lukebrowning has joined #openstack-infra | 22:43 | |
| jeblair | welp, i somehow managed to send the lamest test message manually | 22:43 |
| jeblair | so i don't know why exim got an error | 22:43 |
| *** heyongli has quit IRC | 22:47 | |
| *** heyongli has joined #openstack-infra | 22:47 | |
| *** lukebrowning has quit IRC | 22:48 | |
| *** lukebrowning has joined #openstack-infra | 22:49 | |
| smcginnis | This is surprisingly hard to find an answer online. Does cgit have any kind of REST API like GitHub? | 22:51 |
| clarkb | smcginnis: to do what? it will serve you data using well known parameters and such but I don't know that I'd really call it a rest api or like github | 22:52 |
| clarkb | so you can get a specific file at a specific commit for example | 22:52 |
| clarkb | but there is no issue tracking so no api for that or organizations etc | 22:52 |
| smcginnis | I'm guessing it doesn't have the kind of functionality I'm looking for. | 22:53 |
| smcginnis | clarkb: Basically looking for something like https://api.github.com/repos/openstack/cinder/compare/7cc8d0ea63cc19aefcc0844ce455aead44297c21...stable/pike | 22:53 |
| fungi | it's a web front-end for browsing git repositories | 22:53 |
| fungi | ahh, trying to get arbitrary diffs between git revs? | 22:54 |
| *** lukebrowning has quit IRC | 22:54 | |
| smcginnis | fungi: Yeah. Was thinking this bit of automation I was doing was going to be pretty clever and easy. | 22:54 |
| smcginnis | fungi: Until github started rate limiting me. :) | 22:54 |
| clarkb | smcginnis: why not just do the diffs locally? | 22:54 |
| fungi | you can use git | 22:54 |
| smcginnis | Yeah, I was trying to avoid the need to clone all the repos locally and do it through subprocess, but that will have to be what I do. | 22:55 |
| smcginnis | Just a whole lot easier doing request.get('the data I want'). ;) | 22:56 |
| fungi | i'm probably weird, but i do that anyway and have a cronjob to update my local repos periodically | 22:56 |
| clarkb | smcginnis: cgit will do diffs for you but I don't think it will return them in a serialized format | 22:56 |
| clarkb | might be able to get them in patchformat though | 22:56 |
| smcginnis | Yeah, looks like I could get file diffs. But really less interested in the diffs themselves as I am in "here are all the commits that have happened between point A and point B. | 22:57 |
| *** heyongli has quit IRC | 22:57 | |
| smcginnis | No worries, I can clone and do it locally. | 22:57 |
| *** heyongli has joined #openstack-infra | 22:57 | |
| smcginnis | Just thought I would ask before I backburnered the first approach. | 22:57 |
| clarkb | smcginnis: I believe gerrits api will give you that data | 22:58 |
| clarkb | if you juts want changes between two points of time | 22:58 |
| smcginnis | clarkb: Oh right, that should be a pretty simple query. Good idea. | 22:59 |
| *** lukebrowning has joined #openstack-infra | 23:00 | |
| jeblair | oh i see the issue -- MAILMAN_INSTALL_DIR vs MAILMAN_SITE_DIR | 23:01 |
| jeblair | updating exim4.conf | 23:01 |
| fungi | jeblair: what test message did you send? i didn't receive one to any ml i'm subbed under anyway | 23:01 |
| jeblair | fungi: it bounced | 23:01 |
| fungi | oh, got ity | 23:01 |
| jeblair | fungi: or do you mean the manual one? | 23:01 |
| fungi | the manual one | 23:01 |
| jeblair | mailman still has it because exim isn't running | 23:01 |
| jeblair | sent new test message | 23:04 |
| jeblair | Dec 22 23:03:16 2017 (27749) post to openstack-infra from corvus@inaugust.com, size=2294, message-id=<87shc2fjri.fsf@meyer.lemoncheese.net>, success | 23:04 |
| jeblair | http://lists.openstack.org/pipermail/openstack-infra/2017-December/005750.html | 23:05 |
| clarkb | success sounds good | 23:05 |
| fungi | received | 23:05 |
| jeblair | fungi, clarkb: ^ want to check... | 23:05 |
| fungi | yay | 23:05 |
| jeblair | thanks :) | 23:05 |
| *** lukebrowning has quit IRC | 23:05 | |
| clarkb | I've received it as well | 23:05 |
| fungi | headers look sane | 23:05 |
| jeblair | okay, i've got a list of 3 things to fix | 23:06 |
| *** mhayden has quit IRC | 23:06 | |
| jeblair | since it's back in the emergency file, it shouldn't be a rush | 23:06 |
| clarkb | hopefully https://review.openstack.org/529893 is good for one of them | 23:06 |
| *** lukebrowning has joined #openstack-infra | 23:06 | |
| jeblair | ++ | 23:07 |
| *** heyongli has quit IRC | 23:07 | |
| *** heyongli has joined #openstack-infra | 23:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Alias $name in mailman site for vhost templating https://review.openstack.org/529893 | 23:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Fix multisite archive paths https://review.openstack.org/529894 | 23:08 |
| *** lukebrowning_ has joined #openstack-infra | 23:08 | |
| jeblair | er... why did i push your change up again? | 23:09 |
| jeblair | clarkb: no idea what happened there, sorry | 23:10 |
| clarkb | heh | 23:10 |
| clarkb | did it rebase? | 23:11 |
| jeblair | same parents | 23:11 |
| jeblair | just a different commit timestamp/committer | 23:11 |
| jeblair | i wouldn't think gerrit would allow that | 23:11 |
| jeblair | i did set the topic on my change. maybe that's what got it past gerrit? | 23:11 |
| *** lukebrowning has quit IRC | 23:11 | |
| clarkb | maybe? 2.13 definitely seems to care less about that stuff than 2.11 did | 23:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Export MAILMAN_SITE_DIR in init script https://review.openstack.org/529895 | 23:12 |
| fungi | yeah, this is similar to why our proposal jobs have suddenly started reproposing the same changes over and over | 23:12 |
| fungi | gerrit no longer rejects those | 23:12 |
| *** lukebrowning_ has quit IRC | 23:13 | |
| fungi | you can now push new patchsets virtually identical to their predecessors | 23:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix lists exim config https://review.openstack.org/529896 | 23:14 |
| jeblair | okay, that's everything on my list | 23:14 |
| fungi | i want to say it outputs a warning now but no longer refuses the push | 23:14 |
| *** lukebrowning has joined #openstack-infra | 23:15 | |
| jeblair | fungi: yep: remote: (W) No changes between prior commit 3602e33 and new commit 871e866 | 23:15 |
| fungi | yep, that's the one | 23:15 |
| *** jeblair is now known as corvus | 23:15 | |
| fungi | after i review these fixes, fungi will be known as not responding much for the rest of the evening | 23:16 |
| *** heyongli has quit IRC | 23:17 | |
| fungi | all 3 lgtm | 23:17 |
| fungi | great work on the mmmigration! | 23:18 |
| *** heyongli has joined #openstack-infra | 23:18 | |
| *** lukebrowning has quit IRC | 23:19 | |
| *** lukebrowning has joined #openstack-infra | 23:21 | |
| *** claudiub has joined #openstack-infra | 23:21 | |
| *** markvoelker has joined #openstack-infra | 23:21 | |
| *** hashar has quit IRC | 23:25 | |
| *** lukebrowning has quit IRC | 23:26 | |
| *** lukebrowning has joined #openstack-infra | 23:27 | |
| *** heyongli has quit IRC | 23:28 | |
| *** heyongli has joined #openstack-infra | 23:28 | |
| *** ijw has joined #openstack-infra | 23:31 | |
| *** lukebrowning has quit IRC | 23:32 | |
| *** lukebrowning has joined #openstack-infra | 23:34 | |
| *** Goneri has quit IRC | 23:36 | |
| *** heyongli has quit IRC | 23:38 | |
| *** heyongli has joined #openstack-infra | 23:38 | |
| *** lukebrowning has quit IRC | 23:38 | |
| *** salv-orlando has joined #openstack-infra | 23:41 | |
| *** ijw has quit IRC | 23:41 | |
| *** lukebrowning has joined #openstack-infra | 23:42 | |
| *** poothia has quit IRC | 23:45 | |
| *** salv-orlando has quit IRC | 23:46 | |
| *** lukebrowning has quit IRC | 23:47 | |
| *** marst has quit IRC | 23:48 | |
| *** heyongli has quit IRC | 23:48 | |
| *** heyongli has joined #openstack-infra | 23:48 | |
| *** lukebrowning has joined #openstack-infra | 23:48 | |
| corvus | mmm :) | 23:49 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix lists exim config https://review.openstack.org/529896 | 23:51 |
| *** lukebrowning has quit IRC | 23:53 | |
| *** lukebrowning has joined #openstack-infra | 23:55 | |
| *** markvoelker has quit IRC | 23:55 | |
| *** poothia has joined #openstack-infra | 23:57 | |
| *** heyongli has quit IRC | 23:58 | |
| *** heyongli has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!