| *** timburke has joined #openstack-infra | 00:00 | |
| *** jamesmcarthur has joined #openstack-infra | 00:01 | |
| clarkb | reading more I think it might be related to the footer as mailman's decorate() function shouldn't run when header and footer and nil | 00:02 |
|---|---|---|
| *** ijw has quit IRC | 00:02 | |
| *** ijw has joined #openstack-infra | 00:03 | |
| fungi | seems plausible | 00:03 |
| *** mgoddard has quit IRC | 00:06 | |
| *** ijw has quit IRC | 00:07 | |
| *** wolverineav has joined #openstack-infra | 00:07 | |
| clarkb | but for whatever reason switching the footer off and removing our mm_cfg.py hack doesn't fix it | 00:07 |
| clarkb | back to the bzr diffs could be a change in decorate.py | 00:08 |
| clarkb | bzr says no diff in Decorate.py | 00:08 |
| clarkb | between 2.1.16 and 2.1.20 tags | 00:08 |
| clarkb | so ya this is stumping me | 00:09 |
| clarkb | I've run out of day | 00:09 |
| clarkb | can pick this back up tomorrow. Thanks for all the help on that | 00:09 |
| clarkb | worth noting that functionalyl things are working its just making dmarc/dkim happy now I think | 00:11 |
| *** wolverineav has quit IRC | 00:12 | |
| mnaser | infra-root: looks like openstackgerrit died a while back and hasn't been back for a while | 00:13 |
| mnaser | ~20 minutes now | 00:13 |
| ianw | hrm, it's receiving events | 00:15 |
| *** mgoddard has joined #openstack-infra | 00:22 | |
| *** rlandy has quit IRC | 00:25 | |
| *** antonym has joined #openstack-infra | 00:26 | |
| ianw | i feel like maybe nodes aren't too healthy in the gate atm ... there is a massive nova stacked change being churned on, but still ... | 00:26 |
| clarkb | for the gerrit ircbot it may need to connect to a different freenode node | 00:29 |
| clarkb | for gate mriedem has been debugging an issue with devstacks python3 install method failing to getpython3 trove details from pypj | 00:29 |
| *** jamesmcarthur has quit IRC | 00:31 | |
| *** hongbin has joined #openstack-infra | 00:37 | |
| ianw | like http://zuul.openstack.org/stream/da9b639cdc3140fe93581e1f1aa0505f?logfile=console.log has a node assigned (2607:ff68:100:54:f816:3eff:fe4c:d3c0) which is up and active, but the job isn't yet running | 00:42 |
| ianw | i guess it's executor capacity? | 00:42 |
| clarkb | ya that can do it | 00:43 |
| *** lseki has quit IRC | 00:44 | |
| *** wolverineav has joined #openstack-infra | 00:46 | |
| ianw | yeah, we're in a prelonged period of executors not accepting jobs http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&panelId=22&fullscreen&from=now-7d&to=now | 00:50 |
| ianw | looks like it's trending back up, i guess maybe it was just https://review.openstack.org/#/c/638269/ which is like 44 dependent nova changes all at once | 00:51 |
| *** wolverineav has quit IRC | 00:51 | |
| *** jamesmcarthur has joined #openstack-infra | 00:54 | |
| *** markvoelker has joined #openstack-infra | 00:55 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** yamamoto has joined #openstack-infra | 01:02 | |
| pabelanger | ianw: clarkb: https://review.openstack.org/163922/ was another patch from corvus to maybe help with large stacks of commits for zuul | 01:05 |
| *** yamamoto has quit IRC | 01:07 | |
| ianw | Mar 13, 2015 5:21 AM ... heh i thought that url might be a typo :) | 01:08 |
| pabelanger | I think it is corvus oldest open change :) | 01:09 |
| *** eernst has joined #openstack-infra | 01:10 | |
| *** eernst has quit IRC | 01:14 | |
| *** dulek has quit IRC | 01:21 | |
| *** jamesmcarthur has joined #openstack-infra | 01:27 | |
| *** ricolin has joined #openstack-infra | 01:29 | |
| *** whoami-rajat has joined #openstack-infra | 01:31 | |
| *** dulek has joined #openstack-infra | 01:34 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** ijw has joined #openstack-infra | 01:47 | |
| *** eernst has joined #openstack-infra | 01:50 | |
| *** eernst has quit IRC | 01:53 | |
| *** nicolasbock has quit IRC | 02:05 | |
| *** rcernin_ has joined #openstack-infra | 02:05 | |
| *** hwoarang has quit IRC | 02:05 | |
| *** rcernin has quit IRC | 02:06 | |
| *** hwoarang has joined #openstack-infra | 02:06 | |
| *** bobh has joined #openstack-infra | 02:07 | |
| ianw | #status log restarted gerritbot as it seems to have dropped out | 02:07 |
| openstackstatus | ianw: finished logging | 02:07 |
| *** mrhillsman_afk is now known as mrhillsman | 02:10 | |
| *** rcernin_ has quit IRC | 02:12 | |
| *** rcernin has joined #openstack-infra | 02:15 | |
| *** bobh has quit IRC | 02:17 | |
| *** ijw_ has joined #openstack-infra | 02:18 | |
| *** openstackgerrit has joined #openstack-infra | 02:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove /hosts from log outputs https://review.openstack.org/649210 | 02:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove zonefile from nsd config https://review.openstack.org/648907 | 02:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 02:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 02:20 |
| *** ijw has quit IRC | 02:20 | |
| fungi | looks like it's working again | 02:20 |
| fungi | thanks ianw! | 02:20 |
| ianw | just needed a little holiday i guess | 02:22 |
| *** bhavikdbavishi has joined #openstack-infra | 02:33 | |
| *** jamesmcarthur has joined #openstack-infra | 02:44 | |
| *** wolverineav has joined #openstack-infra | 02:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 02:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 02:48 |
| *** hongbin has quit IRC | 02:57 | |
| *** hongbin has joined #openstack-infra | 02:58 | |
| *** jamesmcarthur has quit IRC | 03:04 | |
| *** yamamoto has joined #openstack-infra | 03:06 | |
| *** hongbin has quit IRC | 03:12 | |
| *** psachin has joined #openstack-infra | 03:14 | |
| *** hongbin has joined #openstack-infra | 03:14 | |
| *** samueldmq has quit IRC | 03:15 | |
| *** wolverineav has quit IRC | 03:22 | |
| *** wolverineav has joined #openstack-infra | 03:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 03:24 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 03:24 |
| *** ykarel|away has joined #openstack-infra | 03:27 | |
| *** hongbin has quit IRC | 03:31 | |
| *** spsurya has joined #openstack-infra | 03:31 | |
| mnaser | fwiw: sjc1 is running an unreleased version of openstack :) -- please ping me if you notice anything odd. | 03:33 |
| mnaser | afaik: things are sailing smoothly but there will be corner cases I assume | 03:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add Debian Buster boot tests https://review.openstack.org/649214 | 03:33 |
| ianw | fungi / zigo : ^ | 03:34 |
| *** ykarel|away is now known as ykarel | 03:40 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** jamesmcarthur has joined #openstack-infra | 04:01 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** yamamoto has quit IRC | 04:06 | |
| *** yamamoto has joined #openstack-infra | 04:06 | |
| *** yamamoto has quit IRC | 04:06 | |
| *** jamesmcarthur has joined #openstack-infra | 04:08 | |
| *** jamesmcarthur has quit IRC | 04:08 | |
| *** jamesmcarthur has joined #openstack-infra | 04:08 | |
| *** ykarel has quit IRC | 04:11 | |
| *** udesale has joined #openstack-infra | 04:11 | |
| *** roman_g has quit IRC | 04:14 | |
| *** jamesmcarthur has quit IRC | 04:17 | |
| *** jamesmcarthur has joined #openstack-infra | 04:18 | |
| *** jamesmcarthur has quit IRC | 04:25 | |
| *** jamesmcarthur_ has joined #openstack-infra | 04:26 | |
| *** wolverineav has quit IRC | 04:28 | |
| *** ykarel has joined #openstack-infra | 04:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 04:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 04:32 |
| *** wolverineav has joined #openstack-infra | 04:32 | |
| *** jamesmcarthur_ has quit IRC | 04:39 | |
| *** jamesmcarthur has joined #openstack-infra | 04:40 | |
| *** yamamoto has joined #openstack-infra | 04:45 | |
| *** jamesmcarthur has quit IRC | 04:46 | |
| *** amansi26 has joined #openstack-infra | 04:59 | |
| *** wolverineav has quit IRC | 05:01 | |
| *** adriant has quit IRC | 05:07 | |
| *** adriant has joined #openstack-infra | 05:08 | |
| *** pcaruana has joined #openstack-infra | 05:35 | |
| *** tbachman has quit IRC | 05:38 | |
| *** ramishra has joined #openstack-infra | 05:38 | |
| *** tbachman has joined #openstack-infra | 05:39 | |
| *** jamesmcarthur has joined #openstack-infra | 05:41 | |
| *** jamesmcarthur has quit IRC | 05:46 | |
| *** yamamoto has quit IRC | 05:48 | |
| *** quiquell|off is now known as quiquell | 05:49 | |
| *** tbachman has quit IRC | 05:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add api-ref-jobs for Cyborg https://review.openstack.org/648914 | 05:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop nova-specs announcing in #openstack-placement https://review.openstack.org/648698 | 05:56 |
| *** jamesmcarthur has joined #openstack-infra | 05:57 | |
| *** markvoelker has quit IRC | 05:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Migrate ironic-staging-drivers jobs in-tree https://review.openstack.org/647389 | 05:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding zuul jobs for rsd-virt-for-nova repo https://review.openstack.org/642500 | 06:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Update the network-ovn failure rate dashboard https://review.openstack.org/642879 | 06:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Add graph for number of periodic neutron jobs run https://review.openstack.org/642836 | 06:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Move wsme jobs in-tree https://review.openstack.org/642737 | 06:01 |
| *** jamesmcarthur has quit IRC | 06:01 | |
| *** openstackgerrit has quit IRC | 06:09 | |
| *** ijw_ has quit IRC | 06:20 | |
| *** jamesmcarthur has joined #openstack-infra | 06:28 | |
| *** jtomasek has joined #openstack-infra | 06:29 | |
| *** markvoelker has joined #openstack-infra | 06:29 | |
| *** hwoarang has quit IRC | 06:31 | |
| *** hwoarang has joined #openstack-infra | 06:33 | |
| *** kjackal has joined #openstack-infra | 06:37 | |
| *** kjackal has quit IRC | 06:43 | |
| *** kjackal_v2 has joined #openstack-infra | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:44 | |
| *** dpawlik has joined #openstack-infra | 06:44 | |
| *** pgaxatte has joined #openstack-infra | 06:52 | |
| ianw | frickler / fungi / corvus / clarkb: all comments addressed, and i've manually tested cert generation, so https://review.openstack.org/#/c/636759/ is ready for review, thanks | 06:52 |
| ianw | note there's a couple of things under it too | 06:53 |
| *** ginopc has joined #openstack-infra | 07:02 | |
| *** kopecmartin|off is now known as kopecmartin | 07:02 | |
| *** kjackal_v2 has quit IRC | 07:04 | |
| *** kjackal has joined #openstack-infra | 07:05 | |
| *** tosky has joined #openstack-infra | 07:09 | |
| *** yamamoto has joined #openstack-infra | 07:13 | |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** iurygregory has joined #openstack-infra | 07:18 | |
| *** rpittau|afk is now known as rpittau | 07:23 | |
| *** ccamacho has joined #openstack-infra | 07:24 | |
| *** jpich has joined #openstack-infra | 07:38 | |
| *** kjackal has quit IRC | 07:38 | |
| *** ccamacho has quit IRC | 07:42 | |
| *** zul has quit IRC | 07:44 | |
| *** kjackal has joined #openstack-infra | 07:45 | |
| *** ccamacho has joined #openstack-infra | 07:49 | |
| *** helenafm has joined #openstack-infra | 07:57 | |
| *** ykarel is now known as ykarel|lunch | 07:58 | |
| *** jpena|off is now known as jpena | 07:58 | |
| *** e0ne has quit IRC | 08:01 | |
| *** e0ne has joined #openstack-infra | 08:08 | |
| *** lucasagomes has joined #openstack-infra | 08:12 | |
| *** markmcd has quit IRC | 08:15 | |
| *** wolverineav has joined #openstack-infra | 08:16 | |
| *** raissa has joined #openstack-infra | 08:17 | |
| frickler | mnaser: unreleased==stein? that'd be pretty cool indeed | 08:19 |
| *** wolverineav has quit IRC | 08:20 | |
| *** tkajinam has quit IRC | 08:21 | |
| *** kjackal_v2 has joined #openstack-infra | 08:22 | |
| *** kjackal has quit IRC | 08:22 | |
| *** zbr|pto is now known as zbr | 08:23 | |
| *** dancek has joined #openstack-infra | 08:27 | |
| dancek | Hi. I made this commit that fixes an issue I have with git-review. https://review.openstack.org/#/c/648646/ Should I do something to get it reviewed and discussed? | 08:28 |
| *** xek has joined #openstack-infra | 08:30 | |
| frickler | dancek: welcome as a new contributor. you already did the most important thing in making sure that your patch passes all checks. in general allow folks a couple of working days in order to review patches, you can then help by mentioning it in this channel | 08:32 |
| dancek | Ok! I just wanted to check I didn't miss something. | 08:33 |
| *** priteau has joined #openstack-infra | 08:34 | |
| *** rossella_s has joined #openstack-infra | 08:38 | |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** openstackgerrit has joined #openstack-infra | 08:48 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration https://review.openstack.org/639855 | 08:48 |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| *** raissa has quit IRC | 08:51 | |
| *** kjackal_v2 has quit IRC | 08:52 | |
| *** kjackal has joined #openstack-infra | 08:52 | |
| *** manjeets has quit IRC | 08:56 | |
| *** manjeets has joined #openstack-infra | 08:57 | |
| *** gfidente has joined #openstack-infra | 08:57 | |
| *** gfidente has quit IRC | 08:59 | |
| *** manjeets has quit IRC | 09:05 | |
| *** e0ne has quit IRC | 09:07 | |
| *** roman_g has joined #openstack-infra | 09:09 | |
| *** rcernin has quit IRC | 09:11 | |
| *** gfidente has joined #openstack-infra | 09:13 | |
| *** ccamacho has quit IRC | 09:14 | |
| *** e0ne has joined #openstack-infra | 09:15 | |
| *** eumel8 has joined #openstack-infra | 09:16 | |
| *** kjackal_v2 has joined #openstack-infra | 09:27 | |
| *** kjackal has quit IRC | 09:27 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration https://review.openstack.org/639855 | 09:28 |
| *** Lucas_Gray has joined #openstack-infra | 09:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove TaskManager and just use keystoneauth https://review.openstack.org/640643 | 09:30 |
| mordred | infra-root: openstacksdk 0.27.0 was released with the taskmanager removal landed. this could have nodepool impact | 09:32 |
| mordred | (SHOULD be fine - but still, we should take care with nodepool things for the next bit) | 09:32 |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** electrofelix has joined #openstack-infra | 09:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove TaskManager and just use keystoneauth https://review.openstack.org/640643 | 09:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Pin openstacsksdk to <0.27.0 https://review.openstack.org/649254 | 09:36 |
| *** manjeets has joined #openstack-infra | 09:38 | |
| *** tbachman has joined #openstack-infra | 09:40 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Test Single provider reload https://review.openstack.org/649259 | 09:41 |
| *** e0ne has quit IRC | 09:51 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver https://review.openstack.org/604404 | 09:52 |
| *** zigo has joined #openstack-infra | 09:55 | |
| *** ccamacho has joined #openstack-infra | 10:00 | |
| *** yamamoto has quit IRC | 10:01 | |
| *** kjackal has joined #openstack-infra | 10:03 | |
| *** kjackal_v2 has quit IRC | 10:04 | |
| mordred | AJaeger: morning! anything I should take a look at? | 10:07 |
| *** priteau has quit IRC | 10:07 | |
| *** priteau has joined #openstack-infra | 10:11 | |
| *** yamamoto has joined #openstack-infra | 10:14 | |
| *** wolverineav has joined #openstack-infra | 10:17 | |
| *** yamamoto has quit IRC | 10:18 | |
| *** wolverineav has quit IRC | 10:21 | |
| *** e0ne has joined #openstack-infra | 10:28 | |
| *** bhavikdbavishi has quit IRC | 10:30 | |
| *** kjackal has quit IRC | 10:36 | |
| *** kjackal has joined #openstack-infra | 10:37 | |
| *** yolanda has quit IRC | 10:39 | |
| *** e0ne has quit IRC | 10:45 | |
| *** tbachman has quit IRC | 10:47 | |
| *** diablo_rojo has joined #openstack-infra | 10:51 | |
| *** e0ne has joined #openstack-infra | 10:52 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** nicolasbock has joined #openstack-infra | 10:58 | |
| *** kjackal has quit IRC | 11:00 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:03 | |
| *** diablo_rojo has quit IRC | 11:10 | |
| *** udesale has quit IRC | 11:10 | |
| openstackgerrit | Merged opendev/base-jobs master: Add opendev-release-python job https://review.openstack.org/649199 | 11:10 |
| zbr | ianw: mordred : maybe you can help me kick the bindep change around base py: https://review.openstack.org/#/c/639951/ | 11:14 |
| *** janki has joined #openstack-infra | 11:14 | |
| *** yamamoto has quit IRC | 11:17 | |
| *** kjackal has joined #openstack-infra | 11:21 | |
| *** yolanda has joined #openstack-infra | 11:26 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add trailing slash for log url https://review.openstack.org/648837 | 11:27 |
| *** jpena is now known as jpena|lunch | 11:29 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:32 | |
| frickler | mordred: fungi: you might want to look at https://review.openstack.org/648632 to adapt opensuse mirroring to suse's repo changes | 11:37 |
| mordred | frickler: done | 11:39 |
| *** yamamoto has joined #openstack-infra | 11:42 | |
| frickler | mordred: if you want something trickier, there's this double gate failure https://review.openstack.org/649183 which I couldn't classify at first sight. might be fixed by a recheck, but might also be worth some digging | 11:42 |
| mordred | frickler: I'm not sure I'm smart enough today for something like that | 11:43 |
| mordred | frickler: I'm currently suffering from the plague - so am only partially present | 11:43 |
| frickler | mordred: sorry to hear that, are you still travelling or is that what you returned with? | 11:45 |
| frickler | mordred: also AJaeger mentioned that he'd be busy at susecon this week, so different time zone and not much attention left for infra things | 11:46 |
| mordred | frickler: still travelling - nothing like plague in a hotel room! :( | 11:47 |
| mordred | frickler: tomorrow I travel home - I'm *very* much looking forward to a long plane ride while ill | 11:47 |
| mordred | frickler: and nod | 11:47 |
| frickler | mordred: meh, good luck with that | 11:48 |
| *** amansi26 has quit IRC | 11:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add tap-as-a-service-tempest-plugin repo https://review.openstack.org/648049 | 11:54 |
| *** e0ne has quit IRC | 11:56 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| *** Lucas_Gray has quit IRC | 12:03 | |
| *** e0ne has joined #openstack-infra | 12:03 | |
| *** Lucas_Gray has joined #openstack-infra | 12:04 | |
| *** tbachman has joined #openstack-infra | 12:05 | |
| *** yamamoto has quit IRC | 12:07 | |
| *** bhavikdbavishi has quit IRC | 12:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove unused use_taskmanager flag https://review.openstack.org/649293 | 12:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove reference to TaskManagerStopped exception https://review.openstack.org/649294 | 12:14 |
| *** manjeets has quit IRC | 12:16 | |
| *** diablo_rojo has joined #openstack-infra | 12:16 | |
| *** wolverineav has joined #openstack-infra | 12:17 | |
| *** Lucas_Gray has quit IRC | 12:21 | |
| *** Lucas_Gray has joined #openstack-infra | 12:22 | |
| *** wolverineav has quit IRC | 12:22 | |
| *** jamesmcarthur has quit IRC | 12:23 | |
| *** jamesmcarthur has joined #openstack-infra | 12:23 | |
| *** trown|outtypewww is now known as trown | 12:24 | |
| frickler | infra-root: if someone has time to dig: Build failed (promote pipeline). infra-prod-zuul_reconfigure http://logs.openstack.org/49/648049/2/promote/infra-prod-zuul_reconfigure/de317ec/ : FAILURE in 54s | 12:25 |
| *** markvoelker has quit IRC | 12:25 | |
| *** markvoelker has joined #openstack-infra | 12:25 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| *** e0ne has quit IRC | 12:31 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** jamesmcarthur has quit IRC | 12:33 | |
| *** jamesmcarthur has joined #openstack-infra | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** tbachman has quit IRC | 12:39 | |
| *** jamesmcarthur has quit IRC | 12:40 | |
| *** tbachman has joined #openstack-infra | 12:41 | |
| *** kgiusti has joined #openstack-infra | 12:42 | |
| *** jamesmcarthur has joined #openstack-infra | 12:44 | |
| *** e0ne has joined #openstack-infra | 12:44 | |
| *** eharney has joined #openstack-infra | 12:50 | |
| *** mriedem has joined #openstack-infra | 12:52 | |
| *** quiquell is now known as quiquell|lunch | 13:00 | |
| *** aaronsheffield has joined #openstack-infra | 13:01 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine https://review.openstack.org/640884 | 13:03 |
| *** rh-jelabarre has joined #openstack-infra | 13:03 | |
| *** dikonoor has joined #openstack-infra | 13:04 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/actions endpoint https://review.openstack.org/641099 | 13:04 |
| *** mriedem has quit IRC | 13:08 | |
| *** mriedem has joined #openstack-infra | 13:09 | |
| frickler | infra-root: is it expected that merging updates to zuul.yaml can cause weird effects on other patches that are being tested at the same time? see https://review.openstack.org/#/c/482161/ which I assume is related to this merge https://review.openstack.org/644586 | 13:18 |
| *** lseki has joined #openstack-infra | 13:20 | |
| pabelanger | frickler: http://logs.openstack.org/49/648049/2/promote/infra-prod-zuul_reconfigure/de317ec/job-output.txt.gz#_2019-04-02_11_55_32_413372 bridge.o.o doesn't have python2, so we need to setup ansible_python_interpreter on add_host task | 13:22 |
| *** quiquell|lunch is now known as quiquell | 13:24 | |
| *** lpetrut has joined #openstack-infra | 13:26 | |
| *** priteau has quit IRC | 13:28 | |
| openstackgerrit | Merged openstack/ptgbot master: The PTG is no longer OpenStack-specific https://review.openstack.org/642509 | 13:30 |
| *** trident has quit IRC | 13:30 | |
| openstackgerrit | Merged openstack/ptgbot master: Update links to point to gitea https://review.openstack.org/642510 | 13:32 |
| *** trident has joined #openstack-infra | 13:33 | |
| *** e0ne has quit IRC | 13:36 | |
| *** sthussey has joined #openstack-infra | 13:37 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.openstack.org/641099 | 13:41 |
| *** e0ne has joined #openstack-infra | 13:42 | |
| *** e0ne has quit IRC | 13:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Set ansible_python_interpreter for bridge.o.o https://review.openstack.org/649331 | 13:44 |
| pabelanger | frickler: ^ | 13:44 |
| *** helenafm has quit IRC | 13:44 | |
| *** nowster has joined #openstack-infra | 13:47 | |
| *** eharney_ has joined #openstack-infra | 13:52 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Retire oslosphinx https://review.openstack.org/649072 | 13:53 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Retire cloud-init https://review.openstack.org/649082 | 13:54 |
| *** eharney has quit IRC | 13:55 | |
| *** lpetrut has quit IRC | 14:00 | |
| *** yamamoto has quit IRC | 14:01 | |
| *** yamamoto has joined #openstack-infra | 14:04 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** yamamoto has joined #openstack-infra | 14:05 | |
| *** kgiusti has quit IRC | 14:07 | |
| *** e0ne has joined #openstack-infra | 14:10 | |
| *** quiquell is now known as quiquell|off | 14:11 | |
| efried | Here's an interesting one: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Background%20on%20this%20error%20at%3A%20http%3A%2F%2Fsqlalche.me%2Fe%2Frvf5%5C%22 | 14:13 |
| efried | (e.g. http://logs.openstack.org/50/638150/7/check/openstack-tox-py27/345d1d1/testr_results.html.gz) | 14:13 |
| efried | Something happens every day around 1am (and sometimes around 5 or 6am) that makes sqla race like hell. | 14:13 |
| *** mloza has joined #openstack-infra | 14:16 | |
| *** smcginnis_pto is now known as smcginnis | 14:16 | |
| *** rkukura has quit IRC | 14:17 | |
| *** wolverineav has joined #openstack-infra | 14:18 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 14:19 |
| frickler | efried: it seems those are periodic-stable jobs, they are triggered each day at the same time, so the daily correlation is a red herring as to where the real issue is | 14:20 |
| frickler | efried: see the build_queue field in those hits | 14:22 |
| efried | frickler: I'm like 24h old where logstash is concerned, thanks for bearing with me. | 14:22 |
| *** wolverineav has quit IRC | 14:23 | |
| *** helenafm has joined #openstack-infra | 14:25 | |
| *** dpawlik has quit IRC | 14:26 | |
| *** ccamacho has quit IRC | 14:41 | |
| *** Sundar has joined #openstack-infra | 14:41 | |
| Sundar | I want to merge Cyborg master branch onto the feature branch named cyborg-nova-pilot. I followed https://docs.openstack.org/infra/manual/drivers.html#merge-commits to merge the master onto my feature branch. When I do 'git review -R feature/cyborg-nova-pilot', I get the error: "you are not allowed to upload merges". This seems like a permission issue. Who sets the permissions? | 14:43 |
| Sundar | More details at: More details at: http://paste.openstack.org/show/748710/ | 14:44 |
| corvus | Sundar: it's described in the first section of the documentation link you cited | 14:47 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add grenade-py3 job to Neutron Grafana dashboard https://review.openstack.org/649352 | 14:49 |
| corvus | Sundar: there's more information about that file here: https://docs.openstack.org/infra/manual/creators.html#add-gerrit-permissions | 14:49 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 14:50 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Raise an error if a file in zuul.d is empty https://review.openstack.org/606062 | 14:50 |
| corvus | infra-root: did anyone act on this message from the gerrit community which mentioned us? http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004377.html | 14:51 |
| *** yolanda has quit IRC | 14:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation https://review.openstack.org/649356 | 14:55 |
| *** janki has quit IRC | 14:56 | |
| frickler | corvus: not yet but I was going to mention it in the meeting | 14:57 |
| corvus | i'll wait for that then! | 14:57 |
| mnaser | hmm | 14:57 |
| mnaser | is opendev.org running on sjc1? | 14:57 |
| clarkb | corvus: frickler I responded on the gerrit list over the weekend | 14:58 |
| frickler | corvus: if you want to propose a response earlier, I'm fine with that too. meetings are late for me now that we've switched to dst, too | 14:58 |
| mordred | mnaser: yes | 14:59 |
| *** lpetrut has joined #openstack-infra | 14:59 | |
| mordred | mnaser: so, you know - don't break too many things :) | 14:59 |
| Sundar | corvus: Thanks. May I ask how to locate the "project’s ACL file in the project-config repo"? Presumably it is gerrit/acls/openstack/<projectname>.config. Which repo does that reside in? | 14:59 |
| clarkb | told them we are running java 8 and they dont need to test java 7 for us. Also got some pointers on upgrade path | 14:59 |
| mnaser | mordred: I'll just stop upgrading OpenStack forever that's how it works right | 14:59 |
| mordred | mnaser: nah - keep upgrading it. I mean - look, the openstack code is hosted on vms running on a cloud running the latest openstack | 15:00 |
| mordred | hey developers - don't break things! :) | 15:00 |
| mnaser | :P | 15:00 |
| mnaser | this was by far | 15:00 |
| mnaser | the smoothest upgrade ever | 15:00 |
| mnaser | hmm | 15:04 |
| mnaser | is openstack-tox-docs broken? | 15:04 |
| mnaser | http://logs.openstack.org/47/648947/2/check/openstack-tox-docs/bf373e2/job-output.txt.gz just failed with "AttributeError: 'Sphinx' object has no attribute 'info'" | 15:04 |
| Sundar | corvus: I did find https://github.com/ossdk/config-project/blob/master/gerrit/acls/openstack/cyborg.config | 15:05 |
| mnaser | oh | 15:05 |
| mnaser | sphinx 2.0.0 came out a few days ago | 15:05 |
| mnaser | im gonna guess that might have broke it | 15:05 |
| Sundar | Checking if it's the right one, as githib has several repos with that path and filename | 15:05 |
| *** kgiusti has joined #openstack-infra | 15:06 | |
| mnaser | dhellmann: I think data templates is broken with the latest sphinx | 15:07 |
| clarkb | corvus: frickler https://groups.google.com/d/msg/repo-discuss/TP9ozTsazfU/9_3yXwDzAAAJ direct link to my reply | 15:07 |
| *** tbachman_ has joined #openstack-infra | 15:07 | |
| *** tbachman has quit IRC | 15:08 | |
| *** tbachman_ is now known as tbachman | 15:08 | |
| *** rh-jelabarre has quit IRC | 15:08 | |
| *** e0ne has quit IRC | 15:11 | |
| *** yolanda has joined #openstack-infra | 15:13 | |
| *** lpetrut has quit IRC | 15:14 | |
| clarkb | corvus: fungi I've noticed that another difference in the configs of my test list and -discuss is the subject prefix on my test list. After tea and breakfast I'll remove that from my test list and send another test email to see if that avoids the base64 conversion as with -discuss | 15:15 |
| clarkb | sometimes fresh eyeballs help | 15:15 |
| fungi | i can't imagine that the subject mangling would influence it since that's with the header and not the body, but... who knows | 15:15 |
| clarkb | fungi: I think it may all pass through the same decorate() method | 15:16 |
| clarkb | so its possible that it is side affecting in an unexpected way | 15:16 |
| fungi | oh, got i | 15:16 |
| fungi | t | 15:16 |
| fungi | clarkb: also, one of your test messages successfully avoided getting b64 encoded | 15:16 |
| clarkb | fungi: yes that was the one where I set en back to ascii | 15:16 |
| clarkb | to confirm that that fixes it | 15:16 |
| fungi | oh, got it | 15:16 |
| clarkb | fungi: if you look at the bottom of /etc/mailman/mm_cfg.py you'll see what I did to do that (currently commented out) | 15:17 |
| fungi | yeah, it was the "Testing again with ascii lang" message, confirmed | 15:17 |
| *** ykarel is now known as ykarel|away | 15:18 | |
| *** ykarel|away has quit IRC | 15:23 | |
| *** eharney_ has quit IRC | 15:23 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add web / fingergw connections for components graph https://review.openstack.org/645852 | 15:24 |
| *** jamesmcarthur has quit IRC | 15:24 | |
| *** jamesmcarthur has joined #openstack-infra | 15:25 | |
| *** diablo_rojo has quit IRC | 15:26 | |
| *** helenafm has quit IRC | 15:26 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine https://review.openstack.org/640884 | 15:32 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.openstack.org/641099 | 15:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add database info for components graph https://review.openstack.org/649371 | 15:34 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry https://review.openstack.org/642408 | 15:37 |
| *** e0ne has joined #openstack-infra | 15:38 | |
| *** michael-beaver has joined #openstack-infra | 15:43 | |
| *** jamesmcarthur has quit IRC | 15:44 | |
| *** pgaxatte has quit IRC | 15:44 | |
| *** ccamacho has joined #openstack-infra | 15:46 | |
| *** yolanda has quit IRC | 15:46 | |
| *** yolanda has joined #openstack-infra | 15:47 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:47 | |
| *** hamzy has quit IRC | 15:49 | |
| clarkb | fungi: corvus: changing the subject prefix thing didn't change the base64 encoding behavior | 15:50 |
| clarkb | the only other difference in the configs that may cause a difference is the # -*- coding: us-ascii -*- prefix is ascii on -discuss and utf8 on upgrate-testing | 15:52 |
| clarkb | that comes from the pickle right | 15:52 |
| zigo | clarkb: Tested and working. | 15:54 |
| zigo | I commented in the github place. | 15:54 |
| clarkb | zigo: great thank you! | 15:55 |
| *** iurygregory has quit IRC | 15:55 | |
| *** lucasagomes has quit IRC | 15:56 | |
| *** michael-beaver has quit IRC | 16:00 | |
| *** michael-beaver has joined #openstack-infra | 16:01 | |
| dhellmann | mnaser : in the past when sphinx has broken things we've just capped it until we could get the fixes out. I know stephenfin is also working on fixing issues. | 16:03 |
| stephenfin | mnaser: What's broken? | 16:03 |
| stephenfin | I've already fixed a few things | 16:04 |
| *** jamesmcarthur has joined #openstack-infra | 16:04 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use xterm.js for live log streaming https://review.openstack.org/648838 | 16:05 |
| fungi | stephenfin: http://logs.openstack.org/47/648947/2/check/openstack-tox-docs/bf373e2/job-output.txt.gz just failed with "AttributeError: 'Sphinx' object has no attribute 'info'" | 16:09 |
| fungi | (from mnaser's message at 15:04z) | 16:09 |
| stephenfin | fungi: Looks like smcginnis has already handled it https://github.com/sphinx-contrib/datatemplates/pull/7 | 16:10 |
| fungi | excellent | 16:11 |
| fungi | thanks smcginnis and stephenfin! | 16:12 |
| *** imacdonn has joined #openstack-infra | 16:12 | |
| *** ykarel|away has joined #openstack-infra | 16:13 | |
| smcginnis | And thanks dhellmann for getting it merged and in a release quickly! | 16:14 |
| mnaser | can we recheck patches as 0.2.0 seems to be on pypi now? | 16:15 |
| stephenfin | mnaser: I think you'll need to bump requirements/lower-constraints which will require a bump of openstack/requirements (for upper) | 16:16 |
| stephenfin | at least that what I assumed for the other Sphinx'y things I've been fixing | 16:16 |
| *** Lucas_Gray has quit IRC | 16:16 | |
| dhellmann | in the past we have just modified the doc jobs to cap sphinx so we didn't have to deal with the firedrill | 16:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation https://review.openstack.org/649356 | 16:16 |
| stephenfin | dhellmann: I'd be inclined to suggest we enable DeprecationWarnings for Sphinx in said job going forward, but that would necessitate someone doing something about them :-/ | 16:17 |
| dhellmann | yeah, that would trigger the issue earlier but not solve the "who owns this" problem | 16:18 |
| stephenfin | I do wonder how the JS community, with their "module for practically everything", solve this | 16:18 |
| stephenfin | *"...everything" mentality | 16:19 |
| dhellmann | pinning requirements? | 16:19 |
| smcginnis | There's probably a module to handle it. | 16:19 |
| smcginnis | :) | 16:19 |
| * dhellmann wanders off to find food | 16:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Increase timeout of zuul-quick-start https://review.openstack.org/649384 | 16:19 |
| *** yolanda has quit IRC | 16:19 | |
| fungi | mnaser: might need to get into upper-constraints.txt first? | 16:19 |
| *** wolverineav has joined #openstack-infra | 16:19 | |
| *** rh-jelabarre has joined #openstack-infra | 16:20 | |
| mnaser | I can do that once OSA meeting (and lunch, concurrently) is over.. unless someone else picks that up | 16:20 |
| fungi | stephenfin: yeah, in javascript land you copy all your dependendencies into your tree and bundle them. so not just pinning to a very specific version but then possibly also patching/forking that embedded copy | 16:20 |
| *** jamesmcarthur has quit IRC | 16:21 | |
| fungi | granted, most of the rest of the python world doesn't use ranges in their dependency declarations, they just do == some specific version and then at some point test whether it's safe to use something newer (which is sort of what we ended up with in our constraints model too) | 16:22 |
| stephenfin | fungi: Hmm, so not much better than us | 16:22 |
| stephenfin | Yeah, there lies the crux of the issue - someone has to do that | 16:22 |
| stephenfin | Having made the mistake of looking at the amount of projects _still_ using oslosphinx, I've come to the conclusion that large scale dependency management/herding is hard | 16:22 |
| fungi | yes, probably a case study in np-completeness | 16:23 |
| *** rpittau is now known as rpittau|afk | 16:24 | |
| mordred | stephenfin: yeah - the javascript ecosystem has much better built in tooling for dependency management and pinning - and javascript things don't expect to install libraries globally but instead to install libraries into the build environment and then produce a combined tree - so they dont have the co-installability problem | 16:24 |
| *** dtantsur is now known as dtantsur|afk | 16:24 | |
| mordred | so - better tools + easier problem space | 16:24 |
| fungi | and indeed, looks like https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt#L400-L402 need updating for newer sphinx now | 16:25 |
| *** wolverineav has quit IRC | 16:25 | |
| stephenfin | mordred: True. I never did get why Python packages couldn't be version namespaced (or a similar scheme), at a minimum | 16:26 |
| *** jamesmcarthur has joined #openstack-infra | 16:27 | |
| stephenfin | mordred: You won't have time to read it, but I imagine you'd agree with most things in here https://speakerdeck.com/mitsuhiko/a-future-python | 16:27 |
| openstackgerrit | Sundar Nadathur proposed openstack-infra/project-config master: Enable Cyborg core team to push merge commits. https://review.openstack.org/649390 | 16:27 |
| clarkb | the way pip installed things in the past they sort of were | 16:27 |
| clarkb | you just couldn't access the other versions easily from your code | 16:27 |
| smcginnis | fungi, mnaser: Does someone want to request a FFE to raise that upper-constraint? | 16:27 |
| clarkb | I believe they fixed this with pip 10 and is part of the whole can't install if old version was distutils installed problem | 16:27 |
| fungi | in fact we can basically blame pip for a lot of this. python has great *internal* spport for using multiple versions of the same modyles | 16:27 |
| fungi | pip on the other hand figures it's not sane to ever have more than one version of any module importable | 16:28 |
| *** e0ne has quit IRC | 16:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation https://review.openstack.org/649356 | 16:30 |
| fungi | smcginnis: i don't think we need a ffe do we? https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-constraints.txt#L389 | 16:30 |
| stephenfin | fungi: Yup, setuptools actually allowed for some of that and a lot more besides, it seems http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ | 16:30 |
| stephenfin | fungi: I suspect their doc jobs aren't using U-C | 16:31 |
| fungi | smcginnis: the broken version is only in master constraints, not in stable/stein | 16:31 |
| fungi | stephenfin: or those jobs are running on master? | 16:31 |
| fungi | does anyone know if stable/stein changes are failing? | 16:32 |
| smcginnis | I didn't think we capped sphinx, so I assumed we would need it both places. | 16:32 |
| *** ccamacho has quit IRC | 16:32 | |
| fungi | we don't cap sphinx, but stable/stein upper requirements froze at Sphinx===1.8.4 for all interpreters | 16:33 |
| fungi | er, upper constraints | 16:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation https://review.openstack.org/649356 | 16:33 |
| stephenfin | we don't cap Sphinx? What's this doing so? https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L399-L402 | 16:33 |
| * stephenfin might be confusing terminology | 16:33 | |
| fungi | stephenfin: a constraint isn't a "cap" per se | 16:34 |
| *** michaelbeaver has joined #openstack-infra | 16:34 | |
| stephenfin | fungi: yeah, you're correct, it's a master problem. Constraints are used, as expected, for openstack/releases anyway https://github.com/openstack/releases/blob/master/tox.ini#L72 | 16:34 |
| *** ramishra has quit IRC | 16:34 | |
| stephenfin | oh, I'd just equated the two | 16:34 |
| fungi | "capping" is generally interpreted as setting something like Sphinx<2 in the global-requirements.txt | 16:35 |
| *** michael-beaver has quit IRC | 16:36 | |
| fungi | global-requirements.txt is where we track the set of dependency versions we expect openstack to work with, upper-constraints.txt is where we set the specific set of versions we test everything with | 16:36 |
| smcginnis | https://review.openstack.org/649395 | 16:37 |
| fungi | ahh, so it was sphinxcontrib.datatemplates which got a new fixed version, not sphinx itself? i obviously wasn't following that closely enough | 16:38 |
| fungi | in theory we could also set !=0.1.0 on the versionspec for that in global-requirements.txt for clarity that we don't expect it to be usable in master (due to the newer sphinx version we're using in master) | 16:40 |
| fungi | but this is probably a discussion better had in #openstack-requirements | 16:40 |
| *** gfidente is now known as gfidente|afk | 16:45 | |
| openstackgerrit | Sundar Nadathur proposed openstack-infra/project-config master: Enable Cyborg release team to push merge commits. https://review.openstack.org/649390 | 16:47 |
| *** trown is now known as trown|lunch | 16:50 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:53 | |
| *** dikonoor has quit IRC | 16:55 | |
| *** jamesmcarthur has quit IRC | 16:57 | |
| clarkb | fungi: looking at the config encoding for lists rust-vmm is utf8 whcih was created relatively recently like openstack-discuss was. Any idea how they end up with different config encodings? | 16:57 |
| *** jpich has quit IRC | 16:58 | |
| *** gfidente|afk has quit IRC | 16:58 | |
| *** hamzy has joined #openstack-infra | 17:05 | |
| clarkb | starlingx-discuss was added before openstack-discuss and starlingx-discuss is utf8 | 17:05 |
| clarkb | so now I am extra stumped | 17:05 |
| clarkb | fungi: we let puppet create the new -discuss list right? so it shouldn't really be any different than the others | 17:06 |
| clarkb | (mostly just fishing for any other differences I can test with) | 17:06 |
| *** dpawlik has joined #openstack-infra | 17:09 | |
| *** eharney has joined #openstack-infra | 17:11 | |
| *** Lucas_Gray has joined #openstack-infra | 17:14 | |
| *** dpawlik has quit IRC | 17:15 | |
| clarkb | infra-root https://review.openstack.org/#/c/647875/3 is a puppet-4 upgrade that I'll be happy to watch after our meeting today if anyone else can review it | 17:17 |
| *** ricolin has quit IRC | 17:17 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add search function to web streaming https://review.openstack.org/648862 | 17:18 |
| *** rkukura has joined #openstack-infra | 17:20 | |
| *** rh-jelabarre has quit IRC | 17:25 | |
| *** jamesmcarthur_ has quit IRC | 17:28 | |
| *** jpena is now known as jpena|off | 17:28 | |
| *** rh-jelabarre has joined #openstack-infra | 17:29 | |
| *** michael-beaver has joined #openstack-infra | 17:32 | |
| *** michaelbeaver has quit IRC | 17:34 | |
| *** wolverineav has joined #openstack-infra | 17:34 | |
| fungi | clarkb: yeah, openstack-discuss was simply created by puppet when https://review.openstack.org/602781 merged | 17:35 |
| fungi | it's possible there's something i set in the admin interface for it which made that happen, i'll dig a little | 17:35 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Make docs secret template expansion safe https://review.openstack.org/649408 | 17:36 |
| corvus | clarkb, fungi: ^ i've confirmed the opendev jobs work for the zuul tenant, that's a backport of the last fix needed to the opendev-tenant versions of those jobs. | 17:37 |
| *** eharney has quit IRC | 17:37 | |
| *** Lucas_Gray has quit IRC | 17:38 | |
| *** shananigans has joined #openstack-infra | 17:39 | |
| *** ijw has joined #openstack-infra | 17:39 | |
| *** tbachman has quit IRC | 17:45 | |
| openstackgerrit | Sundar Nadathur proposed openstack-infra/project-config master: Enable Cyborg release team to push merge commits. https://review.openstack.org/649390 | 17:46 |
| *** armax has joined #openstack-infra | 17:53 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/irc-meetings master: Update keystone meeting chair https://review.openstack.org/649411 | 17:54 |
| *** tbachman has joined #openstack-infra | 17:54 | |
| *** trown|lunch is now known as trown | 17:55 | |
| *** psachin has quit IRC | 17:55 | |
| *** bhavikdbavishi has quit IRC | 17:58 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Make docs secret template expansion safe https://review.openstack.org/649408 | 18:01 |
| *** eharney has joined #openstack-infra | 18:02 | |
| *** kopecmartin is now known as kopecmartin|off | 18:04 | |
| *** e0ne has joined #openstack-infra | 18:04 | |
| *** wolverineav has quit IRC | 18:06 | |
| *** wolverineav has joined #openstack-infra | 18:14 | |
| *** wolverineav has quit IRC | 18:14 | |
| corvus | clarkb: ^ updated to include another fix which didn't get merged in time and conflicted | 18:15 |
| *** electrofelix has quit IRC | 18:18 | |
| *** rlandy is now known as rlandy|brb | 18:19 | |
| *** wolverineav has joined #openstack-infra | 18:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add database info for components graph https://review.openstack.org/649371 | 18:20 |
| *** tbachman has quit IRC | 18:27 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Pass ssh host key to upload-git-mirror for integration tests https://review.openstack.org/649183 | 18:29 |
| *** spsurya has quit IRC | 18:32 | |
| *** tbachman has joined #openstack-infra | 18:33 | |
| *** tbachman_ has joined #openstack-infra | 18:36 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Fix AWS provider ignoring region in providers https://review.openstack.org/649423 | 18:36 |
| *** rlandy|brb is now known as rlandy | 18:36 | |
| *** tbachman has quit IRC | 18:37 | |
| *** tbachman_ is now known as tbachman | 18:37 | |
| openstackgerrit | Merged opendev/base-jobs master: Make docs secret template expansion safe https://review.openstack.org/649408 | 18:38 |
| *** ykarel|away has quit IRC | 18:40 | |
| openstackgerrit | Artom Lifshitz proposed openstack-infra/project-config master: Add whitebox-tempest-plugin project https://review.openstack.org/649425 | 18:43 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/zuul-jobs master: Propose submit-logstash role to export logs https://review.openstack.org/649426 | 18:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prepend path with bin dir of ansible virtualenv https://review.openstack.org/649427 | 18:48 |
| *** wolverineav has quit IRC | 18:49 | |
| clarkb | friendly reminder: Infra meeting in ~10 minutes | 18:49 |
| fungi | wow, where did today go? | 18:50 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Cleanup executor specific requirements https://review.openstack.org/649428 | 18:50 |
| clarkb | I had to update /etc/hosts to point at the real lists server to get the meeting agenda link | 18:51 |
| *** wolverineav has joined #openstack-infra | 18:51 | |
| *** tbachman has quit IRC | 18:53 | |
| *** wolverineav has quit IRC | 18:55 | |
| *** dpawlik has joined #openstack-infra | 18:58 | |
| *** wolverineav has joined #openstack-infra | 18:59 | |
| *** dpawlik has quit IRC | 19:02 | |
| *** wolverineav has quit IRC | 19:02 | |
| *** wolverineav has joined #openstack-infra | 19:03 | |
| *** gmann is now known as gmann_afk | 19:04 | |
| *** wolverineav has quit IRC | 19:08 | |
| *** rosmaita has joined #openstack-infra | 19:13 | |
| *** owalsh_ has joined #openstack-infra | 19:15 | |
| *** owalsh has quit IRC | 19:16 | |
| *** e0ne has quit IRC | 19:20 | |
| *** rh-jelabarre has quit IRC | 19:26 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fixed Scope Groups Update https://review.openstack.org/649431 | 19:26 |
| *** rh-jelabarre has joined #openstack-infra | 19:30 | |
| *** ianychoi has quit IRC | 19:32 | |
| *** ianychoi has joined #openstack-infra | 19:33 | |
| *** wolverineav has joined #openstack-infra | 19:44 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine https://review.openstack.org/640884 | 19:48 |
| *** dpawlik has joined #openstack-infra | 19:48 | |
| corvus | clarkb: i just verified that our override of the mailman GLOBAL_PIPELINE isn't missing any new entries in the current mm version | 19:49 |
| *** wolverineav has quit IRC | 19:49 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.openstack.org/641099 | 19:50 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.openstack.org/641099 | 19:52 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry https://review.openstack.org/642408 | 19:53 |
| *** dpawlik has quit IRC | 19:53 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry https://review.openstack.org/642408 | 19:56 |
| fungi | thanks corvus, i had worried that could be the case so nice to know we don't need to adjust for that anyway | 19:57 |
| *** wolverineav has joined #openstack-infra | 19:59 | |
| *** rkukura has quit IRC | 20:00 | |
| fungi | clarkb: oops, i meant to mention... you know that old cliché about husbands forgetting their wedding anniversaries? well, turns out april 19 should have seemed familiar when we proposed it for the opendev maintenance | 20:00 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 20:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Allow operator to generate auth tokens through the CLI https://review.openstack.org/636197 | 20:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul CLI: allow access via REST https://review.openstack.org/636315 | 20:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Add Authorization Rules configuration https://review.openstack.org/639855 | 20:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 20:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/actions endpoint https://review.openstack.org/641099 | 20:01 |
| *** markmcclain has quit IRC | 20:01 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry https://review.openstack.org/642408 | 20:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend https://review.openstack.org/643536 | 20:01 |
| clarkb | fungi: you need a calendar with a great big mark on that day? | 20:01 |
| *** wolverineav has quit IRC | 20:01 | |
| *** wolverineav has joined #openstack-infra | 20:01 | |
| fungi | my digital calendar already says "wedding anniversary" on that day, i just didn't think to consult it | 20:02 |
| fungi | anyway, i expect to be around a good chunk of the day regardless. chris thinks she might need to show property to some clients that day anyway | 20:02 |
| clarkb | ok I'm going to find lunch then will return to reading diffs and spinning up a xenial test node | 20:03 |
| clarkb | I wish I could identify in mailman where it wants to do that conversion would probably help narrow down the reading | 20:04 |
| clarkb | I thought it was decorate but avoiding that function seems to not change things | 20:04 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 20:05 |
| openstackgerrit | Eric Fried proposed openstack-infra/elastic-recheck master: Update query for 1820892 https://review.openstack.org/649445 | 20:05 |
| efried | mriedem: ^ | 20:05 |
| *** shananigans has quit IRC | 20:06 | |
| fungi | clarkb: the other thing which came to mind when digging into the dkim signature invalidation cases was trying to plumb content through each of the pipeline modules individually and analyzing the alterations done by each in turn | 20:07 |
| clarkb | ooh that sounds like it may help anrrow it down | 20:07 |
| fungi | but having looked at a few, it's less of a unix pipeline model and more of a series of plugins using common function names and parameters | 20:07 |
| clarkb | any idea how to do that? | 20:07 |
| clarkb | ah | 20:07 |
| fungi | so would probably need to import each module and then feed the message pits into the requisite parameters | 20:08 |
| fungi | er, message bits | 20:08 |
| fungi | some of them are probably also pits though | 20:08 |
| *** xek has quit IRC | 20:08 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Allow operator to generate auth tokens through the CLI https://review.openstack.org/636197 | 20:08 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 20:10 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration https://review.openstack.org/639855 | 20:11 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine https://review.openstack.org/640884 | 20:12 |
| *** Lucas_Gray has joined #openstack-infra | 20:12 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.openstack.org/641099 | 20:12 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry https://review.openstack.org/642408 | 20:13 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend https://review.openstack.org/643536 | 20:14 |
| *** Lucas_Gray has quit IRC | 20:14 | |
| *** Lucas_Gray has joined #openstack-infra | 20:15 | |
| *** Lucas_Gray has quit IRC | 20:15 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index. https://review.openstack.org/644927 | 20:16 |
| openstackgerrit | Merged openstack-infra/zuul master: Document how to run eslint locally https://review.openstack.org/648855 | 20:16 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index. https://review.openstack.org/644927 | 20:17 |
| *** Lucas_Gray has joined #openstack-infra | 20:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Use mirror infrastructure during zuul-quick-start https://review.openstack.org/649448 | 20:18 |
| *** Lucas_Gray has quit IRC | 20:20 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Use mirror infrastructure during zuul-quick-start https://review.openstack.org/649448 | 20:21 |
| *** Lucas_Gray has joined #openstack-infra | 20:22 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation https://review.openstack.org/649356 | 20:22 |
| *** markvoelker has quit IRC | 20:23 | |
| *** Lucas_Gray has quit IRC | 20:24 | |
| Sundar | Could somebody review https://review.openstack.org/#/c/649390/ or at least let me know if it is in the right direction? | 20:25 |
| *** Lucas_Gray has joined #openstack-infra | 20:25 | |
| *** Lucas_Gray has quit IRC | 20:25 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Slide presentation for Zuul's OIP Confirmation https://review.openstack.org/649356 | 20:26 |
| *** Lucas_Gray has joined #openstack-infra | 20:28 | |
| *** Lucas_Gray has quit IRC | 20:29 | |
| *** pcaruana has quit IRC | 20:30 | |
| *** whoami-rajat has quit IRC | 20:30 | |
| *** Lucas_Gray has joined #openstack-infra | 20:31 | |
| *** Lucas_Gray has quit IRC | 20:33 | |
| dmsimard | Need a +3 to merge https://review.openstack.org/#/c/649033/ for the upload-git-mirror job | 20:33 |
| *** Lucas_Gray has joined #openstack-infra | 20:34 | |
| fungi | Sundar: lgtm | 20:35 |
| *** Lucas_Gray has quit IRC | 20:36 | |
| *** Lucas_Gray has joined #openstack-infra | 20:37 | |
| *** hamzy has quit IRC | 20:41 | |
| Sundar | fungi: Thank you very much. | 20:41 |
| Sundar | fungi: I thought I was already part of cyborg-release: https://review.openstack.org/#/admin/groups/1244,members. Did I miss something? | 20:42 |
| fungi | Sundar: oh, i didn't realize that group was already referenced by a different acl | 20:46 |
| fungi | you can just ignore my comment in that case | 20:46 |
| openstackgerrit | Terry Wilson proposed openstack/diskimage-builder master: [wip] rhel8 beta support https://review.openstack.org/623137 | 20:47 |
| *** jamesmcarthur has joined #openstack-infra | 20:47 | |
| *** jamesmcarthur has quit IRC | 20:47 | |
| *** jamesmcarthur has joined #openstack-infra | 20:47 | |
| Sundar | fungi: Cool. Thanks. | 20:51 |
| zigo | clarkb: Your patch for python-pip got unblocked for landing in Buster. | 20:53 |
| zigo | :) | 20:53 |
| *** adam_g has quit IRC | 20:53 | |
| zigo | We just need to way 8 days now. | 20:53 |
| *** dpawlik has joined #openstack-infra | 20:54 | |
| *** dpawlik has quit IRC | 20:58 | |
| clarkb | zigo: cool hopefully that makes pip a bit more useable on debuntu | 21:01 |
| clarkb | I guess in this case just debian to start | 21:01 |
| *** priteau has joined #openstack-infra | 21:03 | |
| *** Lucas_Gray has quit IRC | 21:04 | |
| *** guimaluf has joined #openstack-infra | 21:04 | |
| fungi | yeah, need someone like jamespage or coreycb to help us through an sru for bionic next | 21:04 |
| *** trown is now known as trown|outtypewww | 21:04 | |
| clarkb | I've rereviewed theubuntu patches on mailman on xenial and trusty and the xenial set is a subset of the trusty set and the trusty set don't seem to do anything that would change this behavior (other than both trusty and xenail trying hard to default to utf8) | 21:06 |
| *** slaweq has quit IRC | 21:07 | |
| clarkb | another thing to consider is this could be a python email lib difference | 21:07 |
| guimaluf | hi guys, I'm using magnum to deploy a k8s cluster, but when I call the `coe config cluster` it gives me this error "a bytes-like object is required, not 'str'" does anyone have any hint ou tip about it? thank in advance | 21:09 |
| clarkb | guimaluf: the magnum team is probably a better place to ask (this is the channel for the development infrastructure used to build openstack), but if I had to guess you are running that under python3 and it isn't python3 safe. | 21:12 |
| fungi | guimaluf: you might check https://wiki.openstack.org/wiki/IRC for a more appropriate channel | 21:12 |
| guimaluf | sorry clarkb and fungi , I ask here cause I though was something related to operations regards the whole infra. sorry for bothering | 21:12 |
| *** kjackal has quit IRC | 21:13 | |
| fungi | guimaluf: no need to apologize. we see that confusion a lot. our choice to refer to the systems which support our community's efforts as "infrastructure" has led to a lot of folks being confused that it's a channel about running an openstack-based infrastructure | 21:13 |
| *** michael-beaver has left #openstack-infra | 21:14 | |
| guimaluf | fungi, exactly! :) thanks for clarifying. | 21:15 |
| guimaluf | fungi, and thank you for pointing me out the IRC channels, I was trying #openstack-magnum which is empty | 21:15 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Update query for 1820892 https://review.openstack.org/649445 | 21:16 |
| *** wolverineav has quit IRC | 21:16 | |
| fungi | yep, they seem to congregate in #openstack-containers | 21:17 |
| *** wolverineav has joined #openstack-infra | 21:18 | |
| *** wolverineav has quit IRC | 21:18 | |
| *** wolverineav has joined #openstack-infra | 21:19 | |
| *** jamesmcarthur has quit IRC | 21:19 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add parameter to upload-git-mirror to specify ssh host key https://review.openstack.org/649033 | 21:22 |
| *** slaweq has joined #openstack-infra | 21:24 | |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/zuul-jobs master: Propose submit-logstash role to export logs https://review.openstack.org/649426 | 21:24 |
| *** derekh has quit IRC | 21:26 | |
| *** wolverineav has quit IRC | 21:28 | |
| *** slaweq has quit IRC | 21:28 | |
| Sundar | fungi or somebody: Can you please give Workflow +1 to https://review.openstack.org/#/c/649390/ for it to merge? Thanks in advance. | 21:30 |
| *** wolverineav has joined #openstack-infra | 21:30 | |
| fungi | Sundar: ideally we'll have a second config-core reviewer take a look at that | 21:31 |
| openstackgerrit | Tim Burke proposed openstack-infra/zuul-jobs master: Exclude linux_x86_64 wheels from upload to PyPI https://review.openstack.org/649455 | 21:32 |
| *** wolverineav has quit IRC | 21:34 | |
| *** wolverineav has joined #openstack-infra | 21:34 | |
| clarkb | looks like ianw got it | 21:35 |
| ianw | fungi / Sundar : lgtm | 21:35 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend https://review.openstack.org/643536 | 21:38 |
| clarkb | fungi: corvus there are some changes to WrapMessage that looks suspicious | 21:38 |
| clarkb | I'm trying to work backward from my test emails to see if there is a chance they may hae passed through that | 21:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable Cyborg release team to push merge commits. https://review.openstack.org/649390 | 21:43 |
| clarkb | in particular we went from msg.set_payload(omsg.as_string()) to msg.set_payload([omsg]) | 21:44 |
| clarkb | currently thinking that perhaps as_string() was what forced the string output but now with it setting the payload to the msg object it is just taking that format | 21:44 |
| clarkb | but still working to confirm that | 21:45 |
| *** jamesmcarthur has joined #openstack-infra | 21:45 | |
| clarkb | hrm nevermind we don't use that filter | 21:50 |
| clarkb | it is in our global pipeline but I don't think we set the setting to use it | 21:50 |
| *** jamesmcarthur has quit IRC | 21:50 | |
| *** kgiusti has left #openstack-infra | 21:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Run opendev-release-python on release https://review.openstack.org/649456 | 21:54 |
| clarkb | fungi: corvus if I wanted to double check that that isn't the problem I can modify the global list in the vhosted mm_cfg.py and restart mailman then send email right? | 21:54 |
| fungi | by "modify" you mean remove it from the overridden pipeline there? | 21:55 |
| fungi | if so, yes that should be sufficient | 21:55 |
| clarkb | ya. I'm a bit confused about whether or not we are still calling the process() function on the pipeline | 21:56 |
| clarkb | I think we are then process() is expected to check its config and if I'm reading it correctly I think it may still be running just not doing the fully munge | 21:56 |
| clarkb | I'll give it a go pulling it out entirely as that should rule it out or in | 21:56 |
| corvus | clarkb: yes -- you can just remove that section entirely | 21:56 |
| corvus | yeah | 21:56 |
| corvus | it'll use what's in Defaults.py | 21:57 |
| *** wolverineav has quit IRC | 21:57 | |
| *** tbachman has joined #openstack-infra | 21:58 | |
| clarkb | WrapMessage is in Defaults.py so I think I want to modify the overridden value | 21:58 |
| clarkb | I'm trying that now | 21:58 |
| Sundar | ianw, fungi. Thank you very much. | 21:58 |
| *** wolverineav has joined #openstack-infra | 22:00 | |
| *** wolverineav has quit IRC | 22:00 | |
| *** wolverineav has joined #openstack-infra | 22:01 | |
| clarkb | no difference so that rules out this change as being the source of the issue I think | 22:02 |
| clarkb | oooh I can write customer handlers | 22:07 |
| clarkb | fungi: ^ I think writing a custom handler to check message encoding might be easier than checking pipeline one at a time. Then we can binary search the custom handler around in the list and see where it happens to narrow it down | 22:08 |
| clarkb | I'm going to explore this avenue of testing now | 22:08 |
| *** harlowja has joined #openstack-infra | 22:10 | |
| *** priteau has quit IRC | 22:15 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Always pause docker build jobs https://review.openstack.org/649461 | 22:16 |
| *** wolverineav has quit IRC | 22:16 | |
| fungi | clarkb: oh, that's a great idea. basically tackling it in the other direction | 22:18 |
| fungi | inject our own debugging module rather than trying to debug the existing modules | 22:18 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Exclude linux_x86_64 wheels from upload to PyPI https://review.openstack.org/649455 | 22:19 |
| fungi | the interface for those seems pretty simple | 22:19 |
| *** wolverineav has joined #openstack-infra | 22:19 | |
| fungi | what i find a little unclean is that they mostly seem to rely on pass-by-reference with mutable objects | 22:20 |
| fungi | though i suppose it's more efficient than returning from functions, it's a little too far into the implcit-rather-than-explicit realm for my tastes | 22:21 |
| fungi | granted i still sort of write python as if i'm writing c | 22:21 |
| fungi | so you'd think just passing around a pointer from function to function wouldn't weird me out so much | 22:22 |
| *** rcernin has joined #openstack-infra | 22:25 | |
| *** jamesmcarthur has joined #openstack-infra | 22:25 | |
| mordred | fungi: you know - we could ditch all this python - and all the shell scripts - and just revert to writing C for everything | 22:26 |
| fungi | maybe next season | 22:27 |
| fungi | would make a nice cliff-hanger for the season finale | 22:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Use opendev docker jobs https://review.openstack.org/649465 | 22:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Use opendev docker jobs https://review.openstack.org/649466 | 22:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Use the buildset registry in zuul-quick-start https://review.openstack.org/649467 | 22:28 |
| clarkb | cool I have it logging an error now. Not the data I want logged but that means it is running my handler and failing | 22:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Use the buildset registry in zuul-quick-start https://review.openstack.org/649468 | 22:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Use the buildset registry in zuul-quick-start https://review.openstack.org/649467 | 22:31 |
| clarkb | fungi: is it the To: header I need to avoid the implicit destination? | 22:32 |
| corvus | clarkb: yes | 22:32 |
| fungi | yes, or cc | 22:32 |
| fungi | either of those will solve it | 22:32 |
| fungi | the implicit destination check fires when the list address isn't apparent in either the to or cc headers (for example if it was bcc'd) | 22:33 |
| *** slaweq has joined #openstack-infra | 22:38 | |
| *** slaweq has quit IRC | 22:43 | |
| clarkb | ok I put my handler at the end of the list and dumped msg.get_payload() to the error log. By default decode is set to false (supposedly) and I get my message in ascii not base64 | 22:43 |
| clarkb | so this is happening after or external to the handler pipeline | 22:44 |
| fungi | eerie | 22:44 |
| corvus | i guess that puts exim back in the mix as well | 22:45 |
| corvus | clarkb: here's an idea for further debugging: | 22:45 |
| corvus | rather than start all of the mailman processes, only start one at a time. inspect the message in the qfiles directory after each process handles it | 22:46 |
| corvus | i think 3 mailman processes handle messages | 22:46 |
| corvus | incomming, and outgoing certainly... maybe that's it for a normal message | 22:47 |
| corvus | /usr/bin/python /var/lib/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s | 22:48 |
| clarkb | in that case it would be send email, look in qfiles, start incoming, look in qfiles, start outgoing, look in qfiles? | 22:48 |
| corvus | /usr/bin/python /var/lib/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s | 22:48 |
| corvus | so those 2 processes specifically | 22:48 |
| corvus | clarkb: yep | 22:48 |
| clarkb | ok I can try that | 22:48 |
| *** rlandy has quit IRC | 22:49 | |
| corvus | i expect that'll need the site env variable set | 22:50 |
| *** jamesmcarthur has quit IRC | 22:50 | |
| *** eernst has joined #openstack-infra | 22:50 | |
| *** eernst has quit IRC | 22:53 | |
| *** tkajinam has joined #openstack-infra | 22:55 | |
| clarkb | corvus: what qfiles dir will it go into when I start the outgoing running? | 22:56 |
| clarkb | currently I've started incoming which moved it from in/ to out/ | 22:56 |
| clarkb | (and hasn't converted to base64 yet | 22:56 |
| corvus | clarkb: oh, i guess out will go from out/ to exim | 22:56 |
| clarkb | will it go into exim at that point? should I stop exim first? | 22:56 |
| clarkb | or does exim need to be running for them to communicate? | 22:56 |
| corvus | lemme check | 22:57 |
| clarkb | I assume it writes to /var/spool/exim4? | 22:57 |
| clarkb | /var/spool/exim4/input perhaps | 22:57 |
| clarkb | (also I've made copies of the files as I go along | 22:57 |
| corvus | clarkb: why don't you go ahead and stop exim and try it while i continue to confirm which mtp method it uses for outgoing msgs | 23:00 |
| clarkb | corvus: ok stopping now | 23:00 |
| corvus | i *think* it's smtp to localhost, so it probably won't work, but no loss in trying. | 23:01 |
| corvus | fungi: lists.openstack.org has many graylisted emails for you | 23:02 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver https://review.openstack.org/649474 | 23:02 |
| clarkb | note you want to run the mailman processes as root. Got a lot of tracebacks before I figured that out :) | 23:02 |
| corvus | clarkb: they usually run as the 'list' user | 23:03 |
| clarkb | ah | 23:04 |
| clarkb | fwiw I think it is smtp Apr 02 23:02:32 2019 (13539) Low level smtp error: [Errno 111] Connection refused | 23:04 |
| clarkb | and my file is in the retry qfiles dir now | 23:04 |
| clarkb | and that file seems to have the base64 encoding | 23:05 |
| clarkb | /root/clarkb-scratch/one-by-one/ has all three packfiles named in, out, and retry for the stage they were copied out of | 23:05 |
| clarkb | if others want to look | 23:05 |
| clarkb | I think that rules out exim | 23:06 |
| clarkb | but rules in mailmans delivery mechanism | 23:06 |
| corvus | clarkb: what the ip again? | 23:07 |
| clarkb | corvus: 166.78.27.52 | 23:07 |
| corvus | clarkb: neat, so the outgoing runner seemed to do it? | 23:08 |
| clarkb | corvus: yes that appears to be the case | 23:08 |
| clarkb | I'm reading OutgoingRunner.py now :) | 23:09 |
| clarkb | which calls Handlers/SMTPDirect.process if I am reading this correctly | 23:11 |
| *** gmann_afk is now known as gmann | 23:11 | |
| clarkb | heh that calls decorate again independent of the decorate handler if it didn't get decorated by the pipeline | 23:12 |
| mordred | I have not had the braincells to actually understand the state of the project y'all are working on right now - but from a swooping in and out perspective, it's amazing | 23:12 |
| clarkb | mordred: long story short is that mailman on xenial is always base64 encoding our test emails | 23:13 |
| clarkb | mordred: we think this is probably undesirable for a number of reasons but are trying to narrow down what chagned between trusty and xenial becuase our prod server isn't doing this on the prod lists | 23:14 |
| *** jamesmcarthur has joined #openstack-infra | 23:14 | |
| clarkb | I'm going to stop my manually started mailman procs | 23:15 |
| mordred | clarkb: yeah - like - always base64 encoding doesn't seem like the right choice | 23:16 |
| corvus | clarkb: Decorate certainly looks like it could be relevant | 23:16 |
| openstackgerrit | Merged opendev/base-jobs master: Always pause docker build jobs https://review.openstack.org/649461 | 23:18 |
| *** lseki has quit IRC | 23:19 | |
| *** aaronsheffield has quit IRC | 23:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix error in pull-from-intermediate-registry https://review.openstack.org/649475 | 23:22 |
| corvus | clarkb, mordred, AJaeger: ^ can you review that with some urgency? i think we broke that role | 23:22 |
| clarkb | fwiw Decorate doesn't seem ot have changed between 2.1.16 and 2.1.20 | 23:22 |
| clarkb | but ya this may explain the difference since we go out of our way to not decorate -discuss | 23:22 |
| clarkb | corvus: looking | 23:23 |
| *** tosky has quit IRC | 23:24 | |
| fungi | corvus: thanks, they're almost certainly a fraction of (the many thousands of) the qq.com spam hitting the openstack-discuss-owner address | 23:24 |
| fungi | i'll see if i can whitelist it by ip address | 23:25 |
| clarkb | my message should go through the if not msg.is_multipart() and msgtype == 'text/plain': block of Decorate.process since it is not multipart and is text/plain | 23:25 |
| clarkb | payload = payload.encode(lcset) in that block should encode the payload to utf8 beacuse utf8 is set for en | 23:26 |
| clarkb | then del msg['content-transfer-encoding'] happens | 23:27 |
| clarkb | then msg.set_payload(payload, newcset) | 23:28 |
| clarkb | and supposedly the python email lib will set transfer-encoding to base64 if newcset is utf8 | 23:28 |
| clarkb | I think this is the particular lump of code where we want ot look for chagnes though | 23:28 |
| clarkb | also the multipart handling is different | 23:30 |
| clarkb | however openstack-discuss shows the same no base64 encoding for multipart and text/plain emails | 23:31 |
| clarkb | `bzr diff -r2.1.16..2.1.20 Handlers/Decorate.py` shows no delta upstream | 23:33 |
| clarkb | none of the trusty or xenial patches touch that file. I guess I can import email on both trusty and xenial and see if there is a difference in their behavior | 23:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver https://review.openstack.org/649474 | 23:38 |
| clarkb | both trusty and xenial exhibit the same base64 payload conversion when manually tested | 23:39 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix error in pull-from-intermediate-registry https://review.openstack.org/649475 | 23:39 |
| clarkb | that is making me think more and more that somehow -discuss isn't set to a utf8 default | 23:39 |
| clarkb | and the config dumping shows the pickle file is ascii encoded not utf8 | 23:40 |
| clarkb | which may be the clue towards that | 23:40 |
| fungi | nothing obvious under language options | 23:41 |
| corvus | sorry i have to eod | 23:41 |
| *** jamesmcarthur has quit IRC | 23:43 | |
| *** jamesmcarthur has joined #openstack-infra | 23:43 | |
| fungi | have a good evening | 23:44 |
| *** wolverineav has quit IRC | 23:45 | |
| clarkb | oh ya if you read config_list source you see that it gets teh default charset for that list when printing that comment header | 23:46 |
| clarkb | if not charset then charset == us-ascii | 23:46 |
| clarkb | so either it is set to us-ascii or it isn't set at all | 23:47 |
| fungi | weird, i've pored over the admin interface and am not seeing anywhere that would/could be (un)set | 23:48 |
| fungi | closest is the language panel which lets you set the default language for the list | 23:48 |
| clarkb | ya and I'm looking at hte config dump and its preferred language is the same as my test list which is both 'en' and they should look up en in the LC_DESCRIPTIONS list | 23:49 |
| clarkb | I'm going to redump the config for discuss to make sure it isn't pebkac | 23:49 |
| fungi | preferred_language is set to "English (USA)" there and for available_languages only "English (USA)" is set | 23:49 |
| clarkb | hrm now it says utf8 so maybe I did that dump when I manually overrode the en encoding to ascii? | 23:50 |
| fungi | sounds plausible | 23:50 |
| clarkb | http://paste.openstack.org/show/748754/ is my test of the email lib on trusty and xenial fwiw | 23:51 |
| clarkb | given ^ I don't think this is a regression except htat we have the evidence that says otherwise | 23:52 |
| clarkb | and also other lists seem to give us the expected behavior (like the infra list) | 23:54 |
| clarkb | which is why I sort of keep coming back to the idea that -discuss must be configured somehow to act differently | 23:54 |
| clarkb | I'm going to start putting my test server back the way I found it so that we don't get confused via further debugging acting weird | 23:55 |
| clarkb | Then I think I may need to tackle this more tomorrow | 23:55 |
| clarkb | exim has been restarted but I've left mailman-openstack off | 23:56 |
| clarkb | I commented out my changes to the openstack mailman vhost config | 23:56 |
| clarkb | To summarize what we've learned. Trusty and Xenial python2.7 email lib both have the same conversion to base64 behavior when the input encoding is utf8. Lists like openstack-infra seem to exhibit the xenial behavior of always converting to base64. Something seems to make openstack-discuss special | 23:59 |
| clarkb | The conversion happens in output processing like in Mailman/Handlers/Decorate.process() but only as part of a text/plain message. Multipart goes through as is | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!