| *** rlandy|bbl is now known as rlandy | 00:17 | |
| *** rlandy is now known as rlandy|out | 00:20 | |
| ianw | #status log Added 22.03-LTS to the openEuler mirror volume | 00:22 |
|---|---|---|
| opendevstatus | ianw: finished logging | 00:22 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "containerfile: use focal for testing" https://review.opendev.org/c/openstack/diskimage-builder/+/849274 | 00:31 |
| ianw | 2022-07-11 22:43:25.641582 | # Job Information | 00:42 |
| ianw | 2022-07-11 22:43:25.641964 | Ansible Version: 2.12.7 | 00:42 |
| ianw | certainly seems things running with ansible 5. shouldn't complain when things are smoother than expected! | 00:43 |
| fungi | don't look a gift upgrade in the mouth! | 01:01 |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: [WIP]Upgrade openEuler to 22.02 LTS https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 01:17 |
| *** dviroel|out is now known as dviroel | 01:56 | |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 01:56 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 02:02 |
| corvus | ianw: i'm going to eod now; if you feel like doing anything to https://review.opendev.org/848792 after the mergers get through the periodic job backlog you're welcome to do so | 02:12 |
| ianw | ok, will check it out | 02:13 |
| *** dviroel is now known as dviroel|out | 02:18 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: opensuse: better report checksum errors https://review.opendev.org/c/openstack/diskimage-builder/+/849445 | 02:41 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: opensuse: better report checksum errors https://review.opendev.org/c/openstack/diskimage-builder/+/849445 | 02:42 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "containerfile: use focal for testing" https://review.opendev.org/c/openstack/diskimage-builder/+/849274 | 02:55 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: opensuse: better report checksum errors https://review.opendev.org/c/openstack/diskimage-builder/+/849445 | 04:03 |
| *** ysandeep|holiday is now known as ysandeep | 04:14 | |
| *** ysandeep is now known as ysandeep|afk | 06:31 | |
| *** dmsimard0 is now known as dmsimard | 06:44 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] Build a dib reference image https://review.opendev.org/c/openstack/diskimage-builder/+/849454 | 07:02 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] Build a dib reference image https://review.opendev.org/c/openstack/diskimage-builder/+/849454 | 07:04 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: cache-url: turn off -x by default https://review.opendev.org/c/openstack/diskimage-builder/+/849456 | 07:10 |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: Upgrade openEuler to 22.02 LTS https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 07:44 |
| *** ysandeep|afk is now known as ysandeep | 07:59 | |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 08:35 | |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 08:55 | |
| frickler | infra-root: seems we have a node in status "hold" but no matching autohold, 0027999262 at the top of https://zuul.opendev.org/t/openstack/nodes | 09:44 |
| frickler | on a mostly unrelated note I will be mostly offline for the next 2.5 weeks (pto but staying home, so some chance I get bored ;-) | 09:57 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] Build a dib reference image https://review.opendev.org/c/openstack/diskimage-builder/+/849454 | 10:02 |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 10:18 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] Build a dib reference image https://review.opendev.org/c/openstack/diskimage-builder/+/849454 | 10:21 |
| *** rlandy|out is now known as rlandy | 10:27 | |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 10:40 | |
| *** dviroel|out is now known as dviroel|rover | 11:29 | |
| ianw | frickler: enjoy the break! | 11:31 |
| hrw | hi | 11:32 |
| hrw | is "Access was denied for financial reasons." known already on cloud.ovh.net nodes? | 11:32 |
| hrw | https://zuul.opendev.org/t/openstack/build/1c2c236358264556bb0a1fec61242ee0/console build lacks any logs and gives that answer on 'view log' | 11:33 |
| fungi | weird... | 11:37 |
| fungi | and no, first i've heard of it. we probably need to remind them to renew our account | 11:38 |
| hrw | ok | 11:41 |
| rlandy | hello ... there are a bunch of post_failures going on right now ... https://zuul.openstack.org/builds?result=POST_FAILURE&skip=0 | 11:42 |
| rlandy | no log available to check | 11:42 |
| rlandy | possibly centos-8 related | 11:43 |
| rlandy | so hard to tell what is going on - no logs we can see | 11:43 |
| rlandy | dviroel|rover: soniya29: ^^ | 11:44 |
| fungi | rlandy: hrw noticed that we've suddenly started getting errors from ovh's swift about the account needing payment, that may be the cause. i'm looking into it now | 11:44 |
| rlandy | thank you | 11:45 |
| fungi | i've confirmed we didn't receive any notification from them anyway | 11:45 |
| fungi | i'm going to temporarily remove them from our log uploads to contain the damage as much as possible while i start trying to contact people | 11:45 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Temporarily disable log uploads to OVH https://review.opendev.org/c/opendev/base-jobs/+/849499 | 11:48 |
| fungi | infra-root: i'm bypassing zuul to merge that ^ so it takes effect asap | 11:49 |
| opendevreview | Merged opendev/base-jobs master: Temporarily disable log uploads to OVH https://review.opendev.org/c/opendev/base-jobs/+/849499 | 11:50 |
| fungi | #status log Temporarily disabled log uploads to OVH's Swift while we look into an account authorization problem | 11:51 |
| opendevstatus | fungi: finished logging | 11:51 |
| fungi | amorin: do you happen to know how/who is approving our cloud account these days? i'm wondering if our gratis access expired again and needs to be renewed... | 11:52 |
| fungi | i'll poke around in the dashboard and see if i can spot any notifications which didn't go via e-mail or something | 11:53 |
| *** ysandeep is now known as ysandeep|break | 11:59 | |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: DNM: Run tests with ansible 5 https://review.opendev.org/c/zuul/zuul-jobs/+/849502 | 12:06 |
| fungi | unfortunately i can't get into our dashboard account at the moment because it wants to send an e-mail confirmation for the login and the confirmation isn't arriving in our inbox. maybe it's greylisted, or maybe our mail provider has started blocking messages from ovh (which could explain the lack of notifications too) | 12:10 |
| fungi | okay, found it | 12:11 |
| jrosser | i am seeing a rash of POST FAILURE again :( https://zuul.openstack.org/status#openstack-ansible | 12:12 |
| fungi | jrosser: see above | 12:12 |
| * jrosser reads | 12:13 | |
| fungi | probably not ansible-related this time | 12:13 |
| jrosser | ah phew | 12:13 |
| fungi | i found the account expiration notifications too. seems someone had set up a rule to filter notifications from ovh into a separate folder rather than into the inbox | 12:13 |
| fungi | we've been getting stuff from addresses @services.ovh.com into our normal inbox, so i didn't realize there was a rule filtering @ovh.com addresses into a separate place *sigh* | 12:15 |
| fungi | okay, i can see in the dashboard our invoices were all 0€ until a bill a little over a week ago which shows a nonzero amount, and then we got a notification at 11:04 utc (a little over an hour ago) that our account was suspended for nonpayment | 12:25 |
| frickler | fungi: seems we also should disable nodepool things? https://grafana.opendev.org/d/2b4dba9e25/nodepool-ovh?orgId=1 | 12:29 |
| fungi | frickler: we can, but if we're unable to boot nodes there then at least jobs aren't running there for now | 12:31 |
| fungi | i've filed trouble ticket 5886766 with ovh | 12:31 |
| *** ysandeep|break is now known as ysandeep | 12:46 | |
| *** dasm|off is now known as dasm | 13:02 | |
| amorin | Hey fungi, i can double check that internally, can you share the nic or project id? | 13:17 |
| fungi | amorin: jo13881-ovh | 13:35 |
| fungi | thanks! | 13:35 |
| *** dhill is now known as Guest4834 | 14:23 | |
| amorin | fungi refund in progress | 14:28 |
| fungi | amorin: thanks! did we forget to update something? | 14:28 |
| amorin | nop, the voucher expired :( | 14:29 |
| amorin | it's one year old I think | 14:29 |
| fungi | i think we got notifications before prior voucher expirations, but it's possible i missed it | 14:29 |
| fungi | though i don't recall whether we did something to renew it last year either | 14:30 |
| amorin | I think we end-up in the same situation last year :( | 14:33 |
| fungi | ahh, okay | 14:34 |
| amorin | the best would be to catch the mails with invoices | 14:34 |
| fungi | what's the process for renewing the voucher? | 14:34 |
| amorin | before the suspension :) | 14:34 |
| amorin | ask me :) | 14:34 |
| amorin | I can create vouchers each year | 14:34 |
| fungi | and yeah, we get notifications of two invoices each month, i'm not sure why there's two. they normally say 0€ but this time one of them (and not the other) had a non-zero amount | 14:35 |
| amorin | yes, so as soon there is a non zero amount, feel free to ping me | 14:35 |
| amorin | I can cancel the billing and edit a new voucher | 14:36 |
| amorin | it would prevent the services to be suspended :) | 14:36 |
| fungi | thanks! i'll rearrange our mail filtering to try and make the invoice statements easier for us to spot | 14:36 |
| amorin | but anyway, no worries | 14:36 |
| fungi | once service seems to be restored, i'll push up a change to re-add log uploading there | 14:37 |
| hrw | ;) | 14:38 |
| fungi | infra-root: i've adjusted our mail sorting so that all messages from ovh land in the main inbox now. please remember to look at every invoice notification we receive, since if it's for >0€ we have 10 days to get the voucher renewed before service suspension kicks in | 14:43 |
| amorin | fungi service is restored | 14:43 |
| fungi | thanks amorin! any idea when urls like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e4/849419/5/check/tox-py36/0e426b5/ will start returning content again instead of saying "payment required"? | 14:44 |
| amorin | checking | 14:48 |
| amorin | it's good now | 14:52 |
| fungi | yep, lgtm. much appreciated! | 14:52 |
| amorin | small delay to propagate the unsuspend | 14:52 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Revert "Temporarily disable log uploads to OVH" https://review.opendev.org/c/opendev/base-jobs/+/849546 | 14:52 |
| fungi | infra-root: i'll self-approve that ^ now | 14:52 |
| *** soniya29 is now known as soniya29|out | 14:58 | |
| opendevreview | Merged opendev/base-jobs master: Revert "Temporarily disable log uploads to OVH" https://review.opendev.org/c/opendev/base-jobs/+/849546 | 14:59 |
| fungi | #status log Log uploads to OVH's Swift are resuming and our voucher is renewed; thanks again amorin! | 15:02 |
| opendevstatus | fungi: finished logging | 15:02 |
| *** ysandeep is now known as ysandeep|out | 15:05 | |
| *** marios is now known as marios|out | 15:35 | |
| *** dviroel|rover is now known as dviroel|rover|biab | 15:36 | |
| *** frickler is now known as frickler_pto | 16:00 | |
| opendevreview | Merged openstack/diskimage-builder master: ubuntu: more exact match on squashfs file, containerfile: use focal https://review.opendev.org/c/openstack/diskimage-builder/+/849269 | 16:29 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 16:41 |
| *** dviroel|rover|biab is now known as dviroel|rover | 16:42 | |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 16:56 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 17:36 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 17:45 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 17:57 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 18:13 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Add a new openinfra/way project https://review.opendev.org/c/openstack/project-config/+/849576 | 18:50 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Add openinfra/way to Zuul https://review.opendev.org/c/openstack/project-config/+/849577 | 18:50 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 18:59 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 19:55 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 21:58 |
| *** rlandy is now known as rlandy|bbl | 21:59 | |
| ianw | fungi: do the three forces relate to the conjoined triangles of success? | 22:03 |
| fungi | ianw: one can only hope | 22:12 |
| fungi | i dunno, it's the virtuous cycle of developers->users->ecosystem->lather->rinse->repeat-> | 22:13 |
| fungi | on an unrelated note, did you see the new prompt from pypi? | 22:13 |
| *** dasm is now known as dasm|off | 22:14 | |
| fungi | it's like a text adventure... "get 2fa" "ok. you stand before an impressive api token generator" | 22:15 |
| ianw | ahh; i thought it was weird i couldn't see an api token for pypi | 22:15 |
| ianw | it seems both simple in theory and annoying to test in production circumstances | 22:16 |
| fungi | there's a "generate api token" button in the account settings, if memory serves | 22:16 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 22:17 |
| fungi | but yeah, if we enable 2fa for web logins, in the future we'll be required to use an api token instead of username/password for package uploads | 22:17 |
| fungi | which is why i was hesitant to do that just yet | 22:17 |
| fungi | if we hadn't added 2fa, we'd be able to continue using username+password for uploads for the foreseeable future | 22:18 |
| fungi | granted, there's been no indication of 1. when 2fa would become required for webui logins of "critical" package maintainers, and 2. when 2fa using accounts would be required to switch to api tokens for uploads | 22:19 |
| ianw | i have to admit i thought it was using token upload via some other method. it's probably worth updating; i'll take a look and propose something | 22:20 |
| fungi | so for all i know we still have months to figure out the transition | 22:20 |
| TheJulia | question unrelated to ci, do we know if outbound email from lists.opendev.org is in a working state? | 22:22 |
| fungi | i've been receiving messages from it, but i'll check the delivery logs to see if there are a lot of rejections | 22:24 |
| TheJulia | fungi: much appreciated | 22:24 |
| fungi | for some reason, spamhaus has decided that rbl exclusions need to be re-filed yearly now | 22:24 |
| ianw | there's a lot of routing defer (-51): retry time not reached | 22:24 |
| fungi | sending _to_ lists.o.o? | 22:25 |
| fungi | oh, from | 22:25 |
| ianw | also a fairly constant "Failed to get read lock for /var/spool/exim4/db/retry.lockfile" but that's not new | 22:28 |
| fungi | TheJulia: can you be specific about addresses which haven't received something? privmsg me if you prefer | 22:28 |
| fungi | i checked red hat addresses just on a whim and see deliveries logged | 22:28 |
| TheJulia | fungi: I sent a mailing list post to the openinfralabs list, and spotz has not receieved it | 22:29 |
| fungi | the one on 2022-07-08 or the one today (for the record, i received both) | 22:30 |
| TheJulia | fungi: in her case, both, but good to know at least somebody has received the posts :) | 22:31 |
| TheJulia | I almost posted "is this thing on?" | 22:31 |
| fungi | i was waiting for other folks to chime in on the first one from friday, since i planned to try to join the discussion whenever it's held | 22:32 |
| fungi | but maybe i'll reply anyway to make it seem less... lonely | 22:32 |
| TheJulia | less lonely is always good :) | 22:33 |
| TheJulia | otherwise it will be me, some crickets, and plans to take over the world. | 22:33 |
| fungi | it looks like spotz may be using a non-redhat gmail hosted domain address to subscribe there, and should probably check junk mail (the mta log says google accepted the copy to her address) | 22:34 |
| fungi | 2022-07-12 22:08:10 [...] C="250 2.0.0 OK h4-20020a056870170400b0010c07e3e229si12513924oae.117 - gsmtp" | 22:35 |
| fungi | that was the entry for delivery to her subscriber addy | 22:36 |
| fungi | time is in utc | 22:36 |
| TheJulia | le sigh | 22:37 |
| TheJulia | yup, all in the spam folder | 22:38 |
| fungi | these days, gmail likes to flag anything sent from not-gmail as spam, because nobody using gmail would ever send spam (but also they want everyone to use gmail) | 22:38 |
| fungi | apropos of nothing in particular, i find it amusing/sad that red hat sells a platform and support for managing enterprise-grade e-mail systems, but chooses to host their own e-mail at google instead | 22:39 |
| TheJulia | w/r/t spam, yeah. :( I've run into it a few times recently and it is getting annoying. | 22:40 |
| fungi | there is no e-mail, only gmail | 22:40 |
| * JayF had an entire rant typed up about how all the big ESPs collude using spam folder as the cudgel, but deleted it | 22:43 | |
| fungi | are you sure you didn't just mark it as spam? ;) | 22:44 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 22:54 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 23:42 |
| *** rlandy|bbl is now known as rlandy|out | 23:47 | |
| *** ysandeep|out is now known as ysandeep | 23:58 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!