| stewie925 | clarkb: I hope you understand my issue now hehe. hope Ubuntu One can help fix that issue | 00:00 |
|---|---|---|
| *** tosky has quit IRC | 00:00 | |
| clarkb | stewie925: no, that hasn't been something they would typically fix | 00:00 |
| clarkb | I think if you set the ubuntu one email back to the original email address it may start using the old openid again though (its a consistent hash) | 00:00 |
| clarkb | and then you can log in to the old account but not the new (hwoever I'm not certain of that) | 00:00 |
| clarkb | prior to the big gerrit upgrade we would address these issues with db updates | 00:01 |
| stewie925 | clarkb: you are right - I cannot log in with the new email address - I can only log in with old email address | 00:01 |
| clarkb | but now the accounts are not in the sql db and are instead in the notedb. And notedb won't allow us to push fixes until we fix all the other issues which is quite annoying | 00:01 |
| clarkb | its possible we can remove conflicting ids from one account or another to alleviate the issue though if you just wanted to move forward with a new account and the new emails (I suspect you cannot log in with the new email because it conflicts with the old account) | 00:02 |
| stewie925 | clarkb: no rush, I can go along with this for now... :) | 00:02 |
| clarkb | stewie925: I'm not sure what timezone you are in. Its getting close to when I need to start sorting out dinner and family time. However, if you are around tomorrow or friday with some overlap on pacific time I'm happy to dig into this with you more directly | 00:03 |
| clarkb | stewie925: and you can share some of the more private info like account details and I can see what the gerrit backend looks like | 00:03 |
| clarkb | and from that hopefully we can figure out a plan | 00:03 |
| stewie925 | clarkb: thank you so much.... I m in Pacific time. | 00:03 |
| stewie925 | clarkb: what is the best time to discuss with you? maybe 3 PM PST Friday ? | 00:04 |
| clarkb | that works for me, I can do just about anytime between 8am and 3pm though I think | 00:04 |
| stewie925 | you're Pacific too, I guess? | 00:05 |
| clarkb | if you find time tomorrow ping me and I'll see what I'm at with zuul executor replacments (I should be able to page that out easily enough) | 00:05 |
| clarkb | ya | 00:05 |
| stewie925 | oh thank you clarkb - I'll catch up with you tomorrow (around 3 pm) then:) . you have a great night. | 00:06 |
| clarkb | you too | 00:06 |
| ianw | clarkb / corvus : i'm starting to think opendev is the right place to do it ... https://review.opendev.org/c/opendev/project-config/+/777485 | 00:08 |
| clarkb | ianw: but if the job is running in the openstack tenant wouldn't ^ be ignored? | 00:09 |
| ianw | clarkb: i'm really not sure with this one :/ | 00:11 |
| corvus | ianw: i'm going off of https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L64 | 00:14 |
| clarkb | corvus: ianw oh ya in that case I agree opendev is correct here, but I'm not sure why we need it in that tenant now | 00:14 |
| clarkb | I did leave a note on the above change though | 00:16 |
| ianw | clarkb: i don't think i've used "depends-on" with that project before. so it's only been cloning as part of "required-projects" for the system-config jobs, where it has the override to "main" | 00:16 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/777298 tried it, and all the other jobs now fail with the wrong branch; so that's the gist of why it needs to be somewhere | 00:17 |
| clarkb | ianw: I woudl expect we need to add that project to openstack and set its default branch for that to work though since system-config depends on shouldn't evaluate the opendev config but maybe things aren't as separated as I expect | 00:18 |
| ianw | i think i should remove the override-checkout in https://opendev.org/opendev/system-config/src/branch/master/zuul.d/docker-images/gerrit.yaml#L180 too | 00:19 |
| ianw | clarkb: maybe both? | 00:19 |
| clarkb | ya we may want it in both places | 00:21 |
| *** stewie925 has quit IRC | 00:24 | |
| ianw | clarkb: not sure i folow your comment, there's no projects in opendev tenant that don't start with opendev/* ? | 00:26 |
| clarkb | ianw: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L12-L13 those ones for example | 00:27 |
| clarkb | but that list of jobs is also empty so it doesn't really matter, in that case I think we should simply consider dropping the regex match and template entirely though (to avoid confusion) | 00:27 |
| ianw | hrm, ok. alright, let's drop it | 00:27 |
| openstackgerrit | Ian Wienand proposed opendev/project-config master: zuul-results-summary: set default branch https://review.opendev.org/c/opendev/project-config/+/777485 | 00:29 |
| clarkb | I've +2'd it but may want fungi and corvus to weigh in on the removal of the system default template before approving just to make sure I'm not missing something imortant | 00:34 |
| corvus | why drop it? | 00:39 |
| fungi | i think the idea was we'd add required jobs into that template, right? | 00:40 |
| corvus | -1 on change; clarkb, ianw, fungi: pls see comment in project-template | 00:40 |
| ianw | corvus: ok, but then we want to *exclude* the zuul-plugin right, because it's not an opendev project? | 00:41 |
| clarkb | fungi: I think it was that we had them at one point | 00:44 |
| * clarkb finds the comment | 00:45 | |
| clarkb | ah its a zuulism thing | 00:45 |
| clarkb | ya I think we want a way to exclude the project from pipeline config but also set its branch config | 00:45 |
| openstackgerrit | Ian Wienand proposed opendev/project-config master: zuul-results-summary: set default branch https://review.opendev.org/c/opendev/project-config/+/777485 | 00:46 |
| corvus | ianw: that would be most correct, yes; otherwise it may report merge failures. | 00:46 |
| ianw | corvus : ^^ i think *that* might do it ... | 00:46 |
| corvus | to be fair, the config is already "wrong", you just happened to have discovered it while changing this setting. the 2 things aren't really related. | 00:47 |
| openstackgerrit | Merged opendev/project-config master: zuul-results-summary: set default branch https://review.opendev.org/c/opendev/project-config/+/777485 | 00:53 |
| *** hamalq has quit IRC | 00:56 | |
| ianw | ... system-config-run-base-ansible-devel error | 01:01 |
| ianw | it seems ^^ maybe isn't sufficient :/ | 01:01 |
| ianw | corvus: so the results of https://review.opendev.org/c/opendev/system-config/+/777298 seem to show the same, thing, do we need https://review.opendev.org/c/openstack/project-config/+/777480 as well? | 01:07 |
| corvus | ianw: yes; looks like it's in both tenants via a yaml anchor | 01:09 |
| openstackgerrit | Merged opendev/system-config master: install-docker: move rsyslog handler earlier https://review.opendev.org/c/opendev/system-config/+/777476 | 01:12 |
| openstackgerrit | Merged openstack/project-config master: zuul-results-summary: set default branch https://review.opendev.org/c/openstack/project-config/+/777480 | 01:27 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: zuul-summary-status : handle SKIPPED jobs https://review.opendev.org/c/opendev/system-config/+/777298 | 01:28 |
| ianw | yay, jobs seems to be running, thanks for the help :) | 01:36 |
| *** mlavalle has quit IRC | 02:13 | |
| *** LowKey has quit IRC | 02:20 | |
| *** auristor has quit IRC | 02:29 | |
| *** auristor has joined #opendev | 02:43 | |
| *** ysandeep|away is now known as ysandeep | 03:06 | |
| *** ysandeep is now known as ysandeep|bbl | 03:17 | |
| *** sgw has quit IRC | 04:02 | |
| *** sgw has joined #opendev | 04:02 | |
| *** whoami-rajat has joined #opendev | 04:18 | |
| *** dviroel has quit IRC | 04:45 | |
| *** ykarel has joined #opendev | 04:55 | |
| *** bhagyashri|ruck is now known as bhagyashri|rover | 05:15 | |
| *** marios has joined #opendev | 05:53 | |
| *** ralonsoh has joined #opendev | 06:01 | |
| openstackgerrit | Merged opendev/system-config master: install-docker: remove fix from prior change https://review.opendev.org/c/opendev/system-config/+/777477 | 06:03 |
| *** yoctozepto0 has joined #opendev | 06:04 | |
| *** yoctozepto has quit IRC | 06:05 | |
| *** yoctozepto0 is now known as yoctozepto | 06:05 | |
| *** mkowalski_ has joined #opendev | 06:35 | |
| *** ysandeep|bbl is now known as ysandeep | 06:41 | |
| *** zbr has quit IRC | 06:43 | |
| *** icey has quit IRC | 06:58 | |
| *** icey has joined #opendev | 06:58 | |
| *** icey has quit IRC | 07:01 | |
| *** icey has joined #opendev | 07:05 | |
| *** icey has quit IRC | 07:08 | |
| *** slaweq has joined #opendev | 07:12 | |
| *** lpetrut has joined #opendev | 07:19 | |
| *** slaweq has quit IRC | 07:19 | |
| *** slaweq has joined #opendev | 07:23 | |
| *** eolivare has joined #opendev | 07:40 | |
| *** jpena|off is now known as jpena | 07:45 | |
| *** rpittau|afk is now known as rpittau | 07:57 | |
| *** fressi has joined #opendev | 08:05 | |
| *** brinzhang has quit IRC | 08:05 | |
| *** brinzhang has joined #opendev | 08:07 | |
| *** brinzhang has quit IRC | 08:07 | |
| *** brinzhang has joined #opendev | 08:07 | |
| *** brinzhang has quit IRC | 08:08 | |
| *** brinzhang has joined #opendev | 08:09 | |
| *** andrewbonney has joined #opendev | 08:13 | |
| *** ykarel_ has joined #opendev | 08:31 | |
| *** ykarel has quit IRC | 08:33 | |
| *** ykarel_ is now known as ykarel | 08:34 | |
| *** toomer has joined #opendev | 08:37 | |
| *** tosky has joined #opendev | 08:38 | |
| *** jaicaa has quit IRC | 08:44 | |
| *** jaicaa has joined #opendev | 08:47 | |
| openstackgerrit | Lianhao Lu proposed openstack/project-config master: intel-nfv-ci-tests: move zuul definition into repo https://review.opendev.org/c/openstack/project-config/+/777511 | 08:48 |
| *** icey has joined #opendev | 08:57 | |
| *** lpetrut_ has joined #opendev | 09:54 | |
| *** yoctozepto9 has joined #opendev | 09:55 | |
| *** jhesketh_ has joined #opendev | 09:57 | |
| *** avass_ has joined #opendev | 09:58 | |
| *** dhellmann_ has joined #opendev | 09:58 | |
| *** SWAT_ has joined #opendev | 09:59 | |
| *** Dmitrii-Sh7 has joined #opendev | 09:59 | |
| *** arxcruz has quit IRC | 10:00 | |
| *** arxcruz has joined #opendev | 10:01 | |
| *** lpetrut has quit IRC | 10:03 | |
| *** yoctozepto has quit IRC | 10:03 | |
| *** dhellmann has quit IRC | 10:03 | |
| *** jhesketh has quit IRC | 10:03 | |
| *** Dmitrii-Sh has quit IRC | 10:03 | |
| *** SWAT has quit IRC | 10:03 | |
| *** avass has quit IRC | 10:03 | |
| *** Dmitrii-Sh7 is now known as Dmitrii-Sh | 10:03 | |
| *** yoctozepto9 is now known as yoctozepto | 10:03 | |
| *** dhellmann_ is now known as dhellmann | 10:03 | |
| frickler | infra-root: looking at disk usage on bridge, it looks to me like a lot is actually being used by /root/{.ansible,.cache}, which sounds like it could be easily cleaned up? (14G according to du) | 10:06 |
| frickler | in particular there are almost 500k files sized 14k in /root/.cache/python-entrypoints taking up 7.4G in total | 10:10 |
| ianw | ++ sounds like a leak | 10:29 |
| ianw | probably a cron job to mtime delete is a good idea? | 10:29 |
| *** toomer has quit IRC | 10:54 | |
| *** dtantsur|afk is now known as dtantsur | 10:56 | |
| *** dviroel has joined #opendev | 11:03 | |
| *** sshnaidm|afk is now known as sshnaidm|pto | 11:22 | |
| mordred | wow, what even are /root/.cache/python-entrypoints ? | 11:58 |
| *** jpena is now known as jpena|lunch | 12:27 | |
| *** ysandeep is now known as ysandeep|afk | 12:29 | |
| *** zoharm has joined #opendev | 12:31 | |
| *** ysandeep|afk is now known as ysandeep | 13:00 | |
| *** roman_g has joined #opendev | 13:01 | |
| *** zbr has joined #opendev | 13:13 | |
| *** lpetrut__ has joined #opendev | 13:14 | |
| *** lpetrut_ has quit IRC | 13:16 | |
| ttx | it does sound scary | 13:20 |
| *** jpena|lunch is now known as jpena | 13:26 | |
| *** ykarel has quit IRC | 13:32 | |
| *** tkajinam has quit IRC | 13:36 | |
| *** ykarel has joined #opendev | 13:37 | |
| *** iurygregory has quit IRC | 13:43 | |
| *** iurygregory has joined #opendev | 13:44 | |
| *** zbr7 has joined #opendev | 13:47 | |
| *** zbr has quit IRC | 13:49 | |
| *** zbr has joined #opendev | 13:51 | |
| *** zbr7 has quit IRC | 13:53 | |
| *** yoctozepto has quit IRC | 14:01 | |
| *** yoctozepto has joined #opendev | 14:01 | |
| *** zbr0 has joined #opendev | 14:11 | |
| *** zbr has quit IRC | 14:13 | |
| *** zbr0 is now known as zbr | 14:13 | |
| *** whoami-rajat has quit IRC | 14:38 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Correct OpenStack Security URL in sites https://review.opendev.org/c/opendev/system-config/+/777602 | 14:39 |
| *** zbr1 has joined #opendev | 14:54 | |
| *** avass_ is now known as avass | 14:56 | |
| *** zbr has quit IRC | 14:56 | |
| *** zbr1 is now known as zbr | 14:56 | |
| *** fressi has quit IRC | 15:03 | |
| *** sgw has left #opendev | 15:12 | |
| *** marios has quit IRC | 15:33 | |
| openstackgerrit | Mateusz Kowalski proposed openstack/diskimage-builder master: Change paths for bootloader files in iso element https://review.opendev.org/c/openstack/diskimage-builder/+/777606 | 15:37 |
| *** ykarel has quit IRC | 15:38 | |
| *** ysandeep is now known as ysandeep|dinner | 15:41 | |
| *** zbr3 has joined #opendev | 15:50 | |
| *** lpetrut__ has quit IRC | 15:51 | |
| clarkb | frickler: oh nice catch, I had missed that dir (I think because I was using du -hs * to find things and * won't match .) | 15:51 |
| *** zbr has quit IRC | 15:52 | |
| *** zbr3 is now known as zbr | 15:52 | |
| *** marios has joined #opendev | 15:53 | |
| clarkb | I suspect the entrypoints cache is there to speed up how slow loading entrypoints on process startup is with python? | 15:53 |
| clarkb | we can probably trim it back though and then let it repopulate what it actually needs currently | 15:53 |
| fungi | should be able to entirely delete it, and yeah then work out why it keeps growing | 15:58 |
| clarkb | I'm going to find breakfast but then will start on booting a ze01.opendev.org focal server | 15:59 |
| clarkb | if you know of anything different about those other than /var/lib/zuul being the ephemeral drive mount let me know | 15:59 |
| fungi | yeah, and i guess we did that just for the added space? | 16:00 |
| clarkb | yup for the job working dirs | 16:01 |
| clarkb | s/job/build/ | 16:01 |
| clarkb | I looked over an existing server a bit yesterday to see if there was anything else to consider but I think that is it. On the testing side ensuring afs is happy will be important but that will all be after we finish deploying it | 16:01 |
| *** ysandeep|dinner is now known as ysandeep | 16:05 | |
| *** zbr9 has joined #opendev | 16:13 | |
| *** zbr has quit IRC | 16:15 | |
| *** zbr9 is now known as zbr | 16:15 | |
| *** ysandeep is now known as ysandeep|PTO | 16:16 | |
| clarkb | infra-root system-config/launch/make_swap.sh seems to hardcode /opt as the xvde mount path. Do you know how we were convincing it to mount at /var/lib/zuul instead? | 16:23 |
| clarkb | I can manually switch it over to /var/lib/zuul then resync the contents back over to /opt on the root device | 16:23 |
| fungi | i don't, though i wouldn't be surprised if that was just done manually after the systems were booted | 16:24 |
| clarkb | ya, I'm beginning to suspect that may have been what was done | 16:24 |
| fungi | i.e. edit fstab, reboot | 16:24 |
| *** mlavalle has joined #opendev | 16:25 | |
| clarkb | make_swap.sh copies /opt contents onto the new device and then I thnk config management may write some things there too. So probably need to resync it back again | 16:25 |
| clarkb | but ya that should work | 16:25 |
| fungi | i think we only put stateless stuff in /opt which will get recreated automatically by configuration management when it runs, right? | 16:25 |
| fungi | though i suppose for some older services (not zuul) we do software deployment inside /opt instead of /srv | 16:26 |
| fungi | so as a general rule, deleting the contents of /opt on some servers could leave services offline/broken until configuration management reran | 16:27 |
| clarkb | another option may be to run launch node out of my homedir with a modified make_swap that replaces /opt/ with /var/lib/zuul | 16:27 |
| clarkb | I might start there since automating it that way will simplify the other 11 boots we need to do | 16:28 |
| clarkb | though it isn't always clear to me when it is safe to run ansible out of your homedir vs the zuul copy now | 16:30 |
| clarkb | I think for launch node it should be fine since it is fairly detached from everything else | 16:30 |
| clarkb | that is running now, we'll see how it goes | 16:38 |
| *** zbr3 has joined #opendev | 16:39 | |
| *** zbr has quit IRC | 16:41 | |
| *** zbr3 is now known as zbr | 16:41 | |
| clarkb | I think that worked actually /me makes dns and inventory updates | 16:49 |
| *** jpena is now known as jpena|off | 16:54 | |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add ze01.opendev.org to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/777625 | 16:58 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add ze01.opendev.org https://review.opendev.org/c/opendev/system-config/+/777626 | 16:58 |
| mnasiadka | Good afternoon | 16:58 |
| mnasiadka | Is there any way to debug DISK_FULL jobs? | 16:58 |
| clarkb | infra-root ^ I suspect both of those changes are ready to go if they look correct to you | 16:59 |
| clarkb | mnasiadka: can you link to an example? | 16:59 |
| mnasiadka | clarkb: https://zuul.opendev.org/t/openstack/build/59eae5cf0c1740faba7e59a123fc67ab | 16:59 |
| mordred | clarkb: the biggest issue with running in your home dir is the host_vars in system-config | 16:59 |
| mnasiadka | never hit DISK_FULL on that job, so that's why I'm curious :) | 16:59 |
| clarkb | mnasiadka: generally though the disk available to our test jobs isn't infinite and you need to operate within the bounds of the system | 16:59 |
| mordred | clarkb: (I don't think you'll have an issue with this) | 16:59 |
| clarkb | mordred: ya I made sure the latest commit was checked out and it seemed to do the right things (but also launch is suepr basic ansible) | 17:00 |
| fungi | mnasiadka: clarkb: the DISK_FULL result generally means the allowed workspace on the executor though, so has more to do with trying to collect too much log or artifact data from a build | 17:00 |
| mordred | yah | 17:00 |
| mnasiadka | fungi: Ah, now that makes more sense | 17:00 |
| clarkb | ah yup I see it is zuul reporting that message not smething from the remote ansible | 17:01 |
| mnasiadka | fungi: Any idea what's the limit? | 17:01 |
| openstackgerrit | Merged opendev/irc-meetings master: Remove usused I18n meeting slot https://review.opendev.org/c/opendev/irc-meetings/+/777185 | 17:01 |
| *** roman_g has quit IRC | 17:01 | |
| *** marios is now known as marios|out | 17:02 | |
| fungi | mnasiadka: i think we're relying on a zuul default for that, looking now to confirm | 17:02 |
| *** klonn has joined #opendev | 17:04 | |
| *** klonn has quit IRC | 17:04 | |
| mnasiadka | fungi: well, I'm fetching logs from a similar job, just based on centos, and I see we have 6402 files in the attached logs, so probably ara-html report is the one that's doubling the space used... | 17:06 |
| clarkb | mnasiadka: fungi https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zuul/templates/zuul.conf.j2#L57 | 17:11 |
| clarkb | that is MB | 17:12 |
| mnasiadka | 4.8 gig, we shouldn't produce that amount of logs :) | 17:12 |
| fungi | i'm finding which executor that build ran from to get some details | 17:13 |
| mnasiadka | I'll download those 6402 files and check what takes that amount of storage. | 17:13 |
| fungi | looks like ze07 | 17:13 |
| mnasiadka | (but it takes some time) | 17:13 |
| fungi | 2021-02-25 15:02:23,234 WARNING zuul.ExecutorDiskAccountant: /var/lib/zuul/builds/59eae5cf0c1740faba7e59a123fc67ab is using 6319MB (limit=5000) | 17:14 |
| fungi | :q | 17:16 |
| fungi | heh, you're not my vi window | 17:16 |
| *** zbr7 has joined #opendev | 17:17 | |
| mnasiadka | 6319, nice | 17:17 |
| mnasiadka | fungi: thanks for clarification | 17:17 |
| fungi | at one point i thought we included a du -s of the top-level subdirectories under those circumstances, but i'm not finding it | 17:18 |
| clarkb | fungi: do a remote df when jobs fail (as a way to detect running out of disk on the remote side | 17:19 |
| clarkb | but not sure about any printout of the build working dir on the executor | 17:19 |
| *** zbr has quit IRC | 17:19 | |
| *** zbr7 is now known as zbr | 17:19 | |
| clarkb | *we do | 17:22 |
| clarkb | fungi: I +2'd https://review.opendev.org/c/opendev/system-config/+/777602 probably safe to single core approve that one? | 17:28 |
| fungi | yeah | 17:28 |
| fungi | it was the upshot of a post to the openstack-discuss ml where a user was looking for the security site | 17:28 |
| fungi | i'm also working with the foundation webdevs to get some links to that restored | 17:29 |
| clarkb | ok I approved it too now | 17:30 |
| fungi | thanks! | 17:30 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 17:33 |
| *** rpittau is now known as rpittau|afk | 17:34 | |
| *** zoharm has quit IRC | 17:35 | |
| *** zbr3 has joined #opendev | 17:38 | |
| *** dtantsur is now known as dtantsur|afk | 17:39 | |
| openstackgerrit | Mateusz Kowalski proposed openstack/diskimage-builder master: Change paths for bootloader files in iso element https://review.opendev.org/c/openstack/diskimage-builder/+/777606 | 17:39 |
| *** zbr has quit IRC | 17:40 | |
| *** zbr3 is now known as zbr | 17:40 | |
| *** zbr5 has joined #opendev | 17:42 | |
| *** zbr has quit IRC | 17:44 | |
| *** zbr5 is now known as zbr | 17:44 | |
| openstackgerrit | Merged opendev/system-config master: Correct OpenStack Security URL in sites https://review.opendev.org/c/opendev/system-config/+/777602 | 17:55 |
| *** eolivare has quit IRC | 18:05 | |
| mnasiadka | fungi: downloaded logs, there's like 700MB of them - sure that's a lot - but what's taking the rest on the executor? | 18:06 |
| clarkb | ansible itself runs out of there | 18:07 |
| clarkb | could it be ansible stashing things in the build workdir that aren't logs (caches or something?) | 18:07 |
| clarkb | iurygregory: looking at https://review.opendev.org/c/openstack/project-config/+/775244/ is this something that hashtags might solve better? | 18:15 |
| clarkb | iurygregory: you could add a hashtag for BackportCandidate and one for DoNotBackport ? | 18:16 |
| fungi | mnasiadka: it's also possible the failed build failed in such a way that it generated a runaway bunch of log content | 18:16 |
| clarkb | I know other projects aren't doing it this way but ya'll are testing the hashtags and I wonder if this would be a good use casefor them | 18:16 |
| mnasiadka | fungi: makes sense | 18:17 |
| fungi | we've seen this before, for example with mysql slow query logs | 18:18 |
| fungi | but basically something breaks in such a way that one or more services go into a rapid loop logging the same set of error messages over and over, possibly until the job times out | 18:18 |
| fungi | zuul's disk accountant is really just a backstop to prevent a build from using too much disk on the executor, and stops the build cold if it exceeds that threshold | 18:20 |
| mnasiadka | well, let me try to trim the log size a bit and let's see if that helps | 18:20 |
| fungi | so unfortunately we don't get any more detail than the amount of disk it was using at the time it was aborted | 18:20 |
| clarkb | fungi: zuul is happy with https://review.opendev.org/c/opendev/zone-opendev.org/+/777625 and https://review.opendev.org/c/opendev/system-config/+/777626 if you have time to look those over and add ze01.opendev.org | 18:29 |
| openstackgerrit | Merged openstack/project-config master: Create inspur/venus-dashboard repository https://review.opendev.org/c/openstack/project-config/+/776864 | 18:32 |
| *** marios|out has quit IRC | 18:36 | |
| fungi | clarkb: approved both, am around if help troubleshooting is needed too | 18:39 |
| clarkb | thanks! I'll keep an eye out as well | 18:39 |
| clarkb | I don't think the executor will auto start on the new host anyway so we can check it first, then start the service, then stop the service on ze01.openstack.org | 18:40 |
| fungi | that's what i figured, yeah | 18:41 |
| *** andrewbonney has quit IRC | 18:49 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add ze01.opendev.org to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/777625 | 18:50 |
| openstackgerrit | Merged opendev/system-config master: Add ze01.opendev.org https://review.opendev.org/c/opendev/system-config/+/777626 | 19:05 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add install_args role var https://review.opendev.org/c/zuul/zuul-jobs/+/777653 | 19:17 |
| *** ralonsoh has quit IRC | 19:24 | |
| iurygregory | clarkb, hey! sorry for the delay, so the idea is: with the label for Backport-Candidate we will try to have something using gerrit API to check if the label is present and will add the hashtag "ironic-backport" so we can check the patches that will need more easily =) | 19:53 |
| clarkb | iurygregory: that feels like redundant effort | 19:54 |
| clarkb | why not just add the tag instead of voting +/-1 ? | 19:54 |
| fungi | do the same users who have access to vote on backport-candidate also have access to set hashtags? | 19:58 |
| iurygregory | due to the fact that people may forget =) | 19:58 |
| iurygregory | fungi, yep only "cores" | 19:58 |
| fungi | got it, so it's not a workaround for one being more available than the other | 19:58 |
| iurygregory | also the owner of the patch can set hashtags on it | 19:59 |
| clarkb | if it were me I'd chose one or the other but not use both | 20:00 |
| clarkb | if labels work better because they show up when hitting the review button then just use the labels and don't bother with hashtags | 20:00 |
| clarkb | but trying to use both just seems like it would be a mess of missing things and incomplete lists on both sides | 20:00 |
| fungi | maybe that calls for a not-a-backport-candidate hashtag then ;) | 20:01 |
| *** stevebaker has quit IRC | 20:02 | |
| *** hamalq has joined #opendev | 20:02 | |
| iurygregory | maybe we can do a test having the Backport-Candidate label in ironic acl and tell you how it goes? =) | 20:02 |
| *** stevebaker has joined #opendev | 20:02 | |
| *** logan- has quit IRC | 20:35 | |
| *** dtantsur has joined #opendev | 20:35 | |
| *** lbragstad_ has joined #opendev | 20:35 | |
| *** icey_ has joined #opendev | 20:36 | |
| *** ysandeep has joined #opendev | 20:36 | |
| *** mugsie_ has joined #opendev | 20:36 | |
| *** persia_ has joined #opendev | 20:36 | |
| *** zbr has quit IRC | 20:36 | |
| *** fdegir has quit IRC | 20:36 | |
| *** icey has quit IRC | 20:36 | |
| *** zigo has quit IRC | 20:36 | |
| *** stephenfin has quit IRC | 20:36 | |
| *** slaweq has quit IRC | 20:36 | |
| *** persia has quit IRC | 20:36 | |
| *** melwitt has quit IRC | 20:36 | |
| *** mugsie has quit IRC | 20:36 | |
| *** osmanlicilegi has quit IRC | 20:36 | |
| *** swest_ has joined #opendev | 20:36 | |
| *** Tengu has quit IRC | 20:36 | |
| *** melwitt has joined #opendev | 20:36 | |
| *** corvus has quit IRC | 20:36 | |
| *** tosky_ has joined #opendev | 20:37 | |
| *** dhellmann_ has joined #opendev | 20:37 | |
| *** dtantsur has quit IRC | 20:37 | |
| *** dtantsur has joined #opendev | 20:37 | |
| *** ysandeep has quit IRC | 20:37 | |
| *** ysandeep has joined #opendev | 20:37 | |
| *** logan- has joined #opendev | 20:38 | |
| *** JohnnyRa1 has quit IRC | 20:38 | |
| *** mtreinish has quit IRC | 20:38 | |
| *** ysandeep|PTO has quit IRC | 20:38 | |
| openstackgerrit | David Moreau Simard proposed openstack/project-config master: Retire ara projects https://review.opendev.org/c/openstack/project-config/+/777675 | 20:38 |
| *** gibi has quit IRC | 20:38 | |
| *** parallax has quit IRC | 20:38 | |
| *** mattmceuen has quit IRC | 20:38 | |
| *** Tengu has joined #opendev | 20:38 | |
| *** mordred has quit IRC | 20:39 | |
| *** mtreinish has joined #opendev | 20:39 | |
| *** parallax has joined #opendev | 20:39 | |
| *** hillpd_ has joined #opendev | 20:39 | |
| *** jrosser has quit IRC | 20:40 | |
| *** Eighth_Doctor has quit IRC | 20:40 | |
| *** hillpd has quit IRC | 20:40 | |
| *** hillpd_ is now known as hillpd | 20:40 | |
| *** tosky has quit IRC | 20:40 | |
| *** noonedeadpunk has quit IRC | 20:40 | |
| *** kevinz has quit IRC | 20:40 | |
| *** guillaumec has quit IRC | 20:40 | |
| *** dhellmann has quit IRC | 20:40 | |
| *** dhellmann_ is now known as dhellmann | 20:40 | |
| *** jpenag has joined #opendev | 20:40 | |
| *** jrosser has joined #opendev | 20:40 | |
| *** JohnnyRa1 has joined #opendev | 20:40 | |
| *** mnasiadka has quit IRC | 20:41 | |
| *** tosky_ is now known as tosky | 20:42 | |
| *** jpena|off has quit IRC | 20:42 | |
| *** dtantsur|afk has quit IRC | 20:42 | |
| *** mnasiadka has joined #opendev | 20:42 | |
| *** swest has quit IRC | 20:43 | |
| clarkb | iurygregory: one small thing with the syntax of one of the group files I noted in review | 20:43 |
| *** lbragstad has quit IRC | 20:43 | |
| *** corvus has joined #opendev | 20:44 | |
| iurygregory | clarkb, tks! | 20:49 |
| *** mattmceuen has joined #opendev | 20:54 | |
| *** mordred has joined #opendev | 20:58 | |
| *** guillaumec has joined #opendev | 21:10 | |
| *** Eighth_Doctor has joined #opendev | 21:20 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Don't test rebasing with unstaged changes https://review.opendev.org/c/opendev/git-review/+/777456 | 21:28 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Add a test helper for creating unstaged changes https://review.opendev.org/c/opendev/git-review/+/777687 | 21:28 |
| clarkb | openafs is installing on ze01.opendev.org now | 21:41 |
| clarkb | should be able to turn it on soon | 21:41 |
| clarkb | infra-root zuul is done zuuling ze01.opendev.org. I'm going to start it now if there are no objections | 21:46 |
| clarkb | and if that looks reasonably happy I'll try to trigger a graceful stop on ze01.openstack.org through the container | 21:47 |
| clarkb | fungi: left some notes on those git review changes | 21:55 |
| clarkb | I'm not hearing objections I'll start zuul-executor on ze01.opendev.org now | 21:57 |
| clarkb | it is updating lots of repos | 21:58 |
| clarkb | now to gracefully stop the executor on ze01.openstack.org | 21:58 |
| clarkb | the docs say zuul-executor pause will allow the jobs to drain out | 22:00 |
| clarkb | I didn't realize that existed, but I think that is what I want | 22:00 |
| clarkb | corvus: do you have to zuul-executor stop after doing zuul-executor pause and letting the jobs drain out? | 22:00 |
| clarkb | oh reading the code says zuul-executor graceful is pause + stopping | 22:01 |
| clarkb | that doesn't appaer to be documented | 22:01 |
| clarkb | zuul-executor graceful has been issues on ze01.openstack.org | 22:05 |
| *** mkowalski_ has quit IRC | 22:06 | |
| *** mkowalski has joined #opendev | 22:07 | |
| corvus | clarkb: seems like you got your answer. docker-compose down after graceful sgtm | 22:11 |
| clarkb | corvus: yup though I realized our docker-compose file says restart always so I think it will restart the container | 22:11 |
| clarkb | I expect I need to pause, then wait for it to idle then down it | 22:12 |
| clarkb | I'll let it run through its course of the current graceful stop to confirm though | 22:12 |
| corvus | clarkb: right 'docker-compose down' will take it down | 22:12 |
| clarkb | right, but there is a race between zuul-executor idling then stopping and me running that command | 22:12 |
| clarkb | I don't think I can beat a cpu for that | 22:12 |
| clarkb | its not a huge deal, will just be a little longer to get the old executor out of the rotation | 22:13 |
| corvus | clarkb: i agree. we sholud change that | 22:13 |
| mkowalski | Hello everyone! Haven't been contributing for quite some years, so I have a question about changes between gerrit and launchpad... | 22:17 |
| mkowalski | I open a bug in launchpad, then I open a review in gerrit. Review is linked via "Closes-Bug: #id" and indeed is clickable | 22:17 |
| mkowalski | I remember in the past launchpad bug would be automatically assigned to me | 22:17 |
| mkowalski | Has that changed or is it happening only once in a while or did I mess up with some email linking between accounts? | 22:18 |
| clarkb | mkowalski: the hooks that made that happen relied on the old gerrit sql database. We've since upgraded to a much newer version of gerrit that uses notedb instead and we haven't had time to update the hooks or replace them with something else that works | 22:18 |
| mkowalski | clarkb: I see; so the expected workflow would be that I assign myself in launchpad and write a comment with gerrit review id or something, right? | 22:20 |
| clarkb | yes, that should work just fine | 22:20 |
| mkowalski | Great, thanks for the info | 22:22 |
| clarkb | corvus: I think we can use restart: on-failure in the docker-compose file | 22:25 |
| clarkb | as for this case I'll just let it restart, then use pause and monitor for it to idle. Then I can down it | 22:26 |
| *** lbragstad_ is now known as lbragstad | 22:26 | |
| corvus | heh, i'm having trouble keeping up with channel changes; responded in zuul :) | 22:27 |
| corvus | (i'm pretty deep into non-opendev stuff today; only occasionally coming up for air) | 22:28 |
| clarkb | thats fine, I don't think its urgent | 22:28 |
| *** tkajinam has joined #opendev | 22:58 | |
| *** stewie925 has joined #opendev | 23:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: zuul-summary-status : handle SKIPPED and ERROR jobs https://review.opendev.org/c/opendev/system-config/+/777298 | 23:34 |
| clarkb | infra-root I've dug into stewie925's account situation a bit and have infos | 23:45 |
| clarkb | there is an old account and a new account. The email address associated with the openid for the accounts was removed from the old account. This meant on next login gerrit decided that a new account should be created. When it did this it simply updated the accountId record in the externalid for the openid. This means the old account has no openid login ability. | 23:46 |
| clarkb | however, stewie925 is still pushing code via ssh to the old account. | 23:46 |
| clarkb | Unfortunately do to the account inconsistency issues we can't just push a quick fix for that. | 23:46 |
| clarkb | infra-root I guess I'm looking for input on how we might proceed from here. | 23:49 |
| ianw | clarkb: i felt there was a theory that this couldn't happen any more since the upgrade? | 23:50 |
| clarkb | ianw: you can't have a conflicting email after the upgrade | 23:50 |
| clarkb | ianw: but new gerrit has the bug where it will create a new account if the email address on an existing openid isn't found on the old account and it will just move the openid for you | 23:50 |
| clarkb | they are different bugs, but similarly challenging | 23:50 |
| clarkb | note there is no email conflcit here, both accounts have completely distinct emails | 23:51 |
| ianw | hrm, ok. i guess how could gerrit have known the login was for the same account? | 23:51 |
| clarkb | ianw: because the openid itself didn't change | 23:51 |
| clarkb | there is only one openid involved and it associated with the new account because gerrit simply changed the accoundId value in that externalid record. This is a bug imo | 23:52 |
| clarkb | one options is to kindly ask stewie925 to migrate to the new account and start using it for ssh. We would then retire the old account to avoid confusion. If we do that the changes associated to the old account will be somewhat orphaned if you try to associate them to stewie925 | 23:54 |
| clarkb | another option is to plan a downtime and an external id update that reassociates the openid to the old account and adds the openid email back to that account | 23:55 |
| clarkb | a third option is to more quickly fix the account inconsistencies due to external id email conflicts then revisit this with stewie925 in a week or two | 23:56 |
| clarkb | my struggle with all of these is that none are great options. the first one is less ideal for stewie925. The second two rely on us to do gerrit surgery which I'd much prefer to take slowly as we're prepared for it | 23:56 |
| clarkb | if we go with the third option I can reprioritize trying to resolve those conflicts | 23:57 |
| fungi | yeah, this sounds like all options are non-ideal. i lean toward considering it a workflow-type bug with the openid plugin | 23:57 |
| clarkb | and yes, maybe we should file a bug for the openid accounts get reassociated issue | 23:58 |
| stewie925 | thank you clarkb - third option is good for me | 23:58 |
| clarkb | I don't think that has happened yet, but we do seem to understand it fairly well now | 23:58 |
| clarkb | stewie925: if we want to go with the third option then any reviews you do via http will end up being orphaned as belonging to the new account | 23:58 |
| clarkb | stewie925: I take it that is preferable to you? | 23:59 |
| clarkb | also if we go with the third option, maybe we check back in at the end of next week and see if we've made progress with fixing the ~650 email conflict accounts | 23:59 |
| clarkb | I can reprioritize that next week and see if we can make better progress. If we do make progress then we can try and stick it out. If we don't then maybe we reconsider one of the other two options? | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!