| *** ysandeep|out is now known as ysandeep | 06:04 | |
| *** ykarel_ is now known as ykarel | 06:56 | |
| *** bhagyashris__ is now known as bhagyashris | 07:27 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** ykarel is now known as ykarel|lunch | 08:36 | |
| *** ykarel|lunch is now known as ykarel | 10:07 | |
| opendevreview | Merged openstack/project-config master: Set empty nodepool resource lists on inap https://review.opendev.org/c/openstack/project-config/+/807204 | 10:10 |
|---|---|---|
| *** dviroel|out is now known as dviroel | 11:21 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** jpena|lunch is now known as jpena | 12:39 | |
| dtantsur | hi folks! is there a way to opt-out of PDF builds in openstack-docs? | 13:36 |
| fungi | there's probably a switch, but also worth bringing up with the tc i think since there was talk of "pdf docs" being a yoga or z cycle goal | 13:49 |
| fungi | which job are you running which is trying to build pdfs? i'll look at it and see what's involved in making it skip that part | 13:50 |
| fungi | i know there's a bunch of openstack projects not building pdfs (yet anyway) and their docs jobs are still working | 13:51 |
| dtantsur | fungi: we have a lot of garbage in the rendered pdf, and an addition in my patch seems to cross some threshold after which the PDF refuses to render | 13:51 |
| fungi | neat-o | 13:52 |
| dtantsur | the error output is absolute garbage, but I have a guess that I can remove a few images | 13:52 |
| dtantsur | which is fine for now, but if the renderer starts to force us to remove important content... | 13:52 |
| dtantsur | fungi: check out page 605 and on: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_64a/807295/1/check/openstack-tox-docs/64ac6aa/pdf/doc-ironic.pdf | 13:52 |
| fungi | well, i was just asking for the job name so i can find where it decides whether to make pdfs, but i can extract it from that raw logset | 13:53 |
| dtantsur | yeah, sorry, I was finishing my complaint | 13:53 |
| dtantsur | it's openstack-tox-docs | 13:53 |
| fungi | aha, it's openstack-tox-docs right there in the url, yep | 13:54 |
| fungi | don't feel you need to complain to me, i didn't make that job, and i honestly don't care if openstack has pdf documentation or not (though it seems others do) ;) | 13:54 |
| fungi | but yeah, i'll see if i can spot how to turn it off | 13:54 |
| dtantsur | yeah, I don't always know where to come with such things :) | 13:55 |
| dtantsur | re TC goal: is there already a proposal to -1? :) | 13:55 |
| fungi | not sure, i wasn't following all that closely | 13:55 |
| fungi | dtantsur: judging from https://zuul.opendev.org/t/openstack/build/8dd40c7b4ead4238953817129ace6427/console#3/0/31/ubuntu-focal you could probably just remove the [testenv:pdf-docs ] section from the tox.ini file, but i'll dig deeper and see if there's a rolevar or something you can set in a job variant instead | 13:58 |
| fungi | dtantsur: bingo... https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/tox-docs/run.yaml#L6 | 14:00 |
| dtantsur | okay, thanks! we'll use it as the last resort if nothing helps | 14:00 |
| fungi | but yeah, if you add a openstack-tox-docs variant in your check/gate pipelines and set tox_skip_pdf=true in a vars list for it, that should stop trying to build pdfs | 14:02 |
| fungi | dtantsur: here's an example where it's being done in a job inheriting from openstack-tox-docs, but you should be able to do similarly in your project-pipeline with an openstack-tox-docs variant since it's not marked final: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L952 | 14:04 |
| dtantsur | nice | 14:04 |
| fungi | though i'll note that codesearch doesn't turn up any projects doing it in their master branches, at least, so there might be a reason | 14:04 |
| Clark[m] | https://governance.openstack.org/tc/goals/#community-goals looks like it was a train goal | 14:06 |
| fungi | which was never completed i suppose | 14:11 |
| *** ykarel is now known as ykarel|away | 14:44 | |
| yoctozepto | infra-root I have a build that seems to have hung https://zuul.opendev.org/t/openstack/stream/bf48b72319d0434e81ecdc50c00b1a68?logfile=console.log | 14:50 |
| yoctozepto | now it timed out | 14:52 |
| fungi | yoctozepto: i guess it was on post-run tasks? the job looks like it had already failed with the test-zun.sh script exiting 1 | 14:56 |
| fungi | https://zuul.opendev.org/t/openstack/build/bf48b72319d0434e81ecdc50c00b1a68/console#2/1/50/primary | 14:57 |
| yoctozepto | fungi: yeah, that failure was expected; it hang on collecting the logs etc. | 14:57 |
| fungi | interesting, could you tell what it was running exactly or what the kast task name was? | 14:58 |
| fungi | last task name | 14:58 |
| yoctozepto | 2021-09-03 14:22:26.982141 | LOOP [fetch-output : Collect logs, artifacts and docs] | 14:58 |
| yoctozepto | primary and secondary2 gave ok | 14:58 |
| yoctozepto | so it seems secondary1 was lagging | 14:58 |
| fungi | i wonder if ansible was unable to reach it | 14:59 |
| yoctozepto | for half an hour | 14:59 |
| fungi | i think it will wait for a very long time for ssh to time out | 14:59 |
| yoctozepto | half an hour seems extreme (-: | 14:59 |
| fungi | yeah, i agree. it wreaks havoc with our continuous deployment jobs, because if a server is hung then ansible tasks just pile up | 15:00 |
| yoctozepto | meh :-( | 15:00 |
| fungi | i want to say ansible itself will wait up to an hour for the ssh connection to complete, but we likely timed the playbook out before then in the job | 15:00 |
| yoctozepto | guess so | 15:01 |
| fungi | if the playbook timeout were set higher, you'd have eventually seen ansible complain the host was unreachable, i think | 15:01 |
| fungi | after, like, an hour | 15:01 |
| fungi | it's a pathological condition though, only seems to manifest if the server is reachable over the network and will negotiate an ssh key exchange but then never complete the login (extremely high system load, unresponsive rootfs, et cetera) | 15:03 |
| fungi | unfortunately we see those conditions far more often because... clouds | 15:03 |
| yoctozepto | cloudy vision | 15:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade gitea to 1.15.2 https://review.opendev.org/c/opendev/system-config/+/803231 | 16:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force gitea failure for interaction https://review.opendev.org/c/opendev/system-config/+/800516 | 16:01 |
| clarkb | I think https://review.opendev.org/c/opendev/system-config/+/803231 is mergeable from a gitea perspective now. The bug fixes we wanted have landed. However, it may be worth getting the opendev/assets work finished first so that we can stop relying on gitea for those assets as this change forces us to update their urls | 16:02 |
| clarkb | I'll put a hold on 800516 and we can double check the results | 16:02 |
| clarkb | there is also a 1.14.7 which fixes an important bug with gitea dump which we don't use. I think we'll be ok without upgrading to 1.14.7 | 16:05 |
| clarkb | unfortunately they don't link to the issue and I'm having a really hard time finding what issue they fixed :/ | 16:07 |
| fungi | sounds worth checking then | 16:08 |
| fungi | yeesh, 1.6k issues | 16:09 |
| fungi | clarkb: https://github.com/go-gitea/gitea/issues/16933 maybe? | 16:09 |
| fungi | oh, nevermind, wrong link | 16:10 |
| clarkb | https://github.com/go-gitea/gitea/issues/16831 that is the issue | 16:10 |
| clarkb | which is not linked to in the change log :/ | 16:10 |
| fungi | https://github.com/go-gitea/gitea/pull/16898 | 16:10 |
| clarkb | but ya I don't think that is critical for us because we don't use gitea dump and we don't really have user accounts | 16:11 |
| fungi | agreed | 16:11 |
| fungi | yeah, looks like the pr which fixed that issue didn't include a reference to the issue number/url | 16:12 |
| fungi | and we're steering clear of 1.15.x releases for now, right? | 16:13 |
| fungi | looks like 1.15.1 was released yesterday | 16:13 |
| clarkb | fungi: yes the 1.15.x release is mergeable assuming 803231 passes testing and the held job on 800516 looks good (I just pushed those updates). But it changes the url for gerrit to load the opendev url in the theme. This is why the opendev/assets work is happening | 16:14 |
| clarkb | I think it might be better to finish up the opendev/assets work first if we can. But if necessary we can upgrade to 1.15.2 and restart gerrit and fix it up as required | 16:14 |
| fungi | oh, right-o, forgot that was the driver for it | 16:14 |
| clarkb | Basically we're ready if our hand is forced, but the other path is a bit more agreeable | 16:15 |
| clarkb | Other things we should start thinking about are nodepool launcher restarts to pick up my fix for launcher reg cleanup in zk. And a general zuul restart if we can sneak it in around openstack release activity. | 16:16 |
| clarkb | I can do the nodepool launcher restarts later today. Right now my window for garden and bike ride is closing so I should do those things :) | 16:16 |
| clarkb | For the zuul restart it might be good to get corvus' fix for the orphaned build request in first if that hasn't landed yet | 16:16 |
| fungi | sure thing. i'm around, though working from the patio because the weather's conducive | 16:17 |
| corvus | oh it had a flake8 fail, i'll fix it up real q | 16:21 |
| *** hjensas is now known as hjensas|afk | 16:29 | |
| *** ysandeep is now known as ysandeep|out | 16:41 | |
| *** jpena is now known as jpena|off | 16:41 | |
| *** odyssey4me is now known as Guest6198 | 18:03 | |
| clarkb | https://172.99.67.72:3081/opendev/system-config seems to render ok. I think that means we're still good with gitea 1.15.2 should we decide to go there before the assets updates | 19:31 |
| clarkb | I'm going to eat some lunch then do nodepool launcher restarts | 19:31 |
| clarkb | I guess while I'm looking at the launchers I should double check the inap resources got cleaned up | 19:32 |
| *** dviroel is now known as dviroel|af | 19:35 | |
| *** dviroel|af is now known as dviroel|afk | 19:35 | |
| fungi | yeah, that held node lgtm | 19:35 |
| clarkb | I've restarted the launcher on nl02 | 20:17 |
| clarkb | if I see a few nodes properly transition state there I'll proceed to the others | 20:20 |
| clarkb | Detailed node error: MessagingTimeout <- the inmotion cloud seems unhappy I'm going to guess we lost the rabbitmq cluster based on that error and previous issues | 20:21 |
| clarkb | other nodes on nl02 are happy though, I'm going to proceed | 20:22 |
| clarkb | fungi: I think https://review.opendev.org/c/openstack/project-config/+/807205 is ready to be approved if you want to take a look. nodepool list and nodepool image-list both lack inap entries | 20:26 |
| clarkb | #status log Restarted nl01-04 on 4edaeba70265396de415d2c7519b4ff8415e7750 | 20:31 |
| opendevstatus | clarkb: finished logging | 20:31 |
| fungi | looking | 20:39 |
| clarkb | thanks for approving 807205 | 20:43 |
| opendevreview | Gage Hugo proposed opendev/irc-meetings master: Move openstack-helm meeting to #openstack-helm https://review.opendev.org/c/opendev/irc-meetings/+/805094 | 20:46 |
| clarkb | huh I had missed that opensatck wanted to stop doing meetings in meeting channels | 20:49 |
| opendevreview | Merged openstack/project-config master: Remove the inap provider from nodepool https://review.opendev.org/c/openstack/project-config/+/807205 | 20:51 |
| clarkb | Ya checking /var/log/kolla/rabbitmq I see lots of disconnections | 20:53 |
| clarkb | this seems similar to what happened last time. yuriys isn't here, but I wonder if we're still having nic issues there | 20:53 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!