| *** brinzhang0 has quit IRC | 00:02 | |
| openstackgerrit | Merged openstack/project-config master: nodepool: pause opensuse-15 builds https://review.opendev.org/c/openstack/project-config/+/767642 | 00:06 |
|---|---|---|
| ianw | copying it via https locally, and via rsync locally it is *very* slow. wgetting it from another mirror it comes in at ~2mb/s | 00:13 |
| ianw | this is pointing to an afs issues | 00:13 |
| ianw | i/o to the disk is ok 209715200 bytes (210 MB, 200 MiB) copied, 7.11467 s, 29.5 MB/s | 00:15 |
| ianw | i'm going to reboot it there's a few old afs errors but nothing major | 00:17 |
| corvus | infra-root: fyi jitsi merged my etherpad patch, so we should be able to use upstram docker containers next time they update. update frequencies are kinda random though, so may be a while. just need to check https://hub.docker.com/r/jitsi/web/tags?page=1&ordering=last_updated periodically | 00:18 |
| *** mlavalle has quit IRC | 00:32 | |
| ianw | i'm not really sure what to do. the airship mirror gets the big openjdk package faster from the dfw mirror via http than rsync via afs | 00:33 |
| ianw | the cache is working; after i've downloaded it once ... 231,514,016 100% 86.30MB/s 0:00:02 (xfr#1, to-chk=0/1) | 00:35 |
| *** brinzhang has joined #opendev | 00:43 | |
| ianw | for some reason, nodepool still seems to be trying to upload to openedge | 00:44 |
| ianw | might have been an old config from when the builders were on hold | 00:54 |
| ianw | i've manually cleaned up the zk nodes | 00:56 |
| ianw | i feel like the builders are in good shape. they've got free space; images modulo the opensuse-15 issues are all building and there's no dangling images/uploads/etc. | 00:57 |
| fungi | corvus: yay! excellent news | 01:01 |
| fungi | pyn38: yes, you're right (as far as i can see) | 01:05 |
| pyn38 | fungi. Thanks again! ..It got merged :) | 01:18 |
| fungi | great! you know where to find us if you have more questions | 01:24 |
| pyn38 | Yes. I do! May I know which time zone do you work ?. SO that we know right time to ping. | 01:26 |
| *** pyn38 has quit IRC | 02:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: docs: fix wrong key in the lvm sample configuration https://review.opendev.org/c/openstack/diskimage-builder/+/767402 | 02:17 |
| fungi | different folks in here are from various parts of the world, i personally am on the east coast of north america, eastern standard time | 02:21 |
| *** hamalq_ has quit IRC | 02:50 | |
| *** pyn38 has joined #opendev | 04:14 | |
| *** larainema has joined #opendev | 04:19 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: add note about requiring debian-minimal for DIB_APT_SOURCES_CONF https://review.opendev.org/c/openstack/diskimage-builder/+/767661 | 04:41 |
| *** ykarel has joined #opendev | 04:57 | |
| *** lpetrut has joined #opendev | 05:50 | |
| *** lpetrut has quit IRC | 05:54 | |
| *** ysandeep|away is now known as ysandeep | 06:11 | |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin https://review.opendev.org/c/openstack/project-config/+/767665 | 06:16 |
| *** marios has joined #opendev | 06:20 | |
| *** pyn38 has quit IRC | 06:22 | |
| *** lpetrut has joined #opendev | 06:28 | |
| *** lpetrut has quit IRC | 06:30 | |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin https://review.opendev.org/c/openstack/project-config/+/767665 | 06:35 |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin https://review.opendev.org/c/openstack/project-config/+/767665 | 06:35 |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin https://review.opendev.org/c/openstack/project-config/+/767665 | 06:40 |
| openstackgerrit | yatin proposed zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux https://review.opendev.org/c/zuul/zuul-jobs/+/767668 | 06:43 |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin https://review.opendev.org/c/openstack/project-config/+/767665 | 06:52 |
| openstackgerrit | Dinesh Garg proposed zuul/zuul-jobs master: Allow customization of pip and helm binary https://review.opendev.org/c/zuul/zuul-jobs/+/767354 | 07:22 |
| *** eolivare has joined #opendev | 07:33 | |
| openstackgerrit | Dinesh Garg proposed zuul/zuul-jobs master: Allow customization of pip and helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/767354 | 07:35 |
| *** slaweq has joined #opendev | 07:42 | |
| *** ralonsoh has joined #opendev | 08:02 | |
| *** sboyron has joined #opendev | 08:13 | |
| *** andrewbonney has joined #opendev | 08:15 | |
| *** ysandeep is now known as ysandeep|lunch | 08:18 | |
| *** rpittau|afk is now known as rpittau | 08:30 | |
| *** sboyron has quit IRC | 08:38 | |
| *** hashar has joined #opendev | 08:50 | |
| openstackgerrit | yatin proposed zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux https://review.opendev.org/c/zuul/zuul-jobs/+/767668 | 08:53 |
| *** elod is now known as elod_pto | 09:04 | |
| *** slaweq has quit IRC | 09:16 | |
| *** slaweq has joined #opendev | 09:18 | |
| *** slaweq has quit IRC | 09:20 | |
| *** ysandeep|lunch is now known as ysandeep | 09:22 | |
| *** slaweq has joined #opendev | 09:24 | |
| openstackgerrit | Pranali Deore proposed openstack/project-config master: Add new repo for Glance Tempest Plugin https://review.opendev.org/c/openstack/project-config/+/767665 | 10:39 |
| *** DSpider has joined #opendev | 10:43 | |
| *** slaweq has quit IRC | 10:51 | |
| *** slaweq has joined #opendev | 10:55 | |
| *** ysandeep is now known as ysandeep|afk | 11:24 | |
| *** hashar is now known as hasharLunch | 11:27 | |
| *** brinzhang has quit IRC | 12:08 | |
| *** sboyron has joined #opendev | 12:10 | |
| *** ykarel_ has joined #opendev | 12:14 | |
| *** ykarel has quit IRC | 12:17 | |
| *** ykarel_ is now known as ykarel | 12:18 | |
| openstackgerrit | Tobias Henkel proposed openstack/diskimage-builder master: POC: Pre-pull docker images https://review.opendev.org/c/openstack/diskimage-builder/+/767706 | 12:20 |
| *** hasharLunch is now known as hashar | 12:22 | |
| *** sboyron has quit IRC | 12:33 | |
| *** tosky has joined #opendev | 12:58 | |
| *** ysandeep|afk is now known as ysandeep | 13:00 | |
| ykarel | review please https://review.opendev.org/c/zuul/zuul-jobs/+/767668 | 13:33 |
| *** mlavalle has joined #opendev | 13:59 | |
| *** ysandeep is now known as ysandeep|away | 14:22 | |
| ykarel | tobiash, fungi ianw ^ | 14:38 |
| *** iurygregory has joined #opendev | 14:38 | |
| fungi | ykarel: lgtm, but we should probably discuss in #zuul since zuul-jobs is a zuul project repository | 15:12 |
| ykarel | fungi, ok just joined #zuul | 15:12 |
| *** rpittau is now known as rpittau|afk | 15:38 | |
| *** hashar has quit IRC | 15:53 | |
| * prometheanfire would like to see gertty get a release, if only because it's been a while and there are a couple of commits that would be nice to have https://opendev.org/ttygroup/gertty/compare/1.6.0...master | 15:58 | |
| fungi | prometheanfire: that's one of corvus' personal projects, so you might ask him when he's around | 16:07 |
| corvus | prometheanfire: good point, i'll work on that soon | 16:24 |
| prometheanfire | corvus: yarp, thanks | 16:27 |
| prometheanfire | corvus: main reason is because I used the git version (because some py38 issue I thought at the time) and now since the DB upgraded I can't go back :P | 16:28 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux https://review.opendev.org/c/zuul/zuul-jobs/+/767668 | 16:31 |
| *** hashar has joined #opendev | 16:31 | |
| *** gibi is now known as gibi_pto | 16:35 | |
| corvus | prometheanfire: i may make a few more gerrit 3.x changes before making a release | 16:41 |
| openstackgerrit | Merged ttygroup/gertty master: Add version specific changes for git-url https://review.opendev.org/c/ttygroup/gertty/+/763885 | 16:44 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Document removal of /p from default git-url https://review.opendev.org/c/ttygroup/gertty/+/749375 | 16:45 |
| openstackgerrit | Merged ttygroup/gertty master: Document removal of /p from default git-url https://review.opendev.org/c/ttygroup/gertty/+/749375 | 16:46 |
| openstackgerrit | Merged ttygroup/gertty master: Fix auth-type in opendev example config https://review.opendev.org/c/ttygroup/gertty/+/763890 | 16:47 |
| openstackgerrit | Merged ttygroup/gertty master: tokenizer: do not try to decode strings on Python 3 https://review.opendev.org/c/ttygroup/gertty/+/753697 | 16:52 |
| *** chandankumar is now known as raukadah | 16:53 | |
| *** marios is now known as marios|out | 17:01 | |
| corvus | zbr|rover: the big constraints that caused us to decide that slim images are better for opendev: image build/upload time, hitting cloud limits on image sizes, needing newer testing tools, interaction between different tools | 17:08 |
| zbr|rover | my expectation is that a fat image would not be updated more often than twice a month? | 17:10 |
| mordred | and making sure we were testing as much of service install as possible | 17:10 |
| mordred | many years ago we had images with things pre-installed and it caused us to miss places where installation automation didn't actually install something | 17:10 |
| zbr|rover | i will try to check how big their images are, is just personal curiosity. | 17:11 |
| mordred | zbr|rover: oh - I see the question - yeah, one of the big differences is tha tgithub acitons (and other similar systems) aren't design to support end ot end integration tests that involve installation of complex software | 17:12 |
| mordred | fat images can work well when the focus is more on unit tests, etc | 17:12 |
| zbr|rover | mordred: YES. obviously that for integration, gha is uselss. | 17:13 |
| zbr|rover | but for quick tests (lint, unit, functional), it is likely that fat-image approach to be more economical. | 17:13 |
| zbr|rover | maybe they use some different virtualization that makes them care less about the size of the system images. | 17:14 |
| corvus | they only have one cloud | 17:15 |
| mordred | yah. and they don't also have the slim images | 17:15 |
| mordred | for us it would double the number of images we'd need to make- because we have to have the slim ones for integration | 17:16 |
| zbr|rover | yep | 17:16 |
| corvus | so we have local mirrors to make the pre- steps as fast as possible | 17:16 |
| mordred | many of these things are much easier when you only have one cloud | 17:16 |
| zbr|rover | probably that for us it would be easier to use containers for simple tests. | 17:16 |
| mordred | especially when you also run that cloud :) | 17:16 |
| corvus | zbr|rover: sure, we'd just download the container images from our local container image cache. similar network traffic (probably more), but potentially less install time (depending on the speed of the os package manager) | 17:17 |
| *** hamalq has joined #opendev | 17:19 | |
| *** hamalq_ has joined #opendev | 17:21 | |
| *** hamalq has quit IRC | 17:24 | |
| *** marios|out has quit IRC | 17:29 | |
| fungi | to be clear, what we had previously was two sets of images: fat images for things like unit tests and slim images for things like devstack. maintaining twice as many images is also a pain, as is having to debug why things don't install and start breaking your image builds because it's happening outside of testing, or having to reroll images in an emergency because some new release of something you're baking | 17:33 |
| fungi | in no longer works as expected | 17:33 |
| fungi | collapsing all our testing onto a consistent set of images was a lot of work up front, but saved us a ton of maintenance we'd been continuously engaged in trying to support the fat image case | 17:34 |
| fungi | it also allows projects to manage and solve their own dependency problems, rather than relying on us to fix those things when they're included directly in our images | 17:35 |
| fungi | also our fat images were an echo of an even earlier model, where we reused persistent nodes for lightweight jobs like unit testing, so the "fat" images were to some degree a transitional step from those persistent nodes which had lots of tools preinstalled on them to images with mostly the same sets of tools to avoid breaking the jobs as we moved to more and more ephemeral nodes | 17:37 |
| mordred | fungi: ah - the good old days where the reused persistent nodes had all the prerequsites maintained via 'apt-get build-dep $service' - so the process for updating deps was to first land a change to the _packaging_ which would then cause the node maint to update the packages on the test node | 17:43 |
| mordred | that system, horrible as it was, mostly worked without intervention on my part for almost a year- until it completely fell apart | 17:43 |
| mordred | (markmc especially did not like that system since it required him to update debian packaging in order to manipulate depends :) ) | 17:45 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Handle gerrit 3.x style internal urls https://review.opendev.org/c/ttygroup/gertty/+/767928 | 17:46 |
| corvus | that makes clicking links work again for me | 17:46 |
| corvus | (i had been typing in numbers) | 17:46 |
| fungi | corvus: thanks! i was seeing similar problems and thought maybe i'd done something wrong | 17:49 |
| *** ralonsoh has quit IRC | 17:57 | |
| *** eolivare has quit IRC | 18:14 | |
| *** ykarel has quit IRC | 18:28 | |
| *** hashar is now known as hasharDinner | 18:31 | |
| *** andrewbonney has quit IRC | 18:33 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Create tox_package_name for tox role https://review.opendev.org/c/zuul/zuul-jobs/+/767361 | 18:47 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Add WIP support https://review.opendev.org/c/ttygroup/gertty/+/767934 | 18:56 |
| *** hasharDinner is now known as hashar | 19:02 | |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Add WIP support https://review.opendev.org/c/ttygroup/gertty/+/767934 | 19:02 |
| corvus | fungi, prometheanfire: ^ that should come in handy too | 19:02 |
| fungi | oh, yup! | 19:03 |
| *** dviroel has quit IRC | 19:08 | |
| fungi | that ternary assignment trick never gets old, which i could remember to use it | 19:13 |
| fungi | s/which/wish/ | 19:13 |
| fungi | it's so c-like | 19:14 |
| corvus | that's why i like it :) | 19:15 |
| *** mlavalle has quit IRC | 19:48 | |
| hashar | hi corvus, I have send a config to setup the plugins/zuul-results-summary on Google Gerrit . Does that need a googler to approve / deploy? | 20:03 |
| openstackgerrit | melanie witt proposed opendev/elastic-recheck master: Add query for bug 1882521 https://review.opendev.org/c/opendev/elastic-recheck/+/767948 | 20:27 |
| openstack | bug 1882521 in OpenStack Compute (nova) ussuri "Failing device detachments on Focal" [Undecided,New] https://launchpad.net/bugs/1882521 | 20:27 |
| melwitt | ^ an effort to improve our categorization rate | 20:28 |
| melwitt | *classification rate | 20:29 |
| *** slaweq has quit IRC | 20:54 | |
| *** Alex_Gaynor has joined #opendev | 21:16 | |
| Alex_Gaynor | Is anything going on with the arm64 builders for OpenDev? I'm seeing jobs that haven't started for a while: https://zuul.opendev.org/t/pyca/status/ | 21:17 |
| fungi | i can take a look once i'm done cooking dinner. hopefully that linaro cloud hasn't fallen offline again | 21:31 |
| fungi | Alex_Gaynor: not offline i don't think, graphs show we're using nodes there, but only using roughly half the max-servers we set: https://grafana.opendev.org/d/pwrNXt2Mk/nodepool-linaro?orgId=1 | 21:45 |
| Alex_Gaynor | fungi: this is just eyeballing the graphs, but it looks like a non-trivial percentae of node launches failed? | 21:46 |
| fungi | yeah, i concur, i haven't pulled up the launcher logs yet but the low utilization suggests nodepool is probably getting its boot requests rejected for being at/over quota (could happen if there are node leaks in the provider, for example) | 21:48 |
| fungi | in the debug log i'm seeing errors booting in linaro-us due to timeouts (nodes not becoming ready after waiting) | 21:53 |
| fungi | kevinz: ^ not sure if you're around, but is something overloaded there? | 21:54 |
| fungi | utilization has fallen even more but i still see changes waiting in the check-arm64 queue at https://zuul.opendev.org/t/openstack/status too. back to the logs now that dinner is done | 22:37 |
| *** hashar has quit IRC | 22:45 | |
| fungi | the launcher is logging occasional timeouts like this but it's not as if there's a constant stream of them: http://paste.openstack.org/show/801174/ | 22:49 |
| fungi | huh, well it apparently caught up after all, i'm not sure why it managed to clear the queue of pending kolla and kolla-ansible changes which were missing node assignents | 22:53 |
| fungi | er, assignments | 22:53 |
| fungi | i don't see anything queued for https://zuul.opendev.org/t/pyca/status either | 22:54 |
| *** mlavalle has joined #opendev | 22:56 | |
| fungi | three occurrences of boot timeouts in the past two hours | 23:07 |
| ianw | odd; unfortunately not much we can do but report upstream | 23:11 |
| fungi | yeah, i wasn't able to pinpoint why node requests were either not being seen by the launcher or not being claimed | 23:14 |
| fungi | but it eventually worked through whatever was blocking it | 23:14 |
| *** dmsimard5 has joined #opendev | 23:50 | |
| *** dmsimard has quit IRC | 23:52 | |
| *** dmsimard5 is now known as dmsimard | 23:52 | |
| *** DSpider has quit IRC | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!