| prometheanfire | there's like 3-4 diferent ways to add apt keys in dib, heh | 00:05 |
|---|---|---|
| fungi | clarkb: i'm really not here, but the ghost of my connection feels compelled to suggest we check squid's behavior, its proxy caching at least used to be much easier to fine-tune than apache's comparable mods | 00:09 |
| clarkb | ya squid seems like it would get these bits better | 00:09 |
| fungi | i used to front some very popular web sites with squid caches at $oldjob and it did great | 00:10 |
| fungi | including working around some crazy site frameworks which incorporated all sorts of cache-busting tricks in service of making sure users weren't seeing old pages, except those same tricks usually also give caching proxies nightmares | 00:11 |
| fungi | things like setting content dates years in the past, negative cache ttls, et cetera | 00:12 |
| *** DSpider has quit IRC | 00:26 | |
| prometheanfire | ianw: clarkb not fully done, but this seems to be working https://dpaste.com/FWV6QUFCN | 00:31 |
| ianw | cool, that seems reasonable now it can handle keys in the directory | 00:32 |
| prometheanfire | ya, the dir only handles the binary format I think, that'd be the only drawback | 00:33 |
| ianw | i have great deja vu on that | 00:34 |
| ianw | something we did in zuul-jobs with ppas | 00:34 |
| ianw | it has to do with the .asc extensions or something ... | 00:34 |
| prometheanfire | heh, I converted gentoo to use a binary keyring as well (for image verification), easier to use | 00:34 |
| prometheanfire | no need to import stuff, just ship a gpg homedir | 00:35 |
| ianw | https://review.opendev.org/#/c/729401/ | 00:36 |
| ianw | yeah, if it's .gpg it's binary, and then i think .asc works for ascii armored too | 00:36 |
| prometheanfire | ah, well, guess that needs doc'd | 00:36 |
| prometheanfire | I'll submit this once I figure out why the image isn't building (failing on a python pre-install thing) | 00:37 |
| prometheanfire | ah | 00:39 |
| prometheanfire | 2020-08-25 00:38:57.760 | + [[ 2 == 3 ]] | 00:39 |
| prometheanfire | need to set DIB_PYTHON_VERSION=3 | 00:41 |
| openstackgerrit | Merged opendev/system-config master: Add linaro Focal mirror https://review.opendev.org/747617 | 00:41 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: DNM: copy keys into dirs https://review.opendev.org/747810 | 00:55 |
| prometheanfire | ianw: ^ | 00:55 |
| *** tkajinam_ is now known as tkajinam | 01:04 | |
| *** elod has quit IRC | 01:21 | |
| *** elod has joined #opendev | 01:23 | |
| ianw | hrm, infra-prod-base failed :/ | 01:31 |
| ianw | mirror01.us-east.openedge.opendev.org | 01:32 |
| ianw | No usable temporary directory found | 01:33 |
| ianw | odd ... full disk? | 01:33 |
| clarkb | perhaps the cache cleaner isntkeeping up | 01:34 |
| ianw | -bash: /etc/profile: Input/output error | 01:34 |
| ianw | when i try to log in ... i think this host is unhappy | 01:34 |
| donnyd | ianw: btw i fixed the block device thing with OE, so if you need to add more storages to that mirror feel free | 01:35 |
| clarkb | ssh mirror df? | 01:35 |
| clarkb | that should bypass a shell right? | 01:35 |
| ianw | [1650583.042388] Buffer I/O error on dev vda1, logical block 0, lost sync page write | 01:36 |
| ianw | [1650583.045683] EXT4-fs (vda1): I/O error while writing superblock | 01:36 |
| ianw | that's on the console | 01:36 |
| ianw | donnyd: ^ it looks like the mirror is pretty unhappy with it's storage backend | 01:36 |
| donnyd | Hrm | 01:36 |
| clarkb | oh did it remount ro? | 01:36 |
| ianw | clarkb: yeah, i think that's part of it | 01:36 |
| donnyd | Maybe reboots? Every other job on there is using the same backend | 01:36 |
| donnyd | But there was a small hiccup a few days ago. Apparently arista switches will flap all ports when you change the speed to 40g on just one | 01:37 |
| ianw | ok, let me reboot it ... it's very unhappy | 01:38 |
| ianw | mirror01 login: [1364013.350470] blk_update_request: I/O error, dev vda, sector 130667408 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 | 01:38 |
| donnyd | But that was Friday last week | 01:38 |
| ianw | is when it started | 01:38 |
| ianw | ever helpful relative timestamps | 01:38 |
| prometheanfire | ianw: I think I'm gonna move that key stuff into debian minimal (as root.d still) in a earlier number (09 to go after 08-debootstrap) | 01:40 |
| prometheanfire | otherwise it kinda overloads debootstrap | 01:41 |
| ianw | #status log rebooted mirror01.us-east.openedge.opendev.org due to it corrupting it's disk. seems ok after reboot | 01:41 |
| openstackstatus | ianw: finished logging | 01:41 |
| donnyd | ianw: i also noticed a recent patch in dib for python3 with centos and rhel breaks rhel | 01:42 |
| ianw | donnyd: which one is that? | 01:42 |
| donnyd | I mentioned it in the dib channel, but i haven't gotten around to opening up a bug on it yet | 01:42 |
| donnyd | https://opendev.org/openstack/diskimage-builder/commit/ce3b6afd4d6473d4f4c701acfa45c8239dab26bd | 01:43 |
| donnyd | This one i do believe | 01:44 |
| donnyd | It runs in pre-install and for rhel it doesn't get a sub until after pre-install | 01:44 |
| donnyd | At least with satellite it doesn't | 01:44 |
| ianw | arrgghh, hrm, we need python3 to run the package map | 01:45 |
| donnyd | We should move the registration to earlier then IMO | 01:45 |
| donnyd | It makes sense to have the first thing rhel does is subscribe | 01:46 |
| ianw | indeed. where does it happen now? | 01:47 |
| ianw | diskimage_builder/elements/rhel-common/pre-install.d/00-rhel-registration | 01:47 |
| ianw | diskimage_builder/elements/yum/pre-install.d/00-1-centos-python3 | 01:48 |
| ianw | i guess if it was 00-0 it would come first ... | 01:48 |
| ianw | what a mess :/ | 01:49 |
| ianw | we should always start things at 10, to give us at least 9 iterations of being able to realise we need things earlier in the future | 01:49 |
| donnyd | That is interesting | 01:50 |
| donnyd | I would share the logs, but from this particular build system i can't | 01:51 |
| donnyd | But it was throwing the i don't have a sub so i can't install this error | 01:51 |
| prometheanfire | ianw: or to users | 01:52 |
| ianw | donnyd: i'd be interested if you did mv diskimage_builder/elements/rhel-common/pre-install.d/00-rhel-registration diskimage_builder/elements/rhel-common/pre-install.d/00-0-rhel-registration | 01:52 |
| prometheanfire | see https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/gentoo/pre-install.d :| | 01:52 |
| ianw | if it worked | 01:52 |
| prometheanfire | because something else was at 02 iirc | 01:52 |
| donnyd | I can give that a whirl | 01:52 |
| donnyd | I will open a real bug for it, so we can sort it out there. I don't want to derail anything else you already have in flight | 01:54 |
| donnyd | Wait so why does 00-1 come up before 00? I am no bashxerpt | 01:55 |
| ianw | umm, it comes up before 00-<alphabet> | 01:57 |
| donnyd | Fwiw dib is super reliable and works very well for some pretty complex stuff.. so everyones work on it is super appreciated | 01:57 |
| donnyd | Oh yea, that makes sense | 01:57 |
| donnyd | So we should just make rhel reg 00-00 | 01:58 |
| donnyd | Or start the sub nmbering at 10 and move the python3 install to 11 | 01:59 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: copy apt gpg keys directly into trusted.gpg.d https://review.opendev.org/747810 | 02:11 |
| prometheanfire | environment.d happens after root.d iirc, maybe | 02:12 |
| prometheanfire | running a test build now | 02:14 |
| prometheanfire | oh, we are good, nice | 02:14 |
| ianw | environment.d should be sourced before running | 02:26 |
| donnyd | Ok, I got around to opening a bug. I can work on a patch, but probably not tonight. Gotta do that thing where i close my eyes and don't computer for a few hours. | 02:26 |
| ianw | donnyd: thanks, our rhel testing is unfortunately non-existent in the gate so feedback is good | 02:26 |
| ianw | i think i might have to recommit something to effectively do the base->letsencrypt->mirror dance for the new linaro mirror | 02:28 |
| donnyd | For sure - I can tinker something up and submit it. | 02:29 |
| prometheanfire | ianw: ya. built an image fine, now testing it booting | 02:30 |
| prometheanfire | booted too, so that's neat | 02:33 |
| ianw | bonus! | 02:36 |
| prometheanfire | yep, gonna call it good there | 02:38 |
| prometheanfire | made the least invasive change to achieve goals :D | 02:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 02:42 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 02:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org https://review.opendev.org/747813 | 02:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 03:14 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org https://review.opendev.org/747813 | 03:14 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt test: copy account.conf https://review.opendev.org/747814 | 03:14 |
| *** hashar has joined #opendev | 03:21 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt test: fix email match and copy account.conf https://review.opendev.org/747814 | 03:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 03:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org https://review.opendev.org/747813 | 03:43 |
| *** hashar has quit IRC | 03:47 | |
| *** hashar has joined #opendev | 03:48 | |
| prometheanfire | ianw: should I expect dib test to fail for now (should I not recheck)? | 03:59 |
| ianw | umm, not that i'm aware of, but that doesn't mean something hasn't broken :) | 04:03 |
| ianw | /opt/dib/tmp/dib_build.zsqbTWyt/hooks/root.d/09-apt-keyring: line 26: DIB_ADD_APT_KEYS: unbound variable | 04:05 |
| ianw | looks like a valid error | 04:05 |
| prometheanfire | ianw: hmm, it's exported in environment.d | 04:10 |
| prometheanfire | ya, that's the if statement, not even inside it | 04:10 |
| ianw | prometheanfire: environment.d in debian-minimal though? debootstrap is used by all the ubuntu roles too though | 04:11 |
| prometheanfire | ianw: that's where the apt key stuff was originally | 04:12 |
| prometheanfire | oh, I see it | 04:12 |
| prometheanfire | ya, I need to move that | 04:13 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: copy apt gpg keys directly into trusted.gpg.d https://review.opendev.org/747810 | 04:15 |
| prometheanfire | there are so many debian related elements it's hard to keep them in line | 04:15 |
| prometheanfire | at least gentoo keeps it to a single element :P | 04:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt test: fix email match and copy account.conf https://review.opendev.org/747814 | 04:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 04:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org https://review.opendev.org/747813 | 04:15 |
| *** ykarel has joined #opendev | 04:19 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt test: fix email match https://review.opendev.org/747814 | 04:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 04:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org https://review.opendev.org/747813 | 04:43 |
| openstackgerrit | Sampath Priyankara (samP) proposed opendev/irc-meetings master: Change Masakari Meeting time https://review.opendev.org/747819 | 04:43 |
| *** raukadah is now known as chandankumar | 04:56 | |
| *** ysandeep|away is now known as ysandeep | 05:20 | |
| *** hashar has quit IRC | 05:31 | |
| openstackgerrit | Merged opendev/system-config master: letsencrypt test: fix email match https://review.opendev.org/747814 | 05:37 |
| *** elod has quit IRC | 05:39 | |
| *** elod has joined #opendev | 05:53 | |
| openstackgerrit | Merged opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org https://review.opendev.org/747812 | 06:08 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror02.regionone.linaro.us : add missing LE file https://review.opendev.org/747839 | 06:39 |
| *** andrewbonney has joined #opendev | 07:05 | |
| *** DSpider has joined #opendev | 07:14 | |
| *** iurygregory has joined #opendev | 07:21 | |
| *** dtantsur|afk is now known as dtantsur | 07:26 | |
| openstackgerrit | Merged opendev/system-config master: mirror02.regionone.linaro.us : add missing LE file https://review.opendev.org/747839 | 07:32 |
| *** tosky has joined #opendev | 07:36 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update hashicorp jobs file with correct title https://review.opendev.org/747853 | 07:38 |
| *** ysandeep is now known as ysandeep|lunch | 07:48 | |
| *** bolg has quit IRC | 07:51 | |
| *** hashar has joined #opendev | 07:59 | |
| *** bolg has joined #opendev | 07:59 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:03 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add nim roles https://review.opendev.org/747865 | 08:40 |
| *** ysandeep|lunch is now known as ysandeep | 09:02 | |
| openstackgerrit | Xinliang Liu proposed openstack/diskimage-builder master: Make iscsi-boot element support centos 8 https://review.opendev.org/747878 | 09:29 |
| *** johnsom has quit IRC | 10:16 | |
| *** johnsom has joined #opendev | 10:17 | |
| *** aannuusshhkkaa has quit IRC | 10:49 | |
| *** ildikov has quit IRC | 10:49 | |
| *** knikolla has quit IRC | 10:49 | |
| *** dviroel has quit IRC | 10:51 | |
| *** aannuusshhkkaa has joined #opendev | 10:59 | |
| *** dviroel has joined #opendev | 11:00 | |
| *** knikolla has joined #opendev | 11:01 | |
| *** ildikov has joined #opendev | 11:10 | |
| *** hashar is now known as hasharLunch | 11:14 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Add ansible collection roles https://review.opendev.org/730360 | 11:59 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible https://review.opendev.org/734584 | 11:59 |
| *** ysandeep is now known as ysandeep|session | 12:06 | |
| *** hasharLunch is now known as hashar | 12:15 | |
| *** ykarel_ has joined #opendev | 12:16 | |
| *** ykarel has quit IRC | 12:18 | |
| *** ykarel_ is now known as ykarel | 12:19 | |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams https://review.opendev.org/747732 | 12:21 |
| *** Gyuseok_Jung has joined #opendev | 12:41 | |
| *** Guest34946 has joined #opendev | 12:47 | |
| *** Guest34946 is now known as redrobot | 12:50 | |
| *** lpetrut has joined #opendev | 12:57 | |
| openstackgerrit | Merged openstack/project-config master: Do not update upper constraints for intermediate branches https://review.opendev.org/746553 | 12:58 |
| *** ysandeep|session is now known as ysandeep | 13:16 | |
| *** ykarel is now known as ykarel|away | 13:28 | |
| *** ykarel|away has quit IRC | 13:45 | |
| *** roman_g has joined #opendev | 13:53 | |
| *** weshay is now known as weshay|interview | 13:54 | |
| dmsimard | Anyone else see that dockerhub is implementing rate limiting for free/anonymous pulls ? https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ | 14:15 |
| fungi | dmsimard: yep | 14:19 |
| dmsimard | I'd guess this could be a problem :( | 14:19 |
| fungi | they're supposedly going to be publishing guidelines/recommendations for ci systems soon | 14:26 |
| fungi | clarkb brought it up in here at 23:39 utc | 14:27 |
| *** ysandeep is now known as ysandeep|away | 14:37 | |
| clarkb | its worth noting we cached the blob layers that were previously limited. basically we tried to be good citizens there. We'll continue to try but details to be determined | 14:41 |
| *** qchris has quit IRC | 14:57 | |
| clarkb | s/cached/cache/ | 14:58 |
| *** mlavalle has joined #opendev | 15:06 | |
| *** qchris has joined #opendev | 15:10 | |
| *** chandankumar is now known as raukadah | 15:23 | |
| *** lpetrut has quit IRC | 15:32 | |
| corvus | clarkb: i'm mostly unavailable today; may be able to join for 1st part of mtg. | 15:33 |
| clarkb | corvus: ok | 15:33 |
| clarkb | I think fungi is out too, may just make it a less formal discussion if we're largely absent (and thats fine) | 15:33 |
| fungi | yeah, i can make no guarantees as to my availability at 1900z | 15:34 |
| fungi | but i'm clearly failing at this vacation thing so far | 15:34 |
| clarkb | you should fail at it less | 15:34 |
| clarkb | go kayak around the bay or something | 15:35 |
| *** owalsh has quit IRC | 15:36 | |
| *** owalsh has joined #opendev | 15:45 | |
| *** owalsh has quit IRC | 15:45 | |
| frickler | does anybody know something about mordred? I know he was moving some time ago, but that was a month ago or longer? | 15:48 |
| frickler | ha, now that I write that he seems online again | 15:49 |
| clarkb | mwhahaha: related to https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ I wonder if part of hte trouble that tripleo is seeing is the switch from rate limiting blobs to manifests | 15:55 |
| mwhahaha | yes | 15:55 |
| clarkb | mwhahaha: we should cache blobs reasonably well but don't cache manifests and so their switch (who knows when that goes into production) would be problematic | 15:55 |
| mwhahaha | you can't | 15:55 |
| mwhahaha | because they require auth | 15:55 |
| clarkb | well not with apache anyway | 15:55 |
| mwhahaha | yea | 15:55 |
| clarkb | but the cache-control is marked public so a tool that honors that can cache it | 15:56 |
| mwhahaha | we're looking to just stop using docker.io | 15:56 |
| clarkb | ah that would also work | 15:56 |
| mwhahaha | by switching to a build job parent to host content | 15:56 |
| mwhahaha | so the only thing we nmed is a base container | 15:56 |
| mwhahaha | https://review.opendev.org/#/c/747129 | 15:57 |
| mwhahaha | then the dependencies will just be a single container + rdo mirror which we already have. it should reduce overall external requests as well | 15:57 |
| mwhahaha | looks like they are also doing an opensource plan per the blog post so it might be beneficial to look into that as well | 15:58 |
| mwhahaha | anyway it's high on our radar :/ | 15:58 |
| clarkb | ya a lot of the open source and ci details are still TBD | 15:58 |
| clarkb | but will keep an eye on it to see if there are changes we can make to better accomodate their upstream updates | 15:58 |
| *** sshnaidm is now known as sshnaidm|afk | 15:59 | |
| clarkb | and based on that we can decide if we need a different caching tool (fungi suggests squid as a reverse proxy) | 15:59 |
| *** weshay|interview is now known as weshay | 16:04 | |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| clarkb | hrm is our gerrit replication config entirely in priviate host/group vars? | 16:19 |
| clarkb | ah no we moved that into inventory/ | 16:19 |
| * clarkb learns to grep better | 16:19 | |
| *** owalsh has joined #opendev | 16:23 | |
| mwhahaha | so i have a thought, could we use docker-distribution as a daily pull-through mirror where we just nuke the content on a daily basis (or some other time period) to replicate an expiration period | 16:26 |
| clarkb | we would need to set up multiple instances of that and flip flop as you can only delete content with the server off | 16:27 |
| clarkb | this is th emajor reason we've avoided the tool since other web caches don't have that problem | 16:28 |
| mwhahaha | it's just containers tho | 16:29 |
| mwhahaha | + an lb | 16:29 |
| mwhahaha | just a thought | 16:29 |
| clarkb | "just" | 16:30 |
| mwhahaha | we can probably do something similar in our own job config however if they have an opensource credential set that would be best to be handled in infra | 16:30 |
| clarkb | its also all the disk and the orchestration | 16:30 |
| clarkb | we hvae a system for that preexisting which is why we've used it | 16:30 |
| clarkb | you would have to build this all from scratch. it is doable but someone has to do it | 16:30 |
| clarkb | I'd like to see what docker says is their preferred plan for CI systems which tehy claim will be out soon | 16:31 |
| clarkb | then go from there | 16:31 |
| openstackgerrit | Lajos Katona proposed openstack/project-config master: Import netowrking-l2gw & networking-l2gw-tempest-plugin to x/ https://review.opendev.org/744110 | 16:32 |
| mwhahaha | sure but you will likely need to set the crednetials somewhere for the CI system | 16:32 |
| mwhahaha | which i don't think apache provides today | 16:32 |
| mwhahaha | so you'd need something like docker-distribution where you can put those in place | 16:32 |
| mwhahaha | which is what i'm sugesting | 16:32 |
| mwhahaha | i'd be happy to investigate implementation but i'm not certain what the limitations would need to be in to make it opendev friendly | 16:33 |
| clarkb | yes, I think apache is likely to not be the best tool. I'm not convinced the docker registry is either | 16:33 |
| clarkb | we need more information from docker and they haven't provided that yet | 16:33 |
| mwhahaha | docker registry does support swift backend | 16:34 |
| mwhahaha | but yea | 16:34 |
| clarkb | the swift backend doesn't work | 16:34 |
| mwhahaha | of course not | 16:34 |
| * mwhahaha head -> table | 16:34 | |
| clarkb | it periodically writes empty blobs to swift | 16:34 |
| clarkb | then you fetch them back and fail because the shas don't match the manifest | 16:34 |
| clarkb | we ran it in production for a short period for zuul's intermediate registry | 16:35 |
| clarkb | the replacement is zuul's container registry, but that isn't suitable for the pull through cache I don't think (though corvus can probably fill in more detail there) | 16:35 |
| mwhahaha | oh they have garbage collection now | 16:37 |
| mwhahaha | https://docs.docker.com/registry/garbage-collection/ | 16:38 |
| *** roman_g has quit IRC | 16:38 | |
| clarkb | "Note: You should ensure that the registry is in read-only mode or not running at all. If you were to upload an image while garbage collection is running, there is the risk that the image’s layers are mistakenly deleted leading to a corrupted image." | 16:38 |
| clarkb | thats the problem | 16:38 |
| clarkb | we can run mutliple registries and flip flop to work around that or use a different tool | 16:39 |
| mwhahaha | yea i was thinkign haproxy + cron job to rotate | 16:40 |
| clarkb | the problem then becomes disk | 16:40 |
| clarkb | we'd need twice as much of it | 16:40 |
| mwhahaha | what's the disk constraint issue? | 16:40 |
| mwhahaha | like we don't know how much will actually be used? or is it because it's just an unknown | 16:41 |
| clarkb | each registry would need X disk. running two would require 2X disk. We currently have about 200GB of disk for all the caching on those mirror nodes. We'd effectively half our useable cache spcae for docker hub | 16:41 |
| clarkb | oh we use the full amount currnetly | 16:41 |
| clarkb | the apache cache cleaner is very busy | 16:41 |
| clarkb | I think we give 100GB to afs and 100GB to apache currently | 16:42 |
| clarkb | and 2x docker registries probably looks like 20GB for apache, 35GB docker 1, 35GB docker 2 (with some headroom because apache does an after the fact cleanup) | 16:43 |
| mwhahaha | yea the inability to cap storage is likely a larger issue | 16:43 |
| fungi | i'm not here, but 35gb sounds like about enough docker images to make a cup of coffee | 16:47 |
| mwhahaha | accurate | 16:47 |
| fungi | i have a feeling a lot of the blob cache misses are due to us having to aggressively expire images which are actually being used regularly just because 100gb isn't enough | 16:48 |
| mwhahaha | https://github.com/docker/docker.github.io/blob/master/registry/recipes/mirror.md#what-about-my-disk seems like it might purge but not certain if that's based on like the available size on the fs being provided or via garbage collection | 16:51 |
| mwhahaha | we'll see | 16:52 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Stop replicating to the local Gerrit mirror https://review.opendev.org/747986 | 16:59 |
| clarkb | I've been digging into gerrit replication and notedb. We set replicatePermissions to false currently which means that any repo with HEAD set to refs/meta/config won't be replicated at all and the refs/meta/config branch isn't replicated for any repo | 17:00 |
| clarkb | currently All-Projects has HEAD set to refs/meta/config and All-Users has HEAD set to refs/heads/master. The upgrade process seems to change All-Users HEAD to refs/meta/config | 17:02 |
| clarkb | I think that means we're good for not replicating groups/accounts/etc on upgrade | 17:02 |
| clarkb | that leaves the actual change data which is in refs/changes/XY/ABCDXY/meta | 17:03 |
| clarkb | I'm not concerned about replicating those from a disclosure perspective but from a "will this fill our gitea disks" perspective. I'm now reading git refspecs to understand if we can exclude meta files from a refs/changes replication directive | 17:04 |
| clarkb | I'm not sure we'll actually use that if it is possible, but understanding the options here seems worhtwhile | 17:04 |
| *** hashar is now known as hasharDinner | 17:07 | |
| clarkb | it seems we can exlucde revisions but not objects? that sort of makes sense the internal objects are supposed to be largely hidden away | 17:10 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Exclude neutron q-svc logs from indexing https://review.opendev.org/747988 | 17:18 |
| *** olaph has quit IRC | 17:24 | |
| clarkb | ya I wonder if that means we'd have to stop pushing refs/changes/ entirely | 17:24 |
| clarkb | or modify the replication plugin to filter out those bits? | 17:24 |
| *** andrewbonney has quit IRC | 17:44 | |
| *** ildikov has quit IRC | 18:50 | |
| *** dviroel has quit IRC | 18:50 | |
| *** ildikov has joined #opendev | 18:52 | |
| *** dviroel has joined #opendev | 18:53 | |
| ianw | sigh, the new arm64 focal mirror already has an openafs oops and it hasn't even been turned on yet | 19:03 |
| openstackgerrit | Merged opendev/gerritlib master: Add branches arg to createProject https://review.opendev.org/741277 | 19:17 |
| clarkb | zbr: for https://review.opendev.org/#/c/729966/5 were you able to figure out what created the exceptional state in the poll? | 19:35 |
| clarkb | maybe that is what the paramiko link is? | 19:35 |
| zbr | clarkb: that happenedto me only on macos and I never found time to go deep into it in order to discover what was the out of band message send | 19:43 |
| zbr | still the spec is kinda clear that this can happen, so it would not be wrong to implement | 19:43 |
| zbr | if i remember well, i found this while trying to run e-r locally for development | 19:44 |
| zbr | but as you can see paramiko is not really under active maintenance | 19:45 |
| *** hasharDinner has quit IRC | 19:50 | |
| zbr | clarkb: now i looked again at some of my tickets with paramiko and risk going into clinical depression. https://github.com/paramiko/paramiko/issues/1383 | 19:53 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Exclude neutron q-svc logs from indexing https://review.opendev.org/747988 | 20:28 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Skip ansible-lint E208 https://review.opendev.org/748026 | 20:28 |
| clarkb | ianw: ^ lets see if that is happier | 20:28 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Update Ansible roles for ATOS and Thales HSMs https://review.opendev.org/748028 | 20:41 |
| clarkb | ianw: yup jobs pass now | 20:42 |
| ianw | all lgtm, i guess go with it! | 20:52 |
| clarkb | ++ I've approved them | 20:52 |
| clarkb | I'll let that land then go and resurrect the services that have been saddened | 20:53 |
| openstackgerrit | Merged opendev/base-jobs master: Skip ansible-lint E208 https://review.opendev.org/748026 | 20:57 |
| openstackgerrit | Merged opendev/base-jobs master: Exclude neutron q-svc logs from indexing https://review.opendev.org/747988 | 21:01 |
| ianw | sigh openstack-zuul-jobs broken with same linter issues | 21:13 |
| ianw | seems 1.8.6 has made it into groovy, which is nice (no ~pre version) and it's backported to focal/bionic with just some minor tweaks to debhelper versions | 21:24 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add ansible-collections for system-config -devel job https://review.opendev.org/747596 | 22:16 |
| clarkb | I'm trying to fix the limestone cloud cert issue. I've discovered that the cert reported by keystone is different than what we have configured in our /etc/openstack dir | 22:18 |
| clarkb | however, switching to the new cert content doesn't seem to work either | 22:18 |
| clarkb | and firefox says the cert is corrupted? | 22:18 |
| clarkb | I retrieved the new content with openssl s_client | 22:19 |
| clarkb | and it seems that it is self signed so using the cert itself as part of the trust should be fine to verify it? | 22:19 |
| clarkb | using s_client and setting -CAfile to the contents of the cert that says verification is ok | 22:21 |
| clarkb | hrm I think the issue may be my test setup isn't using the overridden clouds.yaml | 22:22 |
| clarkb | ya that seems to be the problem | 22:23 |
| clarkb | aha its the docker exec issue | 22:24 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update the limestone cert in our clouds.yaml https://review.opendev.org/748040 | 22:26 |
| clarkb | infra-root logan- ^ fyi that seems to work. Would probably be good to have logan- confirm that this change is expected and even better is confirming the new cert material is correct | 22:27 |
| clarkb | weird that firefox completely breaks on it | 22:29 |
| clarkb | chrome doesn't break on it | 22:29 |
| ianw | ok https://mirror02.regionone.linaro-us.opendev.org/ working now | 22:41 |
| clarkb | I've filed an upstream firefox bug | 22:45 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Replace linaro 01 mirror with 02, update mirror CNAME https://review.opendev.org/748046 | 22:45 |
| ianw | clarkb: if i accept the self-signed cert i see the response from https://osa.continuous.pw:5000/ | 22:48 |
| clarkb | ya I think the bug is in the cert details viewer | 22:48 |
| clarkb | the underlying tls implentations seem ok with it | 22:48 |
| ianw | clarkb: hrm, you mean clicking on the "view certificate" ? | 22:52 |
| clarkb | ianw: yup | 22:52 |
| clarkb | https://bugzilla.mozilla.org/show_bug.cgi?id=1661163 is the bug fwiw | 22:52 |
| openstack | Mozilla bug 1661163 in Untriaged "New firefox cert details viewer is unable to view a cert" [--,Unconfirmed] - Assigned to nobody | 22:52 |
| ianw | interesting, if i go through the menu bar it sends me to a page "about:certificate?" | 22:56 |
| clarkb | ya thats the page that breaks for me | 22:56 |
| ianw | for another cert it sends me to "about:certificate?cert=...stuff" | 22:56 |
| clarkb | oh it sends me to the ?cert=...stuff page | 22:57 |
| clarkb | if I go through the url bar or the navigation on the warning page | 22:57 |
| *** mlavalle has quit IRC | 22:57 | |
| *** tosky has quit IRC | 23:04 | |
| *** DSpider has quit IRC | 23:07 | |
| ianw | huh, i'm on firefox 79 | 23:19 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Replace linaro 01 mirror with 02, update mirror CNAME https://review.opendev.org/748046 | 23:24 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] Update to ARA 1.4.2 https://review.opendev.org/664478 | 23:42 |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible and galaxy roles https://review.opendev.org/734584 | 23:55 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!