Monday, 2019-10-21

mordredcorvus: so - there should be master gerrit images with checks plugin added for you now00:04
*** d34dh0r53 has quit IRC00:51
*** cloudnull has quit IRC00:52
*** armax has quit IRC01:04
*** armax has joined #openstack-infra01:04
*** ociuhandu has joined #openstack-infra01:08
*** armax has quit IRC01:08
*** ociuhandu has quit IRC01:13
*** jamesmcarthur has joined #openstack-infra01:18
*** jamesmcarthur has quit IRC01:30
*** hamzy_ has quit IRC01:33
*** jamesmcarthur has joined #openstack-infra01:36
*** jamesmcarthur has quit IRC01:51
*** jamesmcarthur has joined #openstack-infra01:57
*** yamamoto has joined #openstack-infra02:00
*** ricolin has joined #openstack-infra02:29
*** larainema has joined #openstack-infra02:31
*** spsurya has joined #openstack-infra02:32
*** xek has quit IRC02:49
*** yamamoto has quit IRC02:59
*** yamamoto has joined #openstack-infra02:59
*** armax has joined #openstack-infra03:03
*** armax has quit IRC03:08
*** jamesmcarthur has quit IRC03:49
*** ociuhandu has joined #openstack-infra04:31
*** factor has quit IRC04:32
*** ociuhandu has quit IRC04:36
*** armax has joined #openstack-infra05:04
*** dave-mccowan has quit IRC05:08
*** armax has quit IRC05:09
*** jhesketh has quit IRC05:11
*** markvoelker has joined #openstack-infra05:21
*** markvoelker has quit IRC05:26
*** ykarel|away has joined #openstack-infra05:45
*** kjackal has joined #openstack-infra05:49
*** iokiwi has quit IRC05:51
*** iokiwi has joined #openstack-infra05:51
*** AJaeger has quit IRC05:54
*** Lucas_Gray has joined #openstack-infra06:02
*** AJaeger has joined #openstack-infra06:04
*** yamamoto has quit IRC06:05
*** slaweq has joined #openstack-infra06:31
*** iurygregory has joined #openstack-infra06:36
*** pkopec has joined #openstack-infra06:39
*** ccamacho has joined #openstack-infra06:39
*** iurygregory_ has joined #openstack-infra06:42
*** iurygregory has quit IRC06:43
*** iurygregory_ is now known as iurygregory06:43
*** SpamapS has quit IRC06:44
*** dpawlik has joined #openstack-infra06:51
*** ianychoi has quit IRC06:51
*** jbadiapa has joined #openstack-infra06:52
*** yamamoto has joined #openstack-infra06:53
*** ianychoi has joined #openstack-infra06:54
*** SpamapS has joined #openstack-infra06:57
*** lmiccini has joined #openstack-infra06:57
*** jtomasek has joined #openstack-infra07:01
*** armax has joined #openstack-infra07:05
*** jbadiapa has quit IRC07:08
*** armax has quit IRC07:11
*** Lucas_Gray has quit IRC07:14
*** tosky has joined #openstack-infra07:14
*** pcaruana has joined #openstack-infra07:16
*** tesseract has joined #openstack-infra07:22
*** gfidente has joined #openstack-infra07:22
*** rcernin has quit IRC07:32
*** kjackal has quit IRC07:35
*** kjackal_v2 has joined #openstack-infra07:35
*** yamamoto has quit IRC07:44
*** yamamoto has joined #openstack-infra07:47
*** gfidente has quit IRC07:48
*** lucasagomes has joined #openstack-infra07:49
*** rpittau|afk is now known as rpittau07:51
*** gfidente has joined #openstack-infra07:51
*** e0ne has joined #openstack-infra08:00
openstackgerritIan Wienand proposed opendev/infra-specs master: Spec to retire static.openstack.org  https://review.opendev.org/68385208:00
ianwfungi / corvus : I'd appreciate if you could check that the updated section there on the CNAME redirects, etc. matches what you were thinking ^08:00
*** ralonsoh has joined #openstack-infra08:07
*** dchen has quit IRC08:12
*** tosky_ has joined #openstack-infra08:15
*** tosky_ has quit IRC08:15
*** tosky_ has joined #openstack-infra08:15
*** tosky is now known as Guest3964008:16
*** tosky_ is now known as tosky08:16
*** pkopec has quit IRC08:28
*** dpawlik has quit IRC08:34
*** electrofelix has joined #openstack-infra08:35
fnordahlWould there be a config-core around with a spare moment for https://review.opendev.org/#/c/689325/ ?08:36
*** derekh has joined #openstack-infra08:36
*** yamamoto has quit IRC08:38
*** yamamoto has joined #openstack-infra08:40
*** vesper has joined #openstack-infra08:40
*** vesper11 has quit IRC08:40
*** hamzy_ has joined #openstack-infra08:41
*** dtantsur|afk is now known as dtantsur08:41
*** dpawlik has joined #openstack-infra08:44
mordredfnordahl: done08:48
fnordahlta mordred08:48
*** ociuhandu has joined #openstack-infra08:56
openstackgerritMerged openstack/project-config master: Add gerritbot trigger for OpenDev/x charm projects  https://review.opendev.org/68932509:00
*** ociuhandu has quit IRC09:00
*** kopecmartin has joined #openstack-infra09:01
*** jaosorior has joined #openstack-infra09:06
*** armax has joined #openstack-infra09:07
*** beagles is now known as beagles|afk09:08
*** ociuhandu has joined #openstack-infra09:10
*** ociuhandu has quit IRC09:11
*** armax has quit IRC09:11
*** ociuhandu has joined #openstack-infra09:12
*** ykarel|away has quit IRC09:15
*** xek has joined #openstack-infra09:18
*** yamamoto has quit IRC09:24
*** yamamoto has joined #openstack-infra09:28
*** openstackgerrit has quit IRC09:37
*** ociuhandu has quit IRC09:47
*** ociuhandu has joined #openstack-infra09:48
*** ykarel|away has joined #openstack-infra09:49
*** ociuhandu has quit IRC09:53
*** ociuhandu has joined #openstack-infra09:54
*** ociuhandu has quit IRC09:55
*** ociuhandu has joined #openstack-infra09:56
*** ociuhandu has quit IRC09:56
*** ociuhandu has joined #openstack-infra09:56
*** slaweq has quit IRC10:04
*** rkukura has quit IRC10:32
jrosserwhere is the right place to ask about stuff to do with pbr?10:35
*** yamamoto has quit IRC10:41
*** rcernin has joined #openstack-infra10:42
*** ykarel|away has quit IRC10:52
*** rcernin has quit IRC10:52
*** ykarel|away has joined #openstack-infra10:55
AJaegerjrosser: pbr is listed here https://opendev.org/openstack/governance/src/branch/master/reference/projects.yaml#L286311:06
AJaegerso, scrolling up should bring you to the right channel which is #openstack-oslo11:06
*** rpioso has quit IRC11:06
AJaegerjrosser: this is also available on the web at https://governance.openstack.org/tc/reference/projects/oslo.html but if you search for pbr, it will not help...11:07
jrosserexcellent - thanks11:07
*** armax has joined #openstack-infra11:08
*** markvoelker has joined #openstack-infra11:10
*** rkukura has joined #openstack-infra11:11
*** trident has quit IRC11:11
*** armax has quit IRC11:13
*** ociuhandu has quit IRC11:14
*** ociuhandu has joined #openstack-infra11:15
*** trident has joined #openstack-infra11:15
*** markvoelker has quit IRC11:16
*** ociuhandu has quit IRC11:19
*** ociuhandu has joined #openstack-infra11:20
*** ricolin_ has joined #openstack-infra11:23
*** rcernin has joined #openstack-infra11:24
*** ociuhandu has quit IRC11:24
*** ykarel|away has quit IRC11:25
*** ricolin has quit IRC11:25
*** goldyfruit has quit IRC11:26
*** lastmikoi has joined #openstack-infra11:29
*** rcernin has quit IRC11:34
*** lmiccini has quit IRC11:40
*** lmiccini has joined #openstack-infra11:41
*** jistr is now known as jistr|mtgs11:47
*** yamamoto has joined #openstack-infra11:52
*** rkukura has quit IRC11:53
*** yamamoto has quit IRC11:54
*** markvoelker has joined #openstack-infra11:59
*** rfolco|ruck has joined #openstack-infra12:04
*** openstackgerrit has joined #openstack-infra12:04
openstackgerritThierry Carrez proposed openstack/project-config master: Fix release jobs to work with prepare-workspace-git  https://review.opendev.org/68972512:04
*** rfolco|ruck is now known as rfolco|rover12:04
*** kopecmartin has quit IRC12:05
*** kopecmartin has joined #openstack-infra12:06
*** rh-jelabarre has joined #openstack-infra12:08
*** armax has joined #openstack-infra12:16
*** ociuhandu has joined #openstack-infra12:16
*** larainema has quit IRC12:19
*** armax has quit IRC12:21
*** another_larsks is now known as larsks12:29
*** yamamoto has joined #openstack-infra12:32
*** jaosorior has quit IRC12:32
*** whoami-rajat has joined #openstack-infra12:33
*** rosmaita has joined #openstack-infra12:38
jovial[m]I'm seeing:  "http://mirror.ord.rax.opendev.org/epel/7/SRPMS/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found" - should this mirror contain the source RPMs?12:38
*** ociuhandu has quit IRC12:44
*** yamamoto has quit IRC12:45
*** ociuhandu has joined #openstack-infra12:45
*** rpioso has joined #openstack-infra12:46
*** ociuhandu has quit IRC12:50
*** jamesmcarthur has joined #openstack-infra12:54
*** ociuhandu has joined #openstack-infra12:55
fungiconfig-core: can i get another reviewer on 689725? openstack release job breakage from the prepare-workspace-git transition13:05
fungiwe didn't consider that some jobs might be relying on there being no git remote13:07
fungithat was a hard one to predict13:07
*** yamamoto has joined #openstack-infra13:13
*** yamamoto has quit IRC13:14
*** yamamoto has joined #openstack-infra13:14
*** yamamoto has quit IRC13:18
*** yamamoto has joined #openstack-infra13:18
fungiwell, i've single-core approved it for now and will reenqueue the failed post ref(s) once it lands13:24
*** dave-mccowan has joined #openstack-infra13:27
*** bnemec has joined #openstack-infra13:29
*** beagles|afk is now known as beagles13:30
*** jamesmcarthur has quit IRC13:30
*** jamesmcarthur has joined #openstack-infra13:31
*** Lucas_Gray has joined #openstack-infra13:31
*** jamesmcarthur has quit IRC13:32
*** jamesmcarthur_ has joined #openstack-infra13:32
*** goldyfruit has joined #openstack-infra13:32
openstackgerritMerged openstack/project-config master: Fix release jobs to work with prepare-workspace-git  https://review.opendev.org/68972513:32
fungijovial[m]: confirmed we exclude srpms for epel: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/epel-mirror-update#L4613:35
*** eharney has joined #openstack-infra13:36
*** jamesmcarthur_ has quit IRC13:36
fungilooks like that choice dates back to addition of the epel mirror ~3.5 years ago in https://review.opendev.org/68972513:36
*** beagles is now known as beagles|afk13:36
*** jaosorior has joined #openstack-infra13:46
*** jamesmcarthur has joined #openstack-infra13:48
*** xek has quit IRC13:49
*** xek has joined #openstack-infra13:51
*** rosmaita has quit IRC13:53
*** jamesmcarthur has quit IRC13:53
*** ykarel|away has joined #openstack-infra13:57
*** ykarel|away is now known as ykarel13:57
fungianybody know why requests and wget don't realize this is being served with gzip encoding? https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/700152da8992cecd2ae365f26b8cdfc7d96e38bc/release-post/tag-releases/e9a111a/zuul-info/inventory.yaml14:03
fungiif i wget that, i end up with a gzip-compressed file which i need to manually decompress14:04
fungiand requests.get is stuffing the compressed data in what should be the text attribute of the response object14:04
*** rosmaita has joined #openstack-infra14:06
*** ociuhandu has quit IRC14:12
clarkbfungi: becausethe actual storedfile is gzip compressed14:17
clarkbif you wget --compression=gzip it should work14:17
*** armax has joined #openstack-infra14:17
clarkband the encoding header should be properly set14:17
*** rkukura has joined #openstack-infra14:17
fungiinteresting, so i guess that causes it to not get served with headers indicating gzip encoding?14:18
fungihappen to know off the top of your head how to do the same with python requests?14:18
*** mriedem has joined #openstack-infra14:20
*** ianychoi has quit IRC14:21
*** armax has quit IRC14:21
clarkbthe headers should indicate gzip14:22
clarkbrequests won't auto convert for you (neither does wget) but your browser does14:23
clarkbfor requests you should check the encoding header and if gzip send it through gzip library in python14:23
clarkbusually with wget or requests you get plain text because you arent sending an accept encoding list that indicated gzil14:24
clarkbbut since the actual data is gzip you get that back in this case14:24
clarkbI think the rfcs call this the identity encoding14:25
clarkbwhich is valid if no accept encoding is set or if unable to convert to an accepted encoding14:26
*** michael-beaver has joined #openstack-infra14:26
*** dmsimard1 is now known as dmsimard14:26
*** ianychoi has joined #openstack-infra14:29
*** jamesmcarthur has joined #openstack-infra14:30
*** yamamoto has quit IRC14:32
fungigot it. the requests documentation talks about how it automatically detects whether it should decompress a response on the fly, but i suppose that doesn't cover cases where you're requesting an already compressed file (which just happens to be getting served with a filename that doesn't indicate it's compressed)14:33
clarkbor mayve it only does it if you set the accept encoding header?14:34
fungithat's a good point. i'll test14:34
fungioh, actually requests is detecting it correctly after all14:39
fungii was simply passing the wrong path into it14:39
fungiand then got thrown off by the wget behavior14:40
*** georgk has quit IRC14:41
*** fdegir has quit IRC14:41
fungizuul_vars = yaml.safe_load(requests.get('https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/700152da8992cecd2ae365f26b8cdfc7d96e38bc/release-post/tag-releases/e9a111a/zuul-info/inventory.yaml').text)['all']['vars']['zuul']14:41
fungithat "just works"14:42
*** fdegir has joined #openstack-infra14:42
*** georgk has joined #openstack-infra14:42
*** dpawlik has quit IRC14:45
clarkbjovial[m]: fungi: we don't mirror source rpms on any of our rpm mirrors iirc. I want to say that is to cut down on mirror size?14:49
*** dklyle has joined #openstack-infra14:50
*** armax has joined #openstack-infra14:52
*** aaronsheffield has joined #openstack-infra14:53
fungiwe don't explicitly exclude them in the rsync commands for those though, so i guess the upstream package repositories we're mirroring from don't have the srpms trees under them14:56
*** fdegir has quit IRC14:57
*** georgk has quit IRC14:57
*** fdegir has joined #openstack-infra14:57
*** georgk has joined #openstack-infra14:58
clarkbwe do --exclude="SRPMS/" in some places14:58
fungiyeah, not the centos mirror script though14:59
openstackgerritDavid Shrewsbury proposed opendev/system-config master: Make sure gcc is installed on python-base image  https://review.opendev.org/68981315:00
clarkbfungi: epel does though15:00
clarkband fedora15:00
fungiright, i linked to where we do it for epel15:00
*** bdodd has joined #openstack-infra15:00
fungijust saying i think the reason we don't get srpms for centos must be that they're not provided in the tree we're mirroring from15:00
clarkbah yup15:01
clarkb(and maybe centos as a whole doesn't produce them?)15:01
openstackgerritDavid Shrewsbury proposed opendev/system-config master: Make sure gcc is installed on python-base image  https://review.opendev.org/68981315:02
*** ykarel is now known as ykarel|away15:04
*** goldyfruit has quit IRC15:08
*** rascasoft has quit IRC15:08
*** goldyfruit has joined #openstack-infra15:08
*** goldyfruit has quit IRC15:09
*** rascasoft has joined #openstack-infra15:10
*** lastmikoi has quit IRC15:13
*** yamamoto has joined #openstack-infra15:14
*** lastmikoi has joined #openstack-infra15:14
*** jamesmcarthur has quit IRC15:14
*** jamesmcarthur has joined #openstack-infra15:15
fungipossible15:18
*** priteau has joined #openstack-infra15:18
*** yamamoto has quit IRC15:18
*** georgk has quit IRC15:19
*** fdegir has quit IRC15:19
*** georgk has joined #openstack-infra15:20
*** fdegir has joined #openstack-infra15:20
*** jamesmcarthur has quit IRC15:21
*** jamesmcarthur has joined #openstack-infra15:22
*** diablo_rojo has joined #openstack-infra15:22
openstackgerritDavid Shrewsbury proposed opendev/system-config master: No-op change to rebuild python-builder  https://review.opendev.org/68981915:24
*** jamesmcarthur has quit IRC15:26
*** mattw4 has joined #openstack-infra15:27
*** igordc has joined #openstack-infra15:27
*** trident has quit IRC15:29
*** trident has joined #openstack-infra15:32
*** igordc has quit IRC15:34
*** priteau has quit IRC15:41
*** ianychoi has quit IRC15:43
*** ianychoi has joined #openstack-infra15:44
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Handle ":" in passwords  https://review.opendev.org/68982915:45
smcginnisAnyone aware of issues with pypi right now?15:45
smcginnis"xmlrpc.client.ProtocolError: <ProtocolError for pypi.org/pypi: 503 Backend is unhealthy>"15:45
smcginnishttps://status.python.org/ says operational15:45
*** jamesmcarthur has joined #openstack-infra15:45
*** markvoelker has quit IRC15:46
*** lmiccini has quit IRC15:48
*** markvoelker has joined #openstack-infra15:49
clarkbsmcginnis: is there context for that?15:51
clarkbhowever the xmlrpc makes me think maybe that is a CDN level error15:52
smcginnisclarkb: We've seen a release patch (proposed, not a tagging job) fail validation a couple times with that error.15:52
*** iurygregory has quit IRC15:52
smcginnisI was going to try rechecking it again later, but figured I'd check if anyone was aware of issues that I should make sure are resolved before randomly rechecking.15:53
smcginnisThe graph shows an uptick on CDN cache misses, so could just be something weird going on there.15:53
*** e0ne has quit IRC15:58
*** slaweq has joined #openstack-infra16:00
*** tesseract has quit IRC16:03
*** jaosorior has quit IRC16:03
fungiyeah, could be a fastly endpoint near that job node got unhappy16:04
*** jistr|mtgs is now known as jistr16:08
*** lucasagomes has quit IRC16:10
*** dtantsur is now known as dtantsur|afk16:14
openstackgerritMerged opendev/system-config master: No-op change to rebuild python-builder  https://review.opendev.org/68981916:20
*** markvoelker has quit IRC16:23
*** Lucas_Gray has quit IRC16:23
*** pkopec has joined #openstack-infra16:29
*** dpawlik has joined #openstack-infra16:35
*** spsurya has quit IRC16:40
*** slaweq has quit IRC16:40
*** pkopec has quit IRC16:40
*** rpioso is now known as rpioso|afk16:42
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Disable namespacing  https://review.opendev.org/68921916:47
*** ociuhandu has joined #openstack-infra16:52
*** electrofelix has quit IRC16:56
*** ociuhandu has quit IRC16:58
*** derekh has quit IRC17:00
*** markvoelker has joined #openstack-infra17:04
*** KeithMnemonic has joined #openstack-infra17:05
*** ralonsoh has quit IRC17:17
*** dpawlik has quit IRC17:23
openstackgerritMerged zuul/nodepool master: Remove unused functional testing playbooks  https://review.opendev.org/68945717:26
*** dpawlik has joined #openstack-infra17:27
openstackgerritMerged opendev/elastic-recheck master: Remove old queries: 2019-10-14  https://review.opendev.org/68853517:30
openstackgerritMerged zuul/nodepool master: Cleanup openshift pre.yaml playbook  https://review.opendev.org/68947917:31
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker  https://review.opendev.org/68928017:31
*** dpawlik has quit IRC17:33
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Handle ":" in passwords  https://review.opendev.org/68982917:35
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Collect docker logs in functional test  https://review.opendev.org/68985117:35
*** rpioso|afk is now known as rpioso17:37
*** dpawlik has joined #openstack-infra17:40
*** ccamacho has quit IRC17:40
*** weifan has joined #openstack-infra17:41
*** weifan has quit IRC17:41
*** weifan has joined #openstack-infra17:41
*** weifan has quit IRC17:46
*** dpawlik has quit IRC17:47
*** ykarel|away has quit IRC17:47
*** tosky has quit IRC17:50
*** weifan has joined #openstack-infra17:53
*** jamesmcarthur has quit IRC17:54
*** prometheanfire has quit IRC17:56
*** prometheanfire has joined #openstack-infra17:57
openstackgerritMonty Taylor proposed opendev/system-config master: Always rebuild base and builder together  https://review.opendev.org/68985717:59
mordredclarkb, Shrews, corvus, fungi: ^^ I think that should prevent the base/builder issue from re-happening17:59
fungiooh, good thinking18:01
*** eharney has quit IRC18:02
fungithere are more subtle places we could hit it, like libraries in one and library headers in the other getting out of sync18:02
openstackgerritMonty Taylor proposed opendev/system-config master: Use deb-docker mirror for upload image too  https://review.opendev.org/68985818:04
mordredfungi: yeah18:05
*** jerryz has joined #openstack-infra18:06
mordredfungi, clarkb, Shrews, corvus: we should probably do that too ^^ - we missed that in the original patch (Although it's good we did, we otherwise would have missed the issue with base and builder needing to be in-sync)18:06
clarkbmordred: I think that may be intentionally not done18:07
clarkbwe dont verify packages from our mirror so we want the published result to use the upstream verified packages18:07
mordredclarkb: this is the mirror for instaling docker on the build host18:08
clarkbya18:08
clarkbwhich is then used to make the image18:09
fungicould be obtained a mitm'd docker because no ssl18:09
fungithough...18:09
fungiare we using reprepro or rsync to make that? or passthrough cache?18:10
mordredI mean - we're using the debian package mirrors which are the same18:10
mordredreprepro18:10
*** weifan has quit IRC18:10
mordredthing is - this is a gate job - if we can't use our mirrors to make it more reliant, I'm not sure what good our mirrors are18:10
fungiso less a lack of ssl that makes mitm a risk, more lack of package list signing18:10
clarkbfungi yes18:10
fungibut yeah, i suspect this possible issue already permeates our gate+promote design18:11
clarkbif we redployed all the mirrors to have ssl then it would probably be less of a concern18:12
fungiwe may just need to see it as a reason to start signing stuff18:12
mordredyeah - if we're worried about it, it's a more pervassive issue due to any use of mirrors in gate already18:12
fungiand adding le certs, yep18:12
mordredin any case - it's not a thing we're avoiding doing on purpose - we just started using the mirror on friday because docker's download location was down18:12
clarkbmostgate jobs dont prodice release artifacts though18:12
mordredwhich isn't to say it's not a valid concern18:12
clarkbreally getting LE certs on all mirrors andusing ssl is probably sufficent fixing for this concern18:13
clarkbjust requires time to redeploy all the mirrors18:13
mordredclarkb: do we not install the debian mirrors on our post/release jobs?18:13
clarkbmordred I thnk it depends on who "our" is18:14
*** weifan has joined #openstack-infra18:14
clarkbI know loci at least does not18:14
clarkband anyone using red hat mirrors is fine18:14
mordrednod18:14
*** jamesmcarthur has joined #openstack-infra18:14
*** eharney has joined #openstack-infra18:15
mordredwhen we le those mirrors - we should make sure to remember to add apt-https-transport to the mirror setup role18:15
*** e0ne has joined #openstack-infra18:15
fungiyep18:16
*** jamesmcarthur has quit IRC18:17
clarkbanother option is to drop reprepro. I believe calebb has a thing to reuse upstream signatures while also keeping files around until they can go away18:17
mordredoh yeah?18:17
mordredI'd love to not lose upstream signatures18:18
clarkbya I think portland state is using it to build their debuntu mirrors?18:18
fungiit's totally doable18:18
calebbclarkb: only for ubuntu, i think i hardcoded some stuff that would need to be changed, i forget why18:19
calebbi think their debian stuff is just rsync18:19
calebbor whatever the common tool for debian stuff is18:19
fungii honestly think we would be best off doing both. https solves some cases for things we're mirroring where there is no signing solution, while secure-apt is less configuration on the client for things where it's a solution18:20
mordred++18:20
mordredcalebb: is the tool published somewhere? if they've done the hard part, generalizing it shouldn't be super horrible18:21
calebbhttps://github.com/squidboylan/apt-package-mirror18:22
calebboh also py218:22
calebbi forgot about that18:22
fungithe officially recommended tool for maintaining debian mirrors is https://salsa.debian.org/mirror-team/archvsync/blob/master/bin/ftpsync18:22
fungibut that's for folks who want to run an official public mirror18:22
fungi(and it does indeed use rsync under the hood)18:23
mordredcalebb: your readme doesn't mention delayed deletes - only Package verification - is that right?18:23
*** weifan has quit IRC18:23
calebbmordred: https://github.com/squidboylan/apt-package-mirror/blob/master/example_config.yaml#L2218:24
mordredso - it'll make sure that new pacakges exist before publishing new indexes, but it won't keep old packages around longer before deleting them18:24
mordredsweet!18:24
calebbit's not in the readme but it tries to do that18:24
calebbthe tool is fairly limited, i basically threw it together because mirrors.cat.pdx.edu/ubuntu was always breaking, so it basically does exactly what they need but nothing more18:25
calebbi'd be happy to hack on it more if you all know what you'd need18:26
calebbalso may need a vm to hack on, or i can find something somewhere18:26
calebbfungi: what's the difference between running a public mirror and what you need? isnt it basically the same rsync stuff but without publishing the URL on a page as a public thing?18:28
fungifor debian proper, it's amd64 and arm64 packages for stretch and buster. really just the "main" component, no contrib or non-free, but for both the normal ftp.debian.org archive and security.debian.org archive (because of ci jobs)18:29
mordredcalebb: the main difference is delayed deletes18:29
fungiwell, and scope of what's mirrored18:29
mordredyeah - that too18:29
fungithere are some dozen architectures, as well as source packages and installer udebs18:29
mordredbut functionally - deleting files from the mirror when they're deleted upstream is bad in the gate18:29
fungiand ddebs with all the stripped debug symbols18:30
calebbmordred: yeah18:30
mordredwe want at least a few hours between a file leaving the index from them leaving the mirror18:30
calebbyeah clark has explained this to me before18:30
mordredcalebb: so - looking at your code, what you're doing is spot-on18:30
mordred(also - nice choice of doing a -n rsync to get the actions list)18:31
*** jamesmcarthur has joined #openstack-infra18:31
*** weifan has joined #openstack-infra18:31
fungiftpsync does delay as well, i think. there's a tempdir used by the --delay-updates/--delete-delay combination i think. would need to verify though18:32
clarkbftpsync doesnt do verifcation that indexes match files on disk iirc18:33
fungibut yeah, for the most part the big differences for us from an official public mirror, logistically, are that we only need to mirror a fairly small subset (as far as architectures, suites and components), but also that we need to mirror security.d.o18:34
*** factor has joined #openstack-infra18:35
openstackgerritMonty Taylor proposed opendev/system-config master: Pin python images to 3.7  https://review.opendev.org/68986318:37
mordredclarkb, fungi, corvus: ^^ there's a pin patch in case we need it -I'mma mark it WIP for now18:38
fungik18:38
*** e0ne has quit IRC18:38
mordredfungi: oh cool - I didn't realize ftpsync had grown a delay-updates/delete-delay thing18:39
fungiwell, those are rsync options18:39
clarkbcalebb: we would need to update your thing to specify what we want then? other than that it should possibly work?18:40
fungicloser inspection suggests that the use of --delay-updates and --delete-delay probably only has that effect because the size of the dataset is so large18:41
mordredoh - delete-delay is still not good enough though - rsync delete-delay just delays the deletes until the end of the rsync run18:41
fungiin reality it saves the delete and update processing until the end of the run, but by then it's likely been an hour from when the indices are read18:41
fungi(when you're doing a full debian mirror i mean)18:42
fungifor a scaled-down partial mirror the natural delay there would not be as significant18:42
mordredyah18:43
fungithe main reason they get away with it is that the release and updates suites are updated very infrequently, and it's part of why they don't recommend mirroring security.d.o18:43
*** goldyfruit has joined #openstack-infra18:43
fungibut also it's just assumed that if you're running testing/unstable/experimental where things are under heavier churn, you should simply expect to need to refresh package indices before each action18:45
*** e0ne has joined #openstack-infra18:45
mordredyeah - and all of this unfortunately hasn't thought of "what if you're doing a bazillion devstack runs per day"18:45
calebbclarkb: yeah so i seem to rememebr wanting to add support for only mirroring specific archs and releases, but realizing that that wasnt easy. I don't remember the details of why i decided it was too much effort at the time18:45
calebbi wanna say it's because you actually need to process the indices and then make decisions based on that of what to grab18:46
calebbwhereas im just grabbing all the new things, checking that all the things that the indices pointed to exist, then call it good18:46
clarkbfungi: even refreshing the index prior to every install doesnt avoid tge package has been removed race18:47
mordredyeah - I think that's right18:47
mordredclarkb: ++18:47
mordredit's one of those "at extreme scale" sorts of issues18:48
fungiif the index can be replaced and the package deleted between when you refreshed your indices and when you requested the package then yes18:49
fungiftpsync does at least attempt to ensure that new packages are retrieved, then indices are replaced, then old packages are deleted in that order18:50
mordredyeah - which is definitely the right order18:50
fungibut depending on how fast that happens (and the smaller the mirror the faster it can happen) you can still race it18:50
*** kopecmartin is now known as kopecmartin|off18:50
corvusmordred: why wip on 3.7 pin?18:50
corvusmordred: oh, i see, waiting on 85718:51
calebbPSU also ran into a lot of "we're cloning a broken mirror so now we're broken" which was the main thing i was trying to address with my tool18:51
clarkbcalebb: that wasus too18:51
*** dpawlik has joined #openstack-infra18:51
fungicalebb: oh, yep, totally18:51
mordredcorvus: removed the WIP - since you were able to reproduce the issue directly on 3.818:52
mordredclarkb, fungi: either of you want to +A https://review.opendev.org/#/c/689863/ ?18:54
AJaegerI think change https://review.opendev.org/#/c/674334 for zuul-jobs ("fetch-subunit-output: collect additional subunits (2nd try)") looks good now - anybody else that wants to review it?18:55
*** weifan has quit IRC18:55
clarkbdo e18:55
*** kjackal_v2 has quit IRC18:58
*** rfolco|rover has quit IRC18:59
corvuswas there a discussion of the name of "zuul_additional_subunit_dirs" ?18:59
AJaegercorvus: I don't remember one19:00
AJaegercorvus: I see - yeah, shall I send a follwoup?19:00
AJaegerchange to subunit_additional_dirs?19:01
corvusAJaeger: that seems reasonable.  see also the comment i just left19:02
*** weifan has joined #openstack-infra19:02
corvusi'm going to grab lunch while images do their image thing19:03
AJaegercorvus: enjoy lunch19:06
openstackgerritMerged opendev/system-config master: Always rebuild base and builder together  https://review.opendev.org/68985719:06
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: fetch_subunit: Change variable  https://review.opendev.org/68986419:06
AJaegerproposed fix for the name ^19:06
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: fetch_subunit: Change variable  https://review.opendev.org/68986419:07
*** slaweq has joined #openstack-infra19:11
*** gfidente has quit IRC19:11
*** rfolco has joined #openstack-infra19:11
*** weifan has quit IRC19:12
*** weifan has joined #openstack-infra19:17
*** dpawlik has quit IRC19:21
*** weifan has quit IRC19:22
openstackgerritMerged opendev/system-config master: Pin python images to 3.7  https://review.opendev.org/68986319:25
*** aedc has quit IRC19:26
*** auristor has quit IRC19:27
*** gyee has joined #openstack-infra19:29
*** jamesmcarthur has quit IRC19:30
*** weifan has joined #openstack-infra19:33
*** bdodd_ has joined #openstack-infra19:33
*** eharney has quit IRC19:33
*** jamesmcarthur has joined #openstack-infra19:35
*** bdodd has quit IRC19:35
AJaegerpassed tests, I'll +A 674334  once I get another +2A on 689864, please19:36
clarkbAJaeger: I +2'd the second one19:37
clarkbas did mordred you should be good to go19:37
AJaegergreat - will merge both now19:37
AJaegerclarkb: you were too fast for me ;)19:38
*** jamesmcarthur has quit IRC19:39
AJaegerconfig-core, openstack/project-config has a few changes ready for +2A, reviews welcome...19:40
clarkbAJaeger: I'll take a look after a bike ride and lunch.19:40
*** bdodd has joined #openstack-infra19:41
AJaegerthat's the right priority order ;) Enjoy!19:42
*** auristor has joined #openstack-infra19:42
*** bdodd_ has quit IRC19:42
*** aedc has joined #openstack-infra19:43
*** jamesmcarthur has joined #openstack-infra19:44
openstackgerritMerged zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try)  https://review.opendev.org/67433419:48
*** jamesmcarthur has quit IRC19:51
*** jamesmcarthur has joined #openstack-infra19:53
*** dpawlik has joined #openstack-infra19:54
*** Goneri has quit IRC19:57
openstackgerritMerged zuul/zuul-jobs master: fetch_subunit: Change variable  https://review.opendev.org/68986419:58
*** weifan has quit IRC19:59
*** diablo_rojo has quit IRC20:00
*** dpawlik has quit IRC20:02
*** jtomasek has quit IRC20:02
*** mriedem is now known as mriedem_afk20:05
*** bdodd has quit IRC20:06
*** dpawlik has joined #openstack-infra20:07
*** bdodd has joined #openstack-infra20:08
*** markvoelker has quit IRC20:11
*** markvoelker has joined #openstack-infra20:12
*** weifan has joined #openstack-infra20:14
*** weifan has quit IRC20:15
*** pcaruana has quit IRC20:16
*** diablo_rojo has joined #openstack-infra20:16
*** slaweq has quit IRC20:16
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Collect docker logs in functional test  https://review.opendev.org/68985120:16
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Handle ":" in passwords  https://review.opendev.org/68982920:17
*** slaweq has joined #openstack-infra20:28
*** weifan has joined #openstack-infra20:29
*** slaweq has quit IRC20:33
*** whoami-rajat has quit IRC20:33
*** jamesmcarthur has quit IRC20:40
*** slaweq has joined #openstack-infra20:43
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Use zuul-registry as buildset registry  https://review.opendev.org/68923820:44
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Make the buildset registry port configurable  https://review.opendev.org/68924020:45
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker  https://review.opendev.org/68928020:45
*** jamesmcarthur has joined #openstack-infra20:46
*** dave-mccowan has quit IRC20:47
*** slaweq has quit IRC20:48
*** mriedem_afk is now known as mriedem20:54
*** dpawlik has quit IRC20:58
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Make the buildset registry port configurable  https://review.opendev.org/68924020:58
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker  https://review.opendev.org/68928020:58
*** jamesmcarthur has quit IRC21:00
*** goldyfruit has quit IRC21:01
*** jamesmcarthur has joined #openstack-infra21:08
*** factor has quit IRC21:08
*** tosky has joined #openstack-infra21:10
*** dpawlik has joined #openstack-infra21:11
*** e0ne has quit IRC21:13
*** mloza has joined #openstack-infra21:15
*** mloza has left #openstack-infra21:15
*** yamamoto has joined #openstack-infra21:16
*** yamamoto has quit IRC21:21
openstackgerritMerged zuul/zuul-registry master: Collect docker logs in functional test  https://review.opendev.org/68985121:23
clarkbfungi: is https://review.opendev.org/#/c/688816/1 still something we want to get in? I think it failed on the fedora 30 dnf segfaults21:23
clarkbfungi: not sure if the symlink plan changes the need for ^ or not21:23
mordredclarkb: because a segfault in dnf is a thing everyone wants21:23
clarkbmordred: yea.... its a known issue upstream and the fix is working through the release path (and maybe hsa already made it into the distro)21:24
*** markvoelker has quit IRC21:25
*** jamesmcarthur has quit IRC21:26
fungiclarkb: the symlink is still a workaround, so we either need something like 688816 or we need to change the mirror paths we're using21:28
fungier, that's what that does21:28
clarkbgotcha21:28
clarkblets recheck then and see if dnf works21:28
fungii meant, or we need to find how to make reprepro mirror to the place it's looking now21:28
fungibut yeah, if we replace reprepro then let's worry about it them21:29
fungithen21:29
clarkbya I think longer term we want to do ^21:29
clarkbto line up with upstream21:29
openstackgerritMerged openstack/project-config master: Add stable notifications to openstack-glance  https://review.opendev.org/68793121:29
openstackgerritMerged openstack/project-config master: Update neutron-tempest-plugin grafana dashboard  https://review.opendev.org/68768621:29
*** mgoddard has quit IRC21:31
mordredmaybe fleshing out calebb's thing won't be that bad. and by not that bad, I mean I think if we just give calebb a vm to work with he'll just do it all for us :)21:31
*** jamesmcarthur has joined #openstack-infra21:32
*** mgoddard has joined #openstack-infra21:32
mordredcorvus: do we have any podman install roles in zuul-jobs yet?21:33
*** eharney has joined #openstack-infra21:36
corvusmordred: no, i only have a little code in zuul-registry (where the test env is constrained to ubuntu right now).  but that could be the start of something for zuul-jobs.  also tristanC started on a change to zuul-quick-start that may be worth a look for this21:37
mordredcorvus: nod21:38
mordredcorvus: I'll look at what's there for both of those21:38
*** goldyfruit has joined #openstack-infra21:38
calebbmordred: hahaha, yeah i can probably make it happen if i have a vm to hack on21:43
calebbi've been trying to get more involved in !work things anyways so this is a good place to start21:44
*** slaweq has joined #openstack-infra21:44
openstackgerritMerged zuul/zuul-registry master: Handle ":" in passwords  https://review.opendev.org/68982921:44
*** whoami-rajat has joined #openstack-infra21:45
*** jamesmcarthur has quit IRC21:46
*** mgoddard has quit IRC21:47
*** slaweq has quit IRC21:49
mordredcalebb: \o/21:49
*** dpawlik has quit IRC21:50
openstackgerritMerged openstack/project-config master: Add openstack/barbican-ui  https://review.opendev.org/68871421:51
*** jamesmcarthur has joined #openstack-infra21:54
*** goldyfruit has quit IRC22:01
*** e0ne has joined #openstack-infra22:01
*** jamesmcarthur has quit IRC22:05
*** openstackgerrit has quit IRC22:07
*** rkukura_ has joined #openstack-infra22:11
*** jhesketh has joined #openstack-infra22:12
*** dave-mccowan has joined #openstack-infra22:13
*** rkukura has quit IRC22:14
*** rkukura_ is now known as rkukura22:14
*** e0ne has quit IRC22:16
*** e0ne has joined #openstack-infra22:18
*** hwoarang has quit IRC22:19
*** hwoarang has joined #openstack-infra22:20
*** openstackgerrit has joined #openstack-infra22:24
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker  https://review.opendev.org/68928022:24
*** bdodd has quit IRC22:33
*** bdodd has joined #openstack-infra22:35
openstackgerritMonty Taylor proposed opendev/system-config master: Run gerrit with podman instead  https://review.opendev.org/68989022:37
*** dklyle has quit IRC22:42
*** david-lyle has joined #openstack-infra22:42
*** slaweq has joined #openstack-infra22:45
*** slaweq has quit IRC22:50
*** weifan has quit IRC22:50
*** weifan has joined #openstack-infra22:51
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Pause Fedora jobs  https://review.opendev.org/68989322:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Pause Fedora jobs  https://review.opendev.org/68989322:54
*** weifan has quit IRC22:54
clarkbianw: +2 on ^ but noticed you stop f29 too and I thought f29 was working22:57
*** ianychoi has quit IRC23:00
*** ianychoi has joined #openstack-infra23:00
clarkbcalebb: re VM let us know when you are at a point where that will help23:02
*** weifan has joined #openstack-infra23:03
clarkband I'm sure we can spin something up. looks like we need about 500GB for our ubuntu mirror and 336GB for debina23:04
clarkbmaybwe we pair it down and can test on a smaller vm than that23:04
clarkbpare?23:05
calebbyeah i was thinking if i test on like 1 arch and 1 component (amd64 non-free or whatever) then i can probably test it locally for a while23:07
fungicalebb: and one release23:08
fungiwell, debian calls them "suites"23:09
calebbheh, i was trying to figure out what all the terminology was23:09
ianwclarkb: yeah, i'm just worried if somehow the packages get moved in there23:09
*** mattw4 has quit IRC23:09
fungitesting, unstable and experimental are examples of why debian repositories don't refer to those strictly as "releases"23:10
ianwclarkb: i've asked for an update on the bug ... it is fixed by the copr versions it seems, but don't want to switch to that and then have another, possibly harder to debug problem :/23:11
ianwi haven't actually checked if it's segfaulting on builders actually23:11
ianwit is, may as well pause that23:13
fungiyeah, i thought that was preventing us from getting new images built23:13
ianwsorry, haven't checked in on it in a few days (was out friday)23:13
fungiwell, it's not like any of us paused them either ;)23:14
fungiopenstack release week is always sorta distractey23:15
ianwit must be a bit of a corner case, otherwise i feel like it would get a lot more attention23:15
clarkbI think pabelanger tracked it to installing locales ?23:16
clarkbits possible it only happens when building the distro up from a very base state23:16
ianwyeah, gettext seems to be it.  maybe that rpm has something magic in it's scripts, etc23:16
openstackgerritMonty Taylor proposed opendev/system-config master: Run gerrit with podman instead  https://review.opendev.org/68989023:17
openstackgerritIan Wienand proposed openstack/project-config master: Pause Fedora 30 builds  https://review.opendev.org/68989523:18
*** rcernin has joined #openstack-infra23:19
*** rcernin has quit IRC23:19
*** rcernin has joined #openstack-infra23:20
*** ianychoi has quit IRC23:24
*** ianychoi has joined #openstack-infra23:26
*** markvoelker has joined #openstack-infra23:26
*** markvoelker has quit IRC23:30
*** tosky has quit IRC23:32
*** aaronsheffield has quit IRC23:33
*** jamesmcarthur has joined #openstack-infra23:40
*** dchen has joined #openstack-infra23:40
*** dchen has joined #openstack-infra23:42
fungiianw: i've reviewed your update to the static hosting spec. i left some minor suggestions/questions inline, but overall i'm in favor. i think it accurately captures the plan we discussed23:42
ianwfungi: thanks ... i think the theoretical static01.opendev.org does the same thing as files.openstack.org23:46
*** slaweq has joined #openstack-infra23:46
*** slaweq has quit IRC23:50
fungiyep, just maybe via different configuration management/automation23:52
*** gagehugo has quit IRC23:53
fungiand not necessarily hosting any of the apache redirect vhosts files.o.o currently serves23:53
fungisince it sounded like you were in favor of a separate containerized service to provide http redirects23:54
*** michael-beaver has quit IRC23:56
*** weifan has quit IRC23:57
*** weifan has joined #openstack-infra23:57
*** weifan has quit IRC23:58
*** weifan has joined #openstack-infra23:58
*** e0ne has quit IRC23:58
*** e0ne has joined #openstack-infra23:58
*** e0ne has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!