Monday, 2020-12-14

*** armax has quit IRC00:05
*** zzzeek has quit IRC00:08
*** zzzeek has joined #openstack-infra00:16
*** prometheanfire has joined #openstack-infra00:20
*** zzzeek has quit IRC00:33
*** zzzeek has joined #openstack-infra00:35
*** tosky has quit IRC00:36
*** dchen has quit IRC00:39
*** armax has joined #openstack-infra00:43
*** zzzeek has quit IRC00:45
*** zzzeek has joined #openstack-infra00:49
*** eharney has quit IRC01:00
*** eharney has joined #openstack-infra01:18
*** dchen has joined #openstack-infra01:24
*** Tengu_ has joined #openstack-infra01:40
*** lifeless has quit IRC01:42
*** Tengu has quit IRC01:43
*** Tengu_ has quit IRC01:43
*** lifeless has joined #openstack-infra01:44
*** Tengu has joined #openstack-infra01:45
*** zzzeek has quit IRC01:46
*** zzzeek has joined #openstack-infra01:47
*** dchen has quit IRC02:03
*** zxiiro has quit IRC02:12
*** hamalq has joined #openstack-infra02:31
*** ociuhandu has joined #openstack-infra02:31
*** zzzeek has quit IRC02:33
*** hamalq has quit IRC02:35
*** zzzeek has joined #openstack-infra02:35
*** ociuhandu has quit IRC02:39
*** rcernin has quit IRC02:53
*** dave-mccowan has joined #openstack-infra02:56
*** auristor has quit IRC03:02
*** rcernin has joined #openstack-infra03:05
*** auristor has joined #openstack-infra03:07
*** ramishra has joined #openstack-infra03:29
*** dchen has joined #openstack-infra03:30
*** psachin has joined #openstack-infra03:32
*** rcernin has quit IRC03:33
*** rcernin has joined #openstack-infra03:40
*** rcernin has quit IRC03:41
*** rcernin has joined #openstack-infra03:42
*** __ministry1 has joined #openstack-infra03:54
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: build-wheel-cache: Ignore missing .whl when removing  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/76687603:56
ianwfungi: ^ seems wheel builds have been broken on this for a while.  filed https://github.com/pypa/pip/issues/9271 upstream but ^ hopefully gets us back on track03:57
*** zzzeek has quit IRC03:59
*** zzzeek has joined #openstack-infra04:01
*** akahat is now known as akahat|ruck04:11
*** ociuhandu has joined #openstack-infra04:17
*** ociuhandu has quit IRC04:21
*** vishalmanchanda has joined #openstack-infra04:23
*** __ministry1 has quit IRC04:23
*** __ministry1 has joined #openstack-infra04:26
*** ykarel has joined #openstack-infra04:29
*** bnemec has joined #openstack-infra04:41
*** hamalq has joined #openstack-infra04:42
*** hamalq has quit IRC04:46
*** bnemec has quit IRC04:50
*** bnemec has joined #openstack-infra05:07
*** __ministry1 has quit IRC05:19
*** bnemec has quit IRC05:20
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-infra05:33
*** ysandeep is now known as ysandeep|afk05:45
*** ociuhandu has joined #openstack-infra06:05
*** ociuhandu has quit IRC06:09
fungiianw: see my comment on the change, but i think that's an intentional (and necessary) behavior of the new dep solver06:28
openstackgerritMerged openstack/openstack-zuul-jobs master: build-wheel-cache: Ignore missing .whl when removing  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/76687606:35
ianwfungi: i did think as much ... downloading 7 versions to keep the 8th, multiplied by X-thousand CI jobs though ...06:38
ianwi think there's an openafs build issue with centos node publishing too, but that's for tomorrow :006:38
fungiianw: good point, an optimization for pip might be to start with the constrained version (if there's a constraint)?06:40
fungithough maybe we're not using constraints there?06:40
ianwwe are using upper-constraints.  but i couldn't repeat it without the full upper-constraints.txt -- i'm guessing it comes down to transitive dependencies that require the whole graph to figure out06:41
*** hamalq has joined #openstack-infra06:43
*** hamalq has quit IRC06:47
*** zbr has quit IRC06:52
*** zbr has joined #openstack-infra06:54
*** zzzeek has quit IRC06:56
*** zzzeek has joined #openstack-infra06:56
*** ociuhandu has joined #openstack-infra07:01
*** rcernin has quit IRC07:01
*** ociuhandu has quit IRC07:06
*** ysandeep|afk is now known as ysandeep07:09
fungimaybe there are some unconstrained packages in the mix07:17
fungiyou'd think if the constraints represent a coherent set, the dep solver wouldn't need to do anything except verify every reqquirement is being satisfied07:18
*** ralonsoh has joined #openstack-infra07:19
* fungi goes back to trying to get some sleep07:19
*** rcernin has joined #openstack-infra07:27
*** rcernin has quit IRC07:33
*** eolivare has joined #openstack-infra07:41
*** rcernin has joined #openstack-infra07:45
*** hamalq has joined #openstack-infra07:48
*** zzzeek has quit IRC07:50
*** rcernin has quit IRC07:50
*** slaweq has joined #openstack-infra07:51
*** ralonsoh has quit IRC07:51
*** sboyron has joined #openstack-infra07:51
*** zzzeek has joined #openstack-infra07:52
*** hamalq has quit IRC07:53
*** ralonsoh has joined #openstack-infra07:53
*** hashar has joined #openstack-infra07:55
*** piotrowskim has joined #openstack-infra07:58
*** ociuhandu has joined #openstack-infra07:59
*** ociuhandu has quit IRC08:03
*** zzzeek has quit IRC08:06
*** rcernin has joined #openstack-infra08:06
*** zzzeek has joined #openstack-infra08:09
*** jcapitao has joined #openstack-infra08:10
*** gfidente has joined #openstack-infra08:11
*** carloss has quit IRC08:14
*** carloss has joined #openstack-infra08:14
*** dchen has quit IRC08:14
*** psachin has quit IRC08:16
*** elod has quit IRC08:16
*** zer0c00l has quit IRC08:16
*** freerunner has quit IRC08:16
*** tbarron has quit IRC08:16
*** dansmith has quit IRC08:16
*** gnuoy has quit IRC08:16
*** valleedelisle has quit IRC08:16
*** nhicher has quit IRC08:16
*** stephenfin has quit IRC08:16
*** zbr is now known as zbr|ruck08:18
*** zbr|ruck has joined #openstack-infra08:18
*** andrewbonney has joined #openstack-infra08:20
*** zzzeek has quit IRC08:20
*** psachin has joined #openstack-infra08:22
*** elod has joined #openstack-infra08:22
*** zer0c00l has joined #openstack-infra08:22
*** freerunner has joined #openstack-infra08:22
*** tbarron has joined #openstack-infra08:22
*** dansmith has joined #openstack-infra08:22
*** gnuoy has joined #openstack-infra08:22
*** valleedelisle has joined #openstack-infra08:22
*** nhicher has joined #openstack-infra08:22
*** stephenfin has joined #openstack-infra08:22
*** zzzeek has joined #openstack-infra08:22
*** rcernin has quit IRC08:23
*** zbr|ruck is now known as zbr|rover08:23
*** zbr|rover has quit IRC08:24
*** zbr|rover has joined #openstack-infra08:24
*** rpittau|afk is now known as rpittau08:27
*** ociuhandu has joined #openstack-infra08:30
*** hamalq has joined #openstack-infra08:30
*** kmarc has quit IRC08:34
*** ociuhandu has quit IRC08:34
*** rcernin has joined #openstack-infra08:35
*** kmarc has joined #openstack-infra08:35
*** zzzeek has quit IRC08:37
*** tosky has joined #openstack-infra08:38
*** zzzeek has joined #openstack-infra08:41
*** rcernin has quit IRC08:49
*** psachin has quit IRC08:57
*** ociuhandu has joined #openstack-infra08:57
*** ociuhandu has quit IRC08:57
*** ociuhandu has joined #openstack-infra08:58
*** jpena|off is now known as jpena08:58
*** rcernin has joined #openstack-infra08:59
*** ociuhandu has quit IRC09:03
*** dtantsur|afk is now known as dtantsur09:07
*** akahat|ruck is now known as akahat|lunch09:08
*** rcernin has quit IRC09:11
*** derekh has joined #openstack-infra09:17
*** jtomasek has joined #openstack-infra09:20
*** ociuhandu has joined #openstack-infra09:22
*** zzzeek has quit IRC09:28
*** zzzeek has joined #openstack-infra09:31
*** hamalq has quit IRC09:34
*** zzzeek has quit IRC09:51
*** zzzeek has joined #openstack-infra09:53
*** rcernin has joined #openstack-infra09:59
*** zzzeek has quit IRC10:01
*** zzzeek has joined #openstack-infra10:03
openstackgerritDaniel Bengtsson proposed openstack/pbr master: Update the tox minversion parameter.  https://review.opendev.org/c/openstack/pbr/+/76089410:08
*** lseki has joined #openstack-infra10:10
*** rcernin has quit IRC10:12
*** akahat|lunch is now known as akahat|ruck10:32
*** rcernin has joined #openstack-infra10:35
*** rcernin has quit IRC10:41
*** sshnaidm|off has quit IRC10:47
*** ociuhandu_ has joined #openstack-infra11:01
*** ociuhandu has quit IRC11:03
*** mgoddard has quit IRC11:08
*** mgoddard has joined #openstack-infra11:13
noonedeadpunkmornings11:15
noonedeadpunkcan I ask for another vote on https://review.opendev.org/c/openstack/project-config/+/765783 ?:)11:15
*** sshnaidm has joined #openstack-infra11:17
*** rcernin has joined #openstack-infra11:32
*** zzzeek has quit IRC11:35
*** zzzeek has joined #openstack-infra11:36
*** jcapitao is now known as jcapitao_lunch11:41
*** rfolco has joined #openstack-infra12:03
*** ykarel_ has joined #openstack-infra12:14
*** ykarel has quit IRC12:16
*** ykarel_ is now known as ykarel12:17
*** jtomasek has quit IRC12:26
*** ociuhandu_ has quit IRC12:33
*** ociuhandu has joined #openstack-infra12:34
*** jpena is now known as jpena|lunch12:36
*** jcapitao_lunch is now known as jcapitao12:45
*** rlandy has joined #openstack-infra12:59
zbr|roverinfra-root: who can help fixing the broken diskimage-builder ?13:03
zbr|roverianw: tristanC: https://review.opendev.org/c/openstack/diskimage-builder/+/765963 please13:07
openstackgerritAurelien Lourot proposed openstack/project-config master: Mirror charm-magpie to GitHub  https://review.opendev.org/c/openstack/project-config/+/76694813:08
*** ociuhandu has quit IRC13:13
*** stevebaker has quit IRC13:15
*** jtomasek has joined #openstack-infra13:19
*** redrobot has quit IRC13:29
*** jpena|lunch is now known as jpena13:29
*** hamalq has joined #openstack-infra13:31
openstackgerritMerged openstack/project-config master: Add google/wycheproof to pyca Zuul tenant  https://review.opendev.org/c/openstack/project-config/+/76686413:32
*** bnemec has joined #openstack-infra13:32
*** hamalq has quit IRC13:36
*** ociuhandu has joined #openstack-infra13:47
*** dave-mccowan has quit IRC13:55
*** rcernin has quit IRC13:58
*** _erlon_ has joined #openstack-infra14:01
*** ociuhandu has quit IRC14:08
*** ociuhandu has joined #openstack-infra14:08
*** ykarel has quit IRC14:25
*** artom has joined #openstack-infra14:45
*** rlandy is now known as rlandy|bbl14:53
*** jpena is now known as jpena|off14:53
*** jpena|off is now known as jpena14:54
*** ykarel has joined #openstack-infra15:06
*** zxiiro has joined #openstack-infra15:16
*** bnemec has quit IRC15:25
openstackgerritTin Lam proposed openstack/project-config master: Adds airship/charts repo to publish job  https://review.opendev.org/c/openstack/project-config/+/76697315:26
*** bnemec has joined #openstack-infra15:26
*** bnemec has quit IRC15:27
openstackgerritMerged openstack/project-config master: Remove noop jobs for galera_client deprecation  https://review.opendev.org/c/openstack/project-config/+/76578315:28
*** hamalq has joined #openstack-infra15:32
*** thogarre has joined #openstack-infra15:33
*** hamalq has quit IRC15:37
*** dklyle has joined #openstack-infra15:44
*** thogarre has quit IRC15:46
*** zbr|rover2 has joined #openstack-infra16:09
*** Guest5440 has joined #openstack-infra16:09
*** ykarel has quit IRC16:11
*** zbr|rover has quit IRC16:12
*** zbr|rover2 is now known as zbr|rover16:12
*** Guest5440 is now known as redrobot16:12
*** ociuhandu has quit IRC16:13
*** ociuhandu has joined #openstack-infra16:14
*** ociuhandu has quit IRC16:18
*** thogarre has joined #openstack-infra16:19
*** fdegir has joined #openstack-infra16:21
*** bnemec has joined #openstack-infra16:31
*** bnemec has quit IRC16:32
zbr|roverfungi: welcome back. any chance to get a look at diskimage-builder patches?16:34
fungizbr|rover: besides 765963 which i already approved?16:35
*** hashar is now known as hasharDinner16:35
fungioh, wait, i haven't approved it yet looks like ;)16:35
zbr|roverand its dependency, they can go in only together16:36
fungilooks like it also needs 76644716:36
zbr|roveras they need each other.16:36
zbr|roverjoys of multiple fail-points16:37
fungican they be squashed? then they'll both pass16:37
zbr|roversure, not the best practice as they are unrelated. doing now the collapse.16:38
fungiwell, bypassing the ci system is also not the best practice, we frequently introduce new bugs by doing that16:38
*** lbragstad_ has joined #openstack-infra16:38
*** bnemec has joined #openstack-infra16:41
*** lbragstad has quit IRC16:41
zbr|roveri wonder what would be the optimal approach if code inside a repo gets very rotten and it needs like 10 different changes in order to bring CI back to green.16:45
zbr|roverprobably setting jobs NV temporary16:45
fungiin the past we've just put the fixes in a single change and clearly enumerated them in the commit message16:46
*** ociuhandu has joined #openstack-infra16:48
fungizbr|rover: did you mean to leave the depends-on in the commit message?16:49
zbr|roverfungi: thanks, saved me an hour.16:55
funginp16:55
*** ociuhandu has quit IRC17:00
*** ociuhandu has joined #openstack-infra17:02
openstackgerritMerged openstack/project-config master: Re-add os_monasca roles to zuul  https://review.opendev.org/c/openstack/project-config/+/76580617:04
*** lbragstad__ has joined #openstack-infra17:13
*** lbragstad_ has quit IRC17:16
zbr|roverfungi: gerritbot load_safe https://review.opendev.org/c/opendev/gerritbot/+/76516417:18
*** gfidente is now known as gfidente|afk17:19
*** hamalq has joined #openstack-infra17:30
*** hamalq_ has joined #openstack-infra17:35
*** hamalq has quit IRC17:38
*** jtomasek_ has joined #openstack-infra17:41
*** jtomasek_ has quit IRC17:43
weshay|ruckany chance for a force merge of two patches https://review.opendev.org/c/openstack/python-tripleoclient/+/757836 https://review.opendev.org/c/openstack/tripleo-common/+/75782117:43
weshay|ruckmwhahaha, ^17:43
*** jtomasek has quit IRC17:43
fungiweshay|ruck: can you at least explain why you're asking for that? a quick skim of the commit messages and comments doesn't mention any reason they can't go through a normal gating process17:51
weshay|ruckfungi, absolutely.. would you like that in irc or email to opendev-discuss17:51
weshay|ruckeither is fine17:51
*** ociuhandu has quit IRC17:51
*** hasharDinner has quit IRC17:52
weshay|ruckwill start here...  I believe these two pathes are last remaining patches that migrate tripleo jobs off of docker.io and use parent jobs that build containers .. reference https://hackmd.io/YH9xtBmOQbSVqPu7fjgoog?view17:53
weshay|ruckwe have train -> ussuri upgrades that getting into a weird state because the containers that are pulled in from train have c8.3 base os rpms and ussuri has c8.2 base os rpms17:53
weshay|ruckwe've been trying to land these patches for weeks.. and keep coming up short w/ timeouts, various unrelated bugs.. etc17:54
fungiwhat i'm trying to get at is, are those two changes blocking each other? or do they not sufficiently fix your jobs?17:54
weshay|ruckthey just align these two repos to do what the rest of the tripleo repos and branches are doing w/ regards to building containers in a parent17:54
weshay|ruckthey fix our trian-> ussuri upgrade jobs..17:55
fungii see, but unrelated jobs you're also running for those changes are insufficiently stable to pass?17:55
weshay|ruckwe're still refreshing docker.io containers.. so it's not like we're blood red.. because of it17:55
weshay|ruckbut dang.. it's just been really tough landing these two17:56
*** eolivare has quit IRC17:56
*** rpittau is now known as rpittau|afk17:56
weshay|ruckfungi, well.. there have been at least 6 seperate issues that blocked them before.. but atm.. it seems to be timeouts17:56
weshay|rucktripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades-ussuri is one culprit17:57
funginetwork timeouts, test timeouts, job/ansible duration timeouts?17:57
weshay|ruckjob17:57
fungigot it, so there are jobs which often can't complete within the configured timeout17:57
weshay|ruckya. we track that... it's been a rough last 8-10 days.. but for the most part we're able to make these jobs behave17:58
fungiwhen they do complete, are they usually running near the timeout? or is the job duration widely varying?17:58
weshay|ruckyes.. but we track it17:58
weshay|ruckI raise it when they go up into the danger zone17:58
weshay|ruckand also try to refresh our rpms to speed it up17:59
fungihave you considered temporarily disabling the less stable jobs and freezing the codebase for anything other than job stability fixes until it's back under control?18:00
*** ociuhandu has joined #openstack-infra18:01
weshay|ruckfungi, yes.. disabling the upgrade jobs is very painful but yes considered18:05
*** ociuhandu has quit IRC18:05
*** ysandeep is now known as ysandeep|away18:07
fungiweshay|ruck: i guess the rdo ci failures on those are expected?18:08
weshay|ruckfungi, vexx was down for a number of days and our public cloud was down for an upgrade.. crap got in.. /me gets the bug18:08
weshay|ruckatm known issues.. landing those patches as well18:09
weshay|ruckwe're in sad shape atm.. tbh18:09
weshay|ruckcomes and goes18:09
fungihow about i reenqueue those into the gate pipeline and promote them and see if we can at least get zuul to merge them?18:10
weshay|ruckhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/766797 this fixed master, should fix victoria and I think we have the same issue in ussuri18:10
fungisounds like these are the two top-priority fixes for that queue?18:10
weshay|ruckfungi, that would be great!!18:10
weshay|ruckyes18:10
weshay|ruckfungi, thanks for hearing me out :)18:10
weshay|ruckmwhahaha, fyi ^18:10
*** jpena is now known as jpena|off18:11
mwhahahak18:11
fungiis one of those two more important than the other, or both roughly equal priority?18:11
weshay|ruckimho.. equal18:11
weshay|ruckdoes the same thing.. just two diff repos18:12
*** derekh has quit IRC18:13
fungi#status log enqueued tripleo job stability fixes 757836,22 and 757821,16 into the gate and promoted them both to the front of the queue at weshay'a request18:16
openstackstatusfungi: finished logging18:16
weshay|ruckthank you!18:17
fungii'll try to keep an eye on those and shuffle them if they start to fail so we keep them at the top until they merge18:17
weshay|ruckrock on!18:17
weshay|ruckfungi++18:17
fungiideally these also reduce the incidence of failure for everything running behind them in your gate queue, saving us resources in the long run18:18
weshay|ruckbtw.. an added benefit of these jobs besides not relying on docker.io is that the baseos updates will not impact containers and baseos where those rpms need to be at the exact same version18:18
*** eharney has quit IRC18:19
*** lbragstad__ has quit IRC18:24
*** piotrowskim has quit IRC18:30
*** eharney has joined #openstack-infra18:31
*** dtantsur is now known as dtantsur|afk18:32
*** rlandy|bbl is now known as rlandy18:33
*** jcapitao has quit IRC18:39
*** eharney has quit IRC18:41
*** eharney has joined #openstack-infra18:41
*** lbragstad__ has joined #openstack-infra18:42
*** openstackgerrit has quit IRC18:50
*** gmann is now known as gmann_lunch18:53
*** gyee has joined #openstack-infra18:58
*** hashar has joined #openstack-infra19:16
*** stevebaker has joined #openstack-infra19:21
* weshay|ruck frowns... perhaps we need https://review.opendev.org/c/openstack/tripleo-common/+/765077 too19:27
weshay|ruckdang it19:27
fungiweshay|ruck: should 765077 go in first then?19:29
fungiit's not been approved though19:30
weshay|ruckfungi, ya.. yet another new thing19:52
weshay|ruckfungi, sorry man.. I thought this was fairly solid19:52
weshay|ruckapparently my mistake19:53
*** tbachman_ has joined #openstack-infra19:54
fungiweshay|ruck: if someone approves that i can bump it ahead of the other two19:54
weshay|ruckmwhahaha, ^ wdyt19:54
fungilooks like it passed zuul testing 10 days ago, but was then rebased a few hours ago and has no new check results yet19:55
mwhahahadone?19:56
fungii'll put it up front and we'll see how that goes19:56
*** tbachman has quit IRC19:56
*** tbachman_ is now known as tbachman19:56
fungiahh, looks like it can't go first because 757821,16 is its parent19:58
fungido those need to be reordered in git?19:58
fungi(or squashed together?)19:58
fungiright now the tripleo gate queue starts with 757821,16 765077,3 757836,2219:59
weshay|ruckI'll +2 but I think I need mwhahaha there as well20:02
weshay|ruckah we got it20:02
*** yonglihe has quit IRC20:03
*** viks____ has quit IRC20:03
*** arne_wiebalck has quit IRC20:03
*** yonglihe has joined #openstack-infra20:03
*** arne_wiebalck has joined #openstack-infra20:03
fungiyeah, it was one of the other two we had promoted previously20:04
*** gmann_lunch is now known as gmann20:08
*** vishalmanchanda has quit IRC20:12
*** andrewbonney has quit IRC20:17
*** lbragstad__ has quit IRC20:19
*** lbragstad has joined #openstack-infra20:22
*** rajinir has joined #openstack-infra20:25
*** tomaw has quit IRC20:47
*** tomaw_ has joined #openstack-infra20:47
*** tomaw_ is now known as tomaw20:48
*** gfidente|afk has quit IRC20:53
*** raissa has joined #openstack-infra20:59
*** raissa has quit IRC20:59
*** raissa has joined #openstack-infra21:00
*** raissa has quit IRC21:00
*** ociuhandu has joined #openstack-infra21:00
*** raissa has joined #openstack-infra21:03
*** ralonsoh has quit IRC21:04
*** raissa has joined #openstack-infra21:04
*** raissa has quit IRC21:04
*** raissa has joined #openstack-infra21:05
*** ociuhandu has quit IRC21:05
*** raissa has quit IRC21:05
*** hashar has quit IRC21:05
*** sboyron has quit IRC21:10
*** raissa has joined #openstack-infra21:11
*** raissa has quit IRC21:11
*** rcernin has joined #openstack-infra21:30
*** ralonsoh has joined #openstack-infra21:50
fungiweshay|ruck: looks like the commit message on 757836 got updated so it was kicked out of the gate and is back in check on a new patchset now. the other two are still looking good at the top of the gate queue though22:45
*** openstackgerrit has joined #openstack-infra23:35
openstackgerritKendall Nelson proposed openstack/project-config master: End project gating for Karbor  https://review.opendev.org/c/openstack/project-config/+/76703023:35
*** dchen has joined #openstack-infra23:40
*** armax has quit IRC23:57
*** armax has joined #openstack-infra23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!