Thursday, 2020-07-02

*** prometheanfire has quit IRC00:20
*** prometheanfire has joined #opendev00:23
*** ryohayakawa has joined #opendev00:36
*** ryo_hayakawa has joined #opendev00:59
*** ryohayakawa has quit IRC01:01
*** shtepanie has quit IRC02:09
*** mugsie has quit IRC02:59
*** mugsie has joined #opendev03:02
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895203:47
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895203:52
openstackgerritIan Wienand proposed opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812504:25
openstackgerritMerged openstack/project-config master: Use python3 for update_constraints  https://review.opendev.org/73893104:36
*** ysandeep|away is now known as ysandeep04:37
*** ykarel|away is now known as ykarel04:49
*** diablo_rojo has quit IRC04:50
*** bhagyashris|afk is now known as bhagyashris05:11
*** priteau has joined #opendev05:15
openstackgerritIan Wienand proposed opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812505:40
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895205:43
openstackgerritIan Wienand proposed opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812505:51
openstackgerritMerged openstack/project-config master: Add Vault app to StarlingX  https://review.opendev.org/73851305:52
openstackgerritMerged openstack/project-config master: Create a new repository for Tobiko DevStack plugin  https://review.opendev.org/73837805:52
openstackgerritMerged openstack/project-config master: Upadting horizon nodejs job name  https://review.opendev.org/73745705:53
openstackgerritMerged openstack/project-config master: Create a new project for ansible-tripleo-ipa-server  https://review.opendev.org/73884205:53
openstackgerritIan Wienand proposed zuul/zuul-jobs master: emit-job-header: add inventory hostname  https://review.opendev.org/73896306:01
*** sshnaidm|afk is now known as sshnaidm|ruck06:03
openstackgerritIan Wienand proposed opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812506:12
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895206:26
*** DSpider has joined #opendev06:30
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895206:37
*** hashar has joined #opendev06:39
openstackgerritIan Wienand proposed opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812506:41
openstackgerritMerged opendev/system-config master: Add copr-lxc3 to list of mirrors  https://review.opendev.org/73894206:43
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895206:54
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895207:16
*** bhagyashris is now known as bhagyashris|lunc07:28
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895207:29
*** tosky has joined #opendev07:46
*** dtantsur|afk is now known as dtantsur07:47
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** ysandeep is now known as ysandeep|lunch08:02
openstackgerritFederico Ressi proposed openstack/project-config master: Fix x/devstack-plugin-tobiko name  https://review.opendev.org/73897908:03
*** bhagyashris|lunc is now known as bhagyashris08:50
AJaegerinfra-root, a typo in the repo name - I fear it's too late now and we need to rename it, correct? ^08:54
AJaegerIt's not created yet in git08:55
*** ysandeep|lunch is now known as ysandeep09:01
*** jaicaa has quit IRC09:05
*** jaicaa has joined #opendev09:07
*** tkajinam has quit IRC09:09
ttxquick update on the mirroring job with retry -- it no longer fails. Not sure we've hit a condition where it retried yet. Will continue watching it for a few days09:13
*** hashar has quit IRC09:16
*** owalsh_ is now known as owalsh09:50
*** priteau has quit IRC09:58
lourotttx, hey o/ regarding the chat we had a few days ago about mirroring a new official openstack repo to github, we just merged https://review.opendev.org/738573 and were hoping it would trigger the mirroring to https://github.com/openstack/charm-neutron-api-plugin-arista . Unfortunately that didn't happen. Do you know what bit might be missing? Thanks!10:02
ttxlooking10:03
ttxthe job did not even try to run apparently, digging10:04
ttxok found the issue10:05
ttxhttps://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L188510:06
ttxThe project is missing the official-openstack-repo-jobs template10:06
ttxwhich contains the mirroring job10:07
lourotttx, alright, I'll extend that, thanks!10:07
ttxopenstack/charm-neutron-api-plugin-arista needs an entry in that file, like openstack/charm-neutron-api-plugin-ovn has10:07
ttxAJaeger: is that something we should check when the project is created?10:08
AJaegerlourot: this needs to be added to project-config repo10:11
lourotAJaeger, yes thanks, sorry to have missed that bit. I'll also add our recent trilio charms in there which are missing as well and make sure I get a +1 from James Page. Thanks for the quick help!10:12
AJaegerttx, this is circling back to a previous discussion we had: We need to document how the workflow works with creation of repos and addingn of the template. I saw your proposed change for creation of repos - and if we can get that in, the addition would be a second step (can only be done after repo creation).10:12
AJaegerlourot: this is a new step, not all details are clear yet...10:13
ttxmy proposed change for creation of repos ? You mean the github repo management job?10:13
AJaegerttx, so we cannot check when the repo is created, Zuul does not allow an addtion to zuul.d before the repo is created10:13
AJaegerttx, yes10:14
AJaegerttx, btw. let me comment on that one...10:14
ttxIt's an early draft to check if that's the way to go :)10:14
AJaegerttx, left my comments finally10:15
AJaegerttx, so I don't know how to handle new repos in openstack namespace today - it needs at some time a change for the template. But should this be a direct followup, waiting for github repo creation to be done? Let'S figure that out and document, please10:18
openstackgerritFederico Ressi proposed openstack/project-config master: Fix x/devstack-plugin-tobiko name  https://review.opendev.org/73897910:19
openstackgerritFederico Ressi proposed openstack/project-config master: Re-create x/devstack-plugin-tobiko project  https://review.opendev.org/73897910:22
openstackgerritFederico Ressi proposed openstack/project-config master: Revert "Create a new repository for Tobiko DevStack plugin"  https://review.opendev.org/73900610:22
openstackgerritFederico Ressi proposed openstack/project-config master: Revert "Create a new repository for Tobiko DevStack plugin"  https://review.opendev.org/73900610:22
* AJaeger asked him to join us here for discussion ^10:30
openstackgerritAurelien Lourot proposed openstack/project-config master: Mirror arista and trilio charms to GitHub  https://review.opendev.org/73900910:30
AJaegerttx, please review ^10:48
*** ryo_hayakawa has quit IRC10:50
*** rajinir has quit IRC11:00
*** hashar has joined #opendev11:34
*** osmanlicilegi has quit IRC11:50
*** ysandeep is now known as ysandeep|afk12:01
*** osmanlicilegi has joined #opendev12:12
openstackgerritMerged openstack/project-config master: Mirror arista and trilio charms to GitHub  https://review.opendev.org/73900912:15
lourotAJaeger, ttx, thanks! so I guess now with the next commit it will trigger the mirroring. I'll give it a try12:19
*** ysandeep|afk is now known as ysandeep12:33
*** fressi has joined #opendev12:42
AJaegerlourot: yes, the next merge will trigger it.12:46
lourotnice, gate jobs are running as we speak12:48
lourotit worked, thanks a lot!12:59
AJaegergreat!13:15
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify  https://review.opendev.org/73904713:55
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify  https://review.opendev.org/73904714:07
openstackgerritMarcin Juszkiewicz proposed opendev/system-config master: UCA: mirror Victoria packages for Focal  https://review.opendev.org/73905114:11
hrwtime to mirror another release ;D14:12
*** bhagyashris is now known as bhagyashris|afk14:15
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify  https://review.opendev.org/73904714:16
*** ysandeep is now known as ysandeep|away14:43
clarkbAJaeger: is there a quick update on the project rename situation?14:43
clarkbIt looks like the project was created in both gitea and gerrit14:44
clarkbthe issue is topiko needs to be tobiko ?14:44
clarkbif so then yes that will need a rename14:44
openstackgerritAlex Schultz proposed zuul/zuul-jobs master: Make persist-iptables more robust  https://review.opendev.org/73906114:45
*** priteau has joined #opendev14:51
*** ysandeep|away is now known as ysandeep14:53
AJaegerclarkb: exactly, p->b. I asked the committer to discuss here but also commented that he needs to do the rename - but he seemed to not understand me ;(14:53
*** mlavalle has joined #opendev14:56
*** sshnaidm|ruck is now known as sshnaidm|mtg14:57
fungithey can still proceed with using the original repo, and then all their work will move to the new repo name when we have an opportunity to take the downtime for it14:58
clarkbfungi: that is a good point14:58
fungiso it's hopefullt only a minor inconvenience14:58
clarkbfungi: since there aren't any major fires this morning I'm thinking I'll hit the etherpad upgrade button after breakfast. Any reason you can think of to delay that further?14:59
clarkbmaybe we want to do another pass of testing it? I'm not sure how many others got a chance to use the zuul test node to test14:59
fungiclarkb: unless they've tagged a new release, i say go for it15:00
clarkbgood question /me checks15:00
fungii'm hoping to find time after/between meetings to focus on my todo backlog15:00
clarkbhttps://github.com/ether/etherpad-lite/tags still 1.8.4 as latest15:01
*** bolg has quit IRC15:01
fungicool, should be green then15:01
clarkbI think my gitea repos listing fix just landed15:06
clarkbwe should also update to v1.12.1 (though the bug fixes don't look super relevant to us)15:07
*** dpawlik2 has joined #opendev15:12
*** yoctozepto2 has joined #opendev15:13
*** JasonF has joined #opendev15:13
*** jaicaa_ has joined #opendev15:13
*** jaicaa has quit IRC15:14
*** redrobot has quit IRC15:14
*** mnasiadka has quit IRC15:14
*** avass has quit IRC15:14
*** mnasiadka_ has joined #opendev15:14
clarkbI'm cleaning up holds again. mnaser youhave two holds. One for minikube update debugging and another for obscure go archtecture errors. Can they be cleaned up?15:14
*** dpawlik4 has quit IRC15:14
*** JayF has quit IRC15:14
*** JasonF is now known as JayF15:14
mnaserclarkb: yes, apologies for not reaching out15:15
*** yoctozepto has quit IRC15:15
*** yoctozepto2 is now known as yoctozepto15:15
*** elod has quit IRC15:15
clarkbmnaser: thanks I'll get thos ecleaned up now.15:15
*** elod has joined #opendev15:16
mnaserclarkb: perhaps we can look at adding default_hold_expiration inside our zuul to be something liek 1-2 weeks15:16
mnaseri cant imagine a need for something _that_ long to be held15:16
clarkbya, can we update that after the fact if we want it longer?15:17
clarkbmy concern is we'll be caught be surprised by that if we se tit15:17
*** ykarel is now known as ykarel|away15:17
openstackgerritFederico Ressi proposed openstack/project-config master: Revert "Create a new repository for Tobiko DevStack plugin"  https://review.opendev.org/73900615:25
openstackgerritFederico Ressi proposed openstack/project-config master: Re-create x/devstack-plugin-tobiko project  https://review.opendev.org/73897915:25
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add node-labeler to the vexxhost tenant  https://review.opendev.org/73907115:26
*** ysandeep is now known as ysandeep|away15:26
fressix/devstack-plugin-tobiko has been created with the wrong name (x/devstack-plugin-topiko) [https://review.opendev.org/#/q/c7ca7592a98b9b9e229f3e73521f5b12a4945a2e]. Because the actual content of the repo is still irrelevant (no commits done yet), the rename operation is not required. We could instead create a new one (with the right name) and delete15:30
fressithe old one. These two operations can be performed in any order.15:30
clarkbfressi: there is no deleting15:30
clarkbunfortunately, the easiest option here is to use the repo as is for now and we'll rename when we can15:30
clarkb(I think if jeepyb didn't create the .gitreview commit deletion may actually have been possible. That may be something to think about changing?)15:31
fressiCan't we simply ingnore wrong named repo and just forbid commits to it? An empty repo should not harm.15:31
clarkbThat is also an option, though one we've strongly avoided in the past as it creates technical debt you can never remove15:32
fungiwe'd rather not end up with a system full of empty repositories left behind due to naming mistakes15:33
fressiOK so the CANONICAL way is using it as it is while waiting for a rename operation? This would mean references to this repo should be updated later.15:33
fungifressi: correct, we can probably schedule a maintenance window to take the downtime we need for renaming it fairly soon15:34
fressiThank you very much! Then I am going on with such way. Have a nice day you all.15:41
openstackgerritAlex Schultz proposed zuul/zuul-jobs master: Make persist-iptables more robust  https://review.opendev.org/73906115:43
clarkbI'd say we could do next week, but I'm out next week so don't want to rely on that :) Tomorrow is potentially an option but also a holiday for several of us and I know some of us intend to treat it as a holiday15:43
clarkbbut maybe in ~10 days we can do a quick rename?15:43
clarkbI'll update the wiki shortly15:43
fressiOK 10 days is a really rezonable time! Thank you15:44
fungiyes, i'll neither be around tomorrow nor monday, but i could do a week from tomorrow (july 10) even though clarkb is out15:46
fungior maybe monday the 13th15:47
clarkbfungi: cool I'll put it on the wiki meeting agenda and if people are able to do it next week great and if not I can help week after15:47
mnasercould we briefly/quickly get https://review.opendev.org/739071 merged so we can move it off github? :> (we'll be squash-importing)15:49
corvusdone15:50
openstackgerritMerged openstack/project-config master: Add node-labeler to the vexxhost tenant  https://review.opendev.org/73907116:03
clarkbwiki has been updated16:09
fungithe crawler noise requests to gitea continue16:09
clarkbhttps://review.opendev.org/#/c/729029/3 has been approved to upgrade etherpad16:10
clarkbfungi: ya maybe if it persists at beginning of next week we roll out ianw's apache filtering and undo the iptables rules?16:10
*** factor has joined #opendev16:21
*** icarusfactor has quit IRC16:21
fungiyeah, i don't like complicating our gitea deployment, but i also don't like blocking every customer of the largest isp in china either16:24
*** factor has quit IRC16:28
*** dtantsur is now known as dtantsur|afk16:28
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: WIP fetch-coverage-output: direct link to coverage data  https://review.opendev.org/73909916:36
mwhahahacorvus: when we turn on verbose, we should recheck https://review.opendev.org/#/c/739061/ to see if it'll trigger it17:11
corvusinfra-root: ^ i'm going to do that to try to catch more info about this weird -13 error17:12
clarkbcorvus: rgr17:13
*** avass has joined #opendev17:14
corvusverbose is set; i rechecked 73906117:14
*** mlavalle has quit IRC17:15
clarkbsystem-config-run-base failed on the etherpad change due to a timeout. I'm going to take a break then will recheck it when back17:18
*** bolg has joined #opendev17:27
*** olaph has joined #opendev17:28
corvusall the jobs have started for that change, so i have issued the unverbose command17:36
*** bolg has quit IRC17:53
mwhahahacorvus: doesn't look like it triggered on any of those17:55
mwhahahacorvus: maybe another job hit it in the same timeframe?17:55
JayFFYI: It appears "search in repository" is broken for opendev.org/openstack/ironic-python-agent ... maybe others as well. "heartbeat" shows up in as text in agent.py but is not in results. So I tried "agent" in case it only searched on filenames, and I don't see agent.py.18:00
JayFI'm not blocked on this or anything, it just seemed completely broken so I thought I'd give a heads up18:01
fungiJayF: thanks, and yeah gitea search still needs some help. in the meantime you can use the advanced search options on codesearch.openstack.org to limit searching to a single repo or specific set of repos18:03
fungithat's why we haven't killed the codesearch service yet18:03
JayF+1 makes sense, it's fine that if it's broken, just wanted to make sure you all were aware18:05
clarkbI've rechecked the etherpad change18:05
corvusmwhahaha: i'll try to look for one18:15
mwhahahacorvus: in theory if it's that function, we should have a 'Error Executing CMD' somewhere18:16
corvusmwhahaha: no joy18:19
mwhahaha:(18:20
mwhahahastory of my week18:20
corvuswe can try again18:20
mwhahahasure we can give it another shot18:20
corvusverbose is set again globally18:20
mwhahahais logstash broken/delayed?18:23
clarkbe-r says we're behind18:24
mwhahahafun18:25
mwhahahai also found a MODULE FAILURE in gather facts18:27
mwhahahaw/ rc -1318:27
mwhahahaso i think it's a core ansible problem as it relates to the execution18:27
mwhahahalogstash seems to say this started on 6/2618:28
clarkbmwhahaha: our data only goes back about that far18:29
mwhahahaaww18:29
clarkbthe volume of input is too large to have a wider window18:29
mwhahahai thought i had results from older than that the other day18:30
clarkb(its already a 6TB database)18:30
mwhahahaoh well18:30
corvusmwhahaha: you probably did the other day :)18:30
clarkbya we expire daily to drop the oldest set18:30
mwhahahai meant > 14 days :D anyway18:30
corvusmwhahaha: so if we're looking for other -13s -- we did snag one during that window, let me get logs18:30
mwhahahayea MODULE FAILURE seems to only be this issue18:31
mwhahahaat least according to logstash18:31
corvusmwhahaha: http://paste.openstack.org/show/795508/18:32
corvusmwhahaha: wow, i don't see any more info there :(18:32
mwhahaha:/18:32
mwhahahait kinda makes sense if it's the execution function. the spot i think it is with the fnctl is specifically to get the stderr/stdout which would be empty :/ it's lame that it doesn't log anything else elsewhere however18:34
mwhahahaguess time to try and reproduce elsewhere18:35
mwhahahawonder if it's some executor scale limit somewhere18:35
corvusmwhahaha: yeah, that's starting to occur to me too.  they are busy, and we're down 1 from normal.18:37
corvusi have issued unverbose; all the jobs are running18:38
corvusi have to afk for a bit; back after lunch18:38
*** fressi has quit IRC19:09
*** fressi has joined #opendev19:10
*** mlavalle has joined #opendev19:25
openstackgerritMerged opendev/system-config master: Upgrade Etherpad to 1.8.4  https://review.opendev.org/72902919:50
clarkbI'm watching ^19:51
*** hashar has quit IRC19:58
clarkbthe job to deploy it is still running20:07
clarkbbut etherpad has upgraded20:07
clarkbIt seems to work on a random etherpad I've pulled up20:08
fungiexcellent, testing from here20:09
clarkbfungi: have an etherpad I should pull up and type into?20:10
clarkb(also fwiw if we hvae to I think we can revert to 1.8.0 as that was tested on the test node several times back and forth and it seemed to work ok there)20:12
clarkb#status log Upgraded etherpad-lite to 1.8.4 on etherpad.opendev.org20:15
openstackstatusclarkb: finished logging20:15
fungithe release management pad is rather large, and looks correct: https://etherpad.opendev.org/p/victoria-relmgt-tracking20:22
fungiour ptg pad looks okay too: https://etherpad.opendev.org/p/opendev-virtual-ptg-june-202020:23
fungidefinitely not seeing author color overlap20:24
clarkbya my css hack seems to be working20:24
fungiagreed20:24
fungihttps://etherpad.opendev.org/p/clarkb-test is still perpetually "loading..."20:25
fungiso i guess the fix i was thinking of didn't address that one20:26
clarkbya I think once the null is miswritten to a pad it may be a goner20:26
clarkbbut hopefully we don't get new pads in that state20:26
fungii was hoping it was a client-side rendering fix not a client-side write fix, but no dice20:27
clarkbah ya I think it has to do with bad writes to the db making it sad, they solved the bad writes so new pads dont' have that happen20:27
*** sshnaidm|mtg is now known as sshnaidm|afk20:27
mordredclarkb: woot re: etherpad!20:36
*** priteau has quit IRC20:44
*** priteau has joined #opendev20:52
*** priteau has quit IRC20:52
ianwcorvus / clarkb: if you could loop back on the graphite change @ https://review.opendev.org/#/c/738125/ ; i re-did it for host networking by overwriting the config files21:12
ianwi'll try and get better ipv6 support and ssl support upstreamed based on that, to avoid us having a fork21:13
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: fetch-coverage-output: direct link to coverage data  https://review.opendev.org/73909921:13
corvusianw: looks great, and i think that's a fine solution21:17
clarkbianw: that looks good. I did leave a minor suggestion on the change but its just to testing21:17
openstackgerritIan Wienand proposed opendev/system-config master: Grafana container deployment  https://review.opendev.org/73740621:17
openstackgerritIan Wienand proposed opendev/system-config master: Graphite container deployment  https://review.opendev.org/73812521:17
ianwclarkb: ^ sorry, that just fixes the grafana one under it to remove that extra dir create21:18
ianwas mentioned, the db for grafana is in the container -- we don't need to persist it since we load from our config files21:18
clarkb+2 on that one now too21:20
ianwthanks, i should be able to deploy them now in a testing phase21:21
openstackgerritIan Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined  https://review.opendev.org/73895221:26
ianwanother related thing is https://review.opendev.org/#/c/737900/ which loads json files directly with grafyaml21:29
ianwthat will make what's in the documentation about being able to edit using the UI and save the json to project-config true21:29
clarkbthats actually pretty simple21:36
clarkbI guess the risk there is that grafana would change formats then old json would break and we'd not know how to recreate it necessarily, but thats true of grafana's yaml -> json translation anyway and would have to be debugged21:36
*** bolg has joined #opendev21:36
ianwyeah; i haven't read it anywhere, but i feel like they try to keep it pretty stable21:37
openstackgerritJeremy Stanley proposed opendev/system-config master: Forward openstack-infra ML to openstack-discuss  https://review.opendev.org/73915221:37
*** bolg has quit IRC21:45
fungistrange, here you can see the crawler botnet turned off for an hour 16:00-17:00z: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66611&rra_id=all22:09
clarkbfungi: its blipped a couple of times but I don't know if that long before22:10
*** bolg has joined #opendev22:15
*** tosky has quit IRC22:54
*** tkajinam has joined #opendev22:55
*** mlavalle has quit IRC23:01
*** DSpider has quit IRC23:19
*** bolg has quit IRC23:32
*** ryohayakawa has joined #opendev23:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!