Saturday, 2020-04-04

« Friday, 2020-04-03 Index Sunday, 2020-04-05 »
*** prometheanfire has joined #opendev00:35
*** rm_work has quit IRC00:42
*** rm_work has joined #opendev00:45
*** jrosser has quit IRC00:45
*** jrosser has joined #opendev00:47
*** mlavalle has quit IRC01:19
*** factor has quit IRC04:10
*** factor has joined #opendev04:10
*** DSpider has joined #opendev06:23
*** diablo_rojo has quit IRC06:35
*** sgw has quit IRC06:44
*** dpawlik has joined #opendev08:07
*** dpawlik has quit IRC08:12
*** tosky has joined #opendev10:48
AJaegerinfra-root, could you review open reviews for hacking repos, please?12:17
fungiwhich "hacking repos"?14:01
AJaegerI meant repo - openstack/hacking14:23
yoctozeptohacking repos :D14:23
mordredAJaeger: all lgtm14:24
AJaegermordred: also https://review.opendev.org/678607 ? Thanks!14:26
yoctozeptobtw, don't you feel hacking is more qa than infra? ;-)14:26
fungiinteresting, it's a qa project, i didn't realize infra-core was included in the review group for it14:26
AJaegerfungi, yoctozepto, it has only qa-release in it, doesn't it?14:26
yoctozeptoAJaeger: hmm, confirmed14:26
AJaegerhttps://review.opendev.org/#/admin/groups/153,members14:27
AJaegermaybe something for clarkb and gmann to discuss...14:27
fungiit's a qa team deliverable at least: https://governance.openstack.org/tc/reference/projects/quality-assurance.html14:27
AJaegerlet me tell gmann, he's not on this channel...14:28
fungiand i don't think we use it for any infra/opendev projects14:28
yoctozeptothere could be more like this14:28
mordredI believe we're in the core team because it's one of those things that has sometimes needed a patch to unbreak the gate14:28
mordredso it's mostly one of those "insurance policy" core memberships14:28
AJaegerAh, understood14:28
yoctozeptowell, the thought has not propagated to e.g. bashate https://review.opendev.org/#/admin/groups/333,members14:30
mordredit might have something to do with it having been me that originally made hacking its own tool - so it might just be pure history14:31
AJaeger;)14:36
yoctozeptolessons of history :-)14:42
AJaegernearly 10 years - lots of history :)14:46
mordredso much history14:47
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-yarn: always install  https://review.opendev.org/71737514:52
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644215:02
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-yarn: always install  https://review.opendev.org/71737515:03
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-nodejs: test on all platforms  https://review.opendev.org/71747015:03
mordred#status log added an acme cname for etherpad01.opendev.org15:10
* fungi will check on openstackstatus if you aren't already15:10
fungiit's in channel15:10
fungi2020-04-04 15:10:08,770 DEBUG statusbot.bot: Ignoring message from unauthenticated user mordred15:15
fungiexcept freenode says you're logged in15:15
fungimordred: try again? if it still doesn't work i'll try restarting the bot15:16
openstackgerritFilip Kaliński proposed opendev/git-review master: Handle branch names from Gerrit plugin for Idea  https://review.opendev.org/71748115:17
mordredfungi: oh! well poop15:17
*** mordred has quit IRC15:20
*** mordred has joined #opendev15:21
mordred#status log added an acme cname for etherpad01.opendev.org15:21
openstackstatusmordred: finished logging15:21
fungithat seems to have worked15:21
mordredwoot15:21
mordred(I'm set up for SASL auth on login - I'm not sure why I wasn't logged in)15:21
fungiokay, so bot's not thoroughly broken, maybe a freenode problem15:21
fungimordred: well, freenode claimed you were logged in when i tried /whois earlier15:22
fungiso i'm not sure what was going on there15:22
mordred_weird_15:22
fungiindeed15:22
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: tox-parse_output: Ignore .tox directory  https://review.opendev.org/71748615:26
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644215:29
openstackgerritFilip Kaliński proposed opendev/git-review master: Handle branch names from Gerrit plugin for Idea  https://review.opendev.org/71748115:36
openstackgerritMonty Taylor proposed opendev/zone-opendev.org master: Add dns entries for new etherpad01.opendev.org  https://review.opendev.org/71749115:48
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644215:50
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644215:55
mordred#status log launched etherpad01.opendev.org server15:56
openstackstatusmordred: finished logging15:56
mordredinfra-root: ok - I think that ^^ should do it for new etherpad - and should appropriately run the etherpad service playbook when we land the change :)15:56
corvusi just visited https://meetpad.opendev.org/meetpadtest2 from 3 browsers and they all worked15:57
mordredcorvus: woot!15:57
corvusmordred: yeah, except we didn't change anything? :)15:57
mordredcorvus: boo15:57
fungimaybe meetpad likes saturdays?15:57
corvusmaybe something was cached from before the proxying?15:59
corvusoh, but meetpadtest3 is not opening the etherpad16:00
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644216:00
corvusit's pretty weird; sometimes the etherpad doesn't open in chromium, but i can't find a pattern to it16:07
corvuson the occasions when it doesn't open the etherpad, exiting chromium and restarting helps16:08
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644216:10
openstackgerritMonty Taylor proposed opendev/system-config master: Remove old etherpad.openstack.org  https://review.opendev.org/71749216:10
corvusi just ran "apt-get install profanity" on the meetpad server16:11
fungi"console based XMPP client"16:17
fungineat16:17
corvusi'm hoping i can join the room and watch the control messages, but i think i need to figure out the ssl cert situation first16:18
AJaegerwow, logstash queue is still growing - now at 254k16:24
mordredcorvus: want me to rebase the new etherpad server change above so that we can land it and have a etherpad with LE certs to work with?16:25
mordred(it's at the end of the stack right now, but doesn't have to be)16:26
corvusmordred: yeah, could come in handy :)16:26
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644216:27
mordredcorvus: k. that should be landable (assuming it passes tests)16:27
mordredand https://review.opendev.org/717491 is also landable16:27
mordredfungi: ^^16:27
corvusi'm trying poezio now, which looks like it has the necessary cert support, however, i can not figure out how to tell it to connect to a server.16:29
corvusthat feels like something that should be in the docs.16:29
corvusoooh it's only in the config file16:30
openstackgerritSean McGinnis proposed openstack/project-config master: Add tooling to update python jobs on branch creation  https://review.opendev.org/67301916:43
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750316:47
mnaserhmm16:48
mnaserinfra-root: zuul is not responding right now16:48
mnaserhttp://zuul.opendev.org/api/info is not responding16:49
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused gerrit puppet things  https://review.opendev.org/71400116:49
mordredmnaser: I agree16:49
corvusrecent memory use looks ok16:50
mordredscheduler and web are running16:50
mnaserfwiw, i don't know how related it is, but i pushed a change to zuul/zuul-jobs which touched a lot of files and likely triggered a lot of jobs16:50
mnaserbecause we have N platforms and it touches many roles so probably easily ended up with (maybe?) some 40 jobs16:50
mnasercontext: https://review.opendev.org/#/c/717503/ but that may just be noise16:51
corvusthat shouldn't be a big deal16:51
corvus(that's happened successfully before)16:51
corvusthere was a gearman disconnect16:51
mordredyeah- that gearman log doesn't look happy16:53
corvuswe used to have a netcat command in our docs to connect to gearman?16:53
corvusi can't find it16:54
mordredme either16:54
corvusor i gues maybe an 'openssl s_client' ?16:54
mnaserfwiw it is responding again now (teh api)16:55
corvuswell neat, that's quite a treat16:55
corvusit self-recovered before we could figure out how to debug it16:56
mordredyeah. so excited about that16:56
corvusi don't know the underlying cause, but i do know the scheduler lost its connection with gearman.  that means it's going to abort all of the jobs and restart them.  it's possible it was unresponsive because of some combination of being very busy with that, or the web server was in a connection backoff from gearman16:57
mordredcorvus: yeah - I see a bunch of 2nd attempt jobs - so that part happened successfully16:58
corvusthis could have been a cloud host migration16:58
mordredyeah. cause otherwise losing connection with gearman  is ... weird16:58
corvus(i don't see anything in dmesg to suggest an oom or similar)16:58
corvusmordred: yep. especcially a connection to localhost :)16:58
mordredagree. server looks mostly healthy16:58
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750317:00
*** factor has quit IRC17:03
*** factor has joined #opendev17:03
openstackgerritMerged zuul/zuul-jobs master: tox-parse_output: Ignore .tox directory  https://review.opendev.org/71748617:04
*** factor has quit IRC17:06
*** factor has joined #opendev17:06
*** factor has quit IRC17:08
*** factor has joined #opendev17:08
*** factor has quit IRC17:09
*** factor has joined #opendev17:09
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750317:15
AJaegerI'm trying to open http://zuul.opendev.org/tenants - takes a looong time17:28
AJaegerinfra-root, this takes ages ;( ^17:29
AJaegeroh, finally there17:29
mordredcorvus: it looks like we're continuing to have gearman issues17:42
mnaseryep, it just occured again17:44
mordredbunch more broken pipe errors17:47
mordredI'm still not seeing anything that would be the cause :(17:48
mordredand it's back17:50
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750317:58
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750318:08
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750318:24
*** factor has quit IRC18:36
*** smcginnis has quit IRC18:36
*** factor has joined #opendev18:37
*** smcginnis has joined #opendev18:37
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750318:39
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750318:42
mnaserhmm18:45
mnaseri wonder if rax is having networking issues assuming zuul is hosted tehre18:45
mnaseri'm seeing job logs hang18:45
mordredmnaser: it could be - although the zuul/gearman thing is over localhost, so that would be super odd18:46
mnaserin this case, gathering facts took 2 minutes for a vm in limestone18:46
mordredmnaser: hrm. why were we gathering facts? was it an explicit call? we're supposed to have fact cache going18:47
mnasermordred: this was before add-build-sshkey18:48
mnasermordred: http://zuul.opendev.org/t/zuul/stream/2b031799b6b34490b948106d84cadb82?logfile=console.log if you click in time :)18:48
mordredah - nod18:48
*** factor has quit IRC18:51
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750318:52
*** factor has joined #opendev19:03
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: DNM: refactor to ensure-repositories  https://review.opendev.org/71750319:07
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644219:16
yoctozeptowhat is it if zuul finished job as 'EXCEPTION'? ;o19:19
yoctozeptoI never doubted our gates are exceptional but -2 is a bad omen19:19
mnaseryoctozepto: got a link to that?19:20
mordredyoctozepto: we had some as-of-yet unexplained issues this morning19:22
yoctozeptomnaser, mordred: sure, here it is: https://review.opendev.org/71500819:34
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-repositories: Add role  https://review.opendev.org/71750719:36
yoctozeptoalso, I have a more general question about how zuul treats jobs from one change, it looks as if one of them on https://review.opendev.org/711186 used a (partially?) older code, weird19:38
mordredmnaser: http://paste.openstack.org/show/791624/ there's the exeception for the yoctozepto change19:39
yoctozeptoah, that is what exception means19:39
mordredyeah - it normally should not happen19:39
mordred:)19:40
mordredyoctozepto: can you point to where you think something was out of date?19:41
mordred(I don't know that job well enough to know what it's expecting vs what it got)19:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666319:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666719:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency  https://review.opendev.org/71667519:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency  https://review.opendev.org/71668219:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency  https://review.opendev.org/71668519:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency  https://review.opendev.org/71668719:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency  https://review.opendev.org/71669219:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency  https://review.opendev.org/71669319:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency  https://review.opendev.org/71669519:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency  https://review.opendev.org/71669819:41
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-repositories: Add role  https://review.opendev.org/71750719:41
mnasermordred: seems to align with the behaviour that was happening today..19:42
mnaseralso i suck at rebasing :)19:42
mordredmnaser: yeah. me too19:42
openstackgerritMerged opendev/system-config master: Correct launch readme link  https://review.opendev.org/71547819:46
yoctozeptomordred: sure, all jobs follow the same path, dependency on tox was removed, yet centos8-source tried to call it failing to get "action" var from zuul which was removed in that same change19:49
yoctozeptoso I'm curious whether zuul could merge things independently19:50
yoctozeptoI ran recheck which should rather pass...19:50
mordredinfra-root: https://review.opendev.org/#/c/716442/ and https://review.opendev.org/#/c/717491/ are ready for review/land next time you pop in19:52
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-nodejs: refactor to use ensure-repositories  https://review.opendev.org/71750819:52
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-yarn: always install  https://review.opendev.org/71737519:58
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: refactor to use ensure-repositories  https://review.opendev.org/71750919:58
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-kubernetes: refactor to use ensure-repositories  https://review.opendev.org/71751020:00
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-podman: refactor to use ensure-repositories  https://review.opendev.org/71751120:05
*** factor has quit IRC20:08
*** factor has joined #opendev20:09
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-docker: refactor to use ensure-repositories  https://review.opendev.org/71751220:13
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: cleanup: move tests to use ensure-repositories  https://review.opendev.org/71751320:29
*** diablo_rojo has joined #opendev20:40
*** factor has quit IRC20:42
*** factor has joined #opendev20:43
*** factor has quit IRC20:49
*** diablo_rojo has quit IRC21:14
*** DSpider has quit IRC22:58
*** tosky has quit IRC23:00
« Friday, 2020-04-03 Index Sunday, 2020-04-05 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!