Tuesday, 2019-02-12

clarkbThe one case of a git clone failing in the gate was to github "remote hung up unexpectedly"00:00
clarkbgithub is consistently unreliable :P00:00
clarkbmwhahaha: zbr ^ fyi that failure was in a tripleo job trying to clone https://github.com/rdo-packages/tripleo-heat-templates-distgit00:02
clarkbhttp://logs.openstack.org/85/635785/5/gate/tripleo-ci-centos-7-standalone/44ec681/job-output.txt#_2019-02-11_09_01_01_15034500:03
mwhahahaYes we don't have options for the package distgits00:03
mwhahahaSo without being able to mirror them, we suffer github00:03
mwhahahaCould probably add retries somewhere00:04
*** wolverineav has quit IRC00:04
corvustripleo depends on unpublished non-openstack software?00:04
corvusor rather, unreleased, not unpublished00:05
*** wolverineav has joined #openstack-infra00:05
clarkbcorvus: this is the weird relationship between rdo and tripleo. AIUI tripleo rebuilds rdo packages in the gate to speculatively test changes to say triple-heat-templates. But because tehy install everything from rpms they have to build rpms which means finding the package spec files00:05
mwhahahacorvus: no but we rely on package specs00:06
clarkbthen when triple-heat-templates merges in the gate RDO will rebuild its RPMs and tripleo will test with those RPMs until the next speculative change00:06
clarkb(I think)00:06
mwhahahaYes it's the pacakaging details because we don't install from source00:07
mwhahahaLike actual usera00:07
mwhahahaI don't know the history between RDO and the rpm package sig or project00:08
*** wolverineav has quit IRC00:10
*** rh-jelabarre has quit IRC00:11
hogepodgeclarkb: I don't know00:12
hogepodgeclarkb: portdirect is probably the person to ask00:13
clarkbhogepodge: thanks00:13
*** rh-jelabarre has joined #openstack-infra00:13
*** wolverineav has joined #openstack-infra00:14
*** Emine has joined #openstack-infra00:15
*** tosky has quit IRC00:18
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: executor: enable zuul_return to update Ansible inventory  https://review.openstack.org/59009200:20
*** Emine has quit IRC00:27
clarkbhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64465&rra_id=all http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64537&rra_id=all an improvement maybe but not a fix to the swap problems00:34
*** rlandy is now known as rlandy|bbl00:36
*** wolverineav has quit IRC00:37
*** rascasoft has joined #openstack-infra00:38
clarkbI should stop looking at that as the data likely to be too noisy in the short term. I'll recheck in 24 hours00:38
*** wolverineav has joined #openstack-infra00:39
*** wolverineav has quit IRC00:43
*** wolverineav has joined #openstack-infra00:44
*** rascasoft has quit IRC00:45
clarkbwow pip is warning people about the python2.7 deprecation now00:46
clarkbor maybe that is setuptools00:46
*** hwoarang_ has quit IRC00:48
*** yamamoto has joined #openstack-infra00:51
*** hwoarang has joined #openstack-infra00:53
*** gyee has quit IRC00:54
*** yamamoto has quit IRC00:56
*** davidlenwell has joined #openstack-infra00:59
*** mnasiadka has joined #openstack-infra00:59
*** mrhillsman has joined #openstack-infra00:59
*** sweston has joined #openstack-infra00:59
*** icey has joined #openstack-infra00:59
*** csatari has joined #openstack-infra00:59
*** TheJulia has joined #openstack-infra00:59
*** rh-jelabarre has quit IRC00:59
*** coreycb has joined #openstack-infra00:59
*** sdake has joined #openstack-infra01:01
clarkbcorvus: in https://review.openstack.org/#/c/634829/2 don't we need something to install skopeo?01:04
*** gcutrini has joined #openstack-infra01:05
*** wolverineav has quit IRC01:09
*** gcutrini has quit IRC01:09
*** wolverineav has joined #openstack-infra01:09
*** gcutrini has joined #openstack-infra01:10
*** hwoarang has quit IRC01:10
clarkbdoesn't look like it is in ubuntus main repos (there is a PPA), it is available on opensuse though01:11
*** hwoarang has joined #openstack-infra01:11
*** ijw has joined #openstack-infra01:12
*** gcutrini has quit IRC01:15
*** gcutrini has joined #openstack-infra01:17
openstackgerritMerged openstack-infra/zuul master: Add a note about sqlalchemy metadata  https://review.openstack.org/63618901:18
*** gcutrini has quit IRC01:22
*** mriedem has quit IRC01:23
*** markvoelker has quit IRC01:30
*** markvoelker has joined #openstack-infra01:31
*** gcutrini has joined #openstack-infra01:32
*** gcutrini has quit IRC01:34
*** markvoelker has quit IRC01:35
*** sdake has quit IRC01:37
corvusclarkb: http://38.108.68.64/openstack-infra/system-config/src/branch/master/manifests/site.pp#L80701:38
corvusi'll update the readme with that since it needs a recheck anyway01:40
*** sdake has joined #openstack-infra01:41
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Add intermediate registry push/pull roles  https://review.openstack.org/63482901:41
corvusclarkb: ^01:41
*** sdake has quit IRC01:43
clarkbah on the executors01:44
corvusyeah, that way we can do it in pre and post trusted playbooks so that they can work on any job without trusting the content of the job itself01:48
*** wolverineav has quit IRC01:49
corvus(since we have to use secrets to use the intermediate registry, we don't want to run the pull role on the node after the job runs; it could own the node then own the creds)01:49
*** sdake has joined #openstack-infra01:50
*** sdake has quit IRC01:51
*** wolverineav has joined #openstack-infra01:52
*** yamamoto has joined #openstack-infra01:53
*** wolverineav has quit IRC01:56
*** ijw has quit IRC02:07
openstackgerritMerged openstack-infra/zuul master: web: add /{tenant}/buildsets route  https://review.openstack.org/63003502:12
*** rascasoft has joined #openstack-infra02:20
*** rascasoft has quit IRC02:26
*** jamesmcarthur has joined #openstack-infra02:28
*** hongbin has joined #openstack-infra02:29
*** markvoelker has joined #openstack-infra02:31
*** jamesmcarthur has quit IRC02:32
*** yamamoto has quit IRC02:48
*** hwoarang has quit IRC02:48
*** hwoarang has joined #openstack-infra02:56
*** rascasoft has joined #openstack-infra02:57
*** markvoelker has quit IRC03:05
*** rascasoft has quit IRC03:07
*** rlandy|bbl is now known as rlandy03:12
*** armax has quit IRC03:12
*** yamamoto has joined #openstack-infra03:14
*** rlandy has quit IRC03:19
*** ykarel has joined #openstack-infra03:26
*** armax has joined #openstack-infra03:34
*** sdake has joined #openstack-infra03:40
*** hwoarang has quit IRC03:40
*** hwoarang has joined #openstack-infra03:45
*** dave-mccowan has quit IRC03:58
*** sdake has quit IRC03:59
*** markvoelker has joined #openstack-infra04:02
*** udesale has joined #openstack-infra04:02
*** ramishra has joined #openstack-infra04:06
*** diablo_rojo has quit IRC04:09
*** armax has quit IRC04:28
*** jamesmcarthur has joined #openstack-infra04:29
*** janki has joined #openstack-infra04:30
*** hwoarang has quit IRC04:30
*** armax has joined #openstack-infra04:30
*** hwoarang has joined #openstack-infra04:31
*** jamesmcarthur has quit IRC04:34
*** markvoelker has quit IRC04:34
*** manjeets has quit IRC04:41
*** ykarel has quit IRC04:47
*** armax has quit IRC04:59
*** ykarel has joined #openstack-infra05:05
openstackgerritjacky06 proposed openstack/diskimage-builder master: Update hacking version to latest  https://review.openstack.org/62846605:09
openstackgerritjacky06 proposed openstack/diskimage-builder master: Update hacking version to latest  https://review.openstack.org/62846605:10
openstackgerritAndreas Jaeger proposed opendev/base-jobs master: DNM: Testing all jobs  https://review.openstack.org/63626505:12
*** adriant has quit IRC05:16
openstackgerritMerged openstack-infra/project-config master: [rally] Remove readthedocs jobs  https://review.openstack.org/61586805:19
*** hongbin has quit IRC05:30
*** markvoelker has joined #openstack-infra05:32
*** udesale has quit IRC05:37
*** hwoarang has quit IRC05:42
*** hwoarang has joined #openstack-infra05:44
*** gvrangan has joined #openstack-infra05:46
*** chandankumar is now known as chkumar|ruck05:47
*** wolverineav has joined #openstack-infra05:55
openstackgerritMerged openstack-infra/zuul master: Update SQLAlchemy url in docs  https://review.openstack.org/63514805:59
*** markvoelker has quit IRC06:05
*** whoami-rajat has joined #openstack-infra06:19
*** e0ne has joined #openstack-infra06:24
*** jamesmcarthur has joined #openstack-infra06:30
*** jamesmcarthur has quit IRC06:34
*** e0ne has quit IRC06:35
*** wolverineav has quit IRC06:38
*** sshnaidm is now known as sshnaidm|afk06:39
*** zhangfei has joined #openstack-infra06:44
*** ccamacho has quit IRC06:49
*** zaneb has quit IRC06:53
openstackgerritFelix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API  https://review.openstack.org/63571406:56
AJaegerclarkb: your configure-unbound change is broken, see http://logs.openstack.org/65/636265/1/check/openstack-zuul-jobs-linters/42f385c/job-output.txt.gz#_2019-02-12_05_15_52_161674 - all of opendev/base-jobs fail now06:57
*** pgaxatte has joined #openstack-infra06:59
AJaegerclarkb: I'm confused - looking at the change, I don't see what can be wrong...07:00
*** quiquell|off is now known as quiquell|rover07:02
*** markvoelker has joined #openstack-infra07:02
*** kopecmartin|off is now known as kopecmartin07:05
AJaegerand I don't understand why the failure shows that ansible checks "opendev/base-jobs/playbooks/base/roles" instead of "opendev/base-jobs/roles" - any help welcome!07:06
*** udesale has joined #openstack-infra07:09
*** e0ne has joined #openstack-infra07:12
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add opendev/base-jobs to openstack-zuul-linters  https://review.openstack.org/63627407:13
AJaegerclarkb: the change above should fix it ^07:13
openstackgerritAndreas Jaeger proposed opendev/base-jobs master: Update lint testing  https://review.openstack.org/63627507:16
*** e0ne has quit IRC07:16
AJaegerclarkb: we can simplify testing with the above ^ - and IMHO still need 636274 for other repos.07:16
AJaegercorvus, clarkb, could you review this later, please?07:17
*** snapiri has joined #openstack-infra07:24
*** e0ne has joined #openstack-infra07:24
AJaegerI'll abandon 636275, it won't work - we reference roles in zuul-jobs from base-jobs; so 636274 is all we need.07:26
*** e0ne has quit IRC07:32
*** Emine has joined #openstack-infra07:32
*** ykarel is now known as ykarel|lunch07:33
*** markvoelker has quit IRC07:35
*** ramishra_ has joined #openstack-infra07:37
*** ramishra has quit IRC07:38
*** ykarel|lunch has quit IRC07:41
*** e0ne has joined #openstack-infra07:42
*** ykarel has joined #openstack-infra07:43
*** slaweq has joined #openstack-infra07:44
*** Emine has quit IRC07:49
*** aojea has joined #openstack-infra07:52
*** quiquell|rover is now known as quique|rover|brb07:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Install pyliblzma with pip instead of distro packages  https://review.openstack.org/63589207:54
*** ramishra_ is now known as ramishra07:56
*** gvrangan has quit IRC07:57
*** ccamacho has joined #openstack-infra07:58
*** ccamacho has quit IRC07:59
*** ccamacho has joined #openstack-infra07:59
*** zhangfei has quit IRC08:00
*** ginopc has joined #openstack-infra08:01
openstackgerritOpenStack Proposal Bot proposed openstack-infra/storyboard master: Imported Translations from Zanata  https://review.openstack.org/62539308:02
*** rascasoft has joined #openstack-infra08:02
*** quique|rover|brb is now known as quiquell|rover08:12
*** zhangfei has joined #openstack-infra08:12
*** rpittau has joined #openstack-infra08:13
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Escape jinja2 template from commit message  https://review.openstack.org/63393008:21
*** xek_ has joined #openstack-infra08:28
*** tkajinam has quit IRC08:29
*** gfidente has joined #openstack-infra08:29
*** jamesmcarthur has joined #openstack-infra08:31
*** markvoelker has joined #openstack-infra08:32
*** e0ne has quit IRC08:35
*** jamesmcarthur has quit IRC08:35
*** e0ne has joined #openstack-infra08:37
*** e0ne has quit IRC08:38
*** tosky has joined #openstack-infra08:40
*** yboaron_ has joined #openstack-infra08:44
openstackgerritWilliam Huang proposed openstack-infra/grafyaml master: Add prometheus data source  https://review.openstack.org/63628008:45
*** e0ne has joined #openstack-infra08:46
*** e0ne has quit IRC08:48
*** jpena|off is now known as jpena08:53
*** e0ne has joined #openstack-infra08:54
*** whoami-rajat has quit IRC08:54
*** e0ne has quit IRC08:56
*** priteau has joined #openstack-infra08:58
*** jpich has joined #openstack-infra08:58
*** panda|off is now known as panda09:04
*** markvoelker has quit IRC09:05
*** roman_g has joined #openstack-infra09:07
*** jistr_ is now known as jistr09:09
*** Emine has joined #openstack-infra09:15
*** ykarel has quit IRC09:20
*** shardy has joined #openstack-infra09:21
*** ramishra has quit IRC09:22
*** e0ne has joined #openstack-infra09:25
*** e0ne has quit IRC09:26
*** Emine has quit IRC09:26
*** Emine has joined #openstack-infra09:26
*** yboaron_ has quit IRC09:26
*** yboaron_ has joined #openstack-infra09:27
*** ykarel has joined #openstack-infra09:27
*** ramishra has joined #openstack-infra09:29
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add opendev/base-jobs to openstack-zuul-linters  https://review.openstack.org/63627409:32
*** jpich has quit IRC09:37
*** jpich has joined #openstack-infra09:38
AJaegerfrickler: fixed the nit ^09:41
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Escape jinja2 template from commit message  https://review.openstack.org/63393009:43
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory  https://review.openstack.org/63393009:45
*** sshnaidm|afk is now known as sshnaidm09:52
roman_gHello. Could I get reviews on https://review.openstack.org/#/c/635381/ - Replace deprecated docs job in docs-on-readthedocs template (Change docs-on-readthedocs project template from using deprecated build-openstack-sphinx-docs job for check and gate to openstack-tox-docs job.). Thank you.09:54
roman_gThanks clarkb for +2.09:54
slaweqfrickler: hi09:57
openstackgerritStephen Finucane proposed openstack-infra/git-review master: Switch from testrepository to stestr  https://review.openstack.org/62865509:59
slaweqfrickler: can You help me with transition of one job to zuulv3 and put on hold node with my job neutron-tempest-dvr-ha-multinode-full from patch https://review.openstack.org/#/c/633979/ ?09:59
*** jamesmcarthur has joined #openstack-infra09:59
slaweqfrickler: it is failing, probably there is some issue with connectivity between nodes but I'm not sure what is wrong there, access to such nodes would be very helpful09:59
slaweqor maybe someone else from infra team can help me with ^^? :)10:00
*** stephenfin_ is now known as stephenfin10:01
*** markvoelker has joined #openstack-infra10:02
*** jamesmcarthur has quit IRC10:04
*** gfidente has quit IRC10:09
*** kjackal_v2 has quit IRC10:16
*** kjackal has joined #openstack-infra10:16
*** derekh has joined #openstack-infra10:27
openstackgerritMerged openstack-infra/git-review master: Switch from testrepository to stestr  https://review.openstack.org/62865510:29
*** markvoelker has quit IRC10:35
fricklerslaweq: setting up a hold now. maybe for testing you can amend your .zuul.yaml and skip running all the other neutron jobs?10:41
slaweqfrickler: thx, I will send such debug patch in few minutes with disabled other jobs10:42
*** gfidente has joined #openstack-infra10:42
*** jtomasek has quit IRC10:46
*** udesale has quit IRC10:49
fricklerslaweq: o.k., can you remind me of your ssh key again?10:51
* frickler ponders whether it would make sense to collect those on an etherpad or in a wiki page10:51
*** zhangfei has quit IRC10:56
*** jtomasek has joined #openstack-infra10:56
*** Emine has quit IRC11:09
*** Emine has joined #openstack-infra11:10
*** lpetrut has joined #openstack-infra11:15
*** markvoelker has joined #openstack-infra11:32
iurygregoryHi everyone, while moving jobs to zuulv3 in ironic we notice that ironic-tempest-plugin is not running all jobs we have set in project.yaml https://github.com/openstack/ironic-tempest-plugin/blob/master/zuul.d/project.yaml, i checked if the parent jobs really exists and they do anyone have ideas what it could be? Some jobs that arent running are defined in stable-jobs.yaml https://github.com/openstack/ironic-tempest-plugin/b11:47
iurygregorylob/master/zuul.d/stable-jobs.yaml11:47
iurygregoryhttps://github.com/openstack/ironic-tempest-plugin/blob/master/zuul.d/stable-jobs.yaml*11:47
iurygregorys/*/11:48
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul CLI: allow access via REST  https://review.openstack.org/63631511:50
*** markvoelker has quit IRC12:05
frickleriurygregory: I might be mistaken, but from the zuul docs, with "override-checkout: stable/rocky" it checks the stable branch for the job definition, too. which fails, since ironic-tempest-plugin doesn't have stable branches12:12
iurygregoryfrickler, yeah ironic-tempest-plugin doesnt have any stable/* branches, i think before it checkouts to run stable/* for the projects (ironic etc)12:13
iurygregoryto test if the tempest for that release is not broken12:13
iurygregoryi should override the branch of the openstack/ironic using required-projects?12:14
*** rkukura has quit IRC12:18
frickleriurygregory: hmm, neutron seems to set things up similarly, so maybe you have a different issue https://github.com/openstack/neutron-tempest-plugin/blob/master/.zuul.yaml#L23012:21
* iurygregory looking12:21
frickleriurygregory: maybe clarkb or corvus can help better12:21
iurygregoryfrickler, tks o/12:21
*** rkukura has joined #openstack-infra12:22
*** kaisers has quit IRC12:24
*** jpena is now known as jpena|lunch12:26
*** emine__ has joined #openstack-infra12:28
*** Emine has quit IRC12:30
*** kaisers has joined #openstack-infra12:31
*** panda is now known as panda|lunch12:32
*** coreycb has quit IRC12:36
*** coreycb has joined #openstack-infra12:37
*** lpetrut has quit IRC12:38
*** lpetrut has joined #openstack-infra12:39
iurygregoryfrickler, seems that override-branch works instead of checkout12:42
*** janki has quit IRC12:44
*** janki has joined #openstack-infra12:44
*** ykarel is now known as ykarel|away12:48
*** emine__ has quit IRC12:50
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory  https://review.openstack.org/63393012:53
*** ykarel|away has quit IRC12:54
*** rh-jelabarre has joined #openstack-infra13:01
*** markvoelker has joined #openstack-infra13:02
*** yamamoto has quit IRC13:03
*** mriedem has joined #openstack-infra13:04
*** yamamoto has joined #openstack-infra13:08
*** udesale has joined #openstack-infra13:10
*** chkumar|ruck is now known as chkumar|pto13:10
*** trown|outtypewww is now known as trown13:11
*** panda|lunch is now known as panda13:13
*** yamamoto has quit IRC13:15
*** quiquell|rover is now known as quique|rover|eat13:17
*** rlandy has joined #openstack-infra13:17
openstackgerritMerged openstack-infra/grafyaml master: Add prometheus data source  https://review.openstack.org/63628013:18
*** whoami-rajat has joined #openstack-infra13:22
*** agopi has joined #openstack-infra13:26
*** ociuhandu has joined #openstack-infra13:26
slaweqfrickler: my job failed: cd ../../Fedora/multinode_dvr13:29
slaweqhttp://logs.openstack.org/79/633979/7/check/neutron-tempest-dvr-ha-multinode-full/2ed9446/job-output.txt.gz13:29
slaweqcan You add ssh key to those nodes maybe?13:30
*** jpena|lunch is now known as jpena13:30
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for peakmem_tracker crash bug 1815604  https://review.openstack.org/63633113:30
openstackbug 1815604 in OpenStack-Gate "peakmem_tracker crashes with "Failure listing processes locking memory: [Errno 3] No such process"" [Undecided,Confirmed] https://launchpad.net/bugs/181560413:30
*** sdake has joined #openstack-infra13:32
openstackgerritMerged openstack-infra/project-config master: swift: gerritbot alert on all the branches  https://review.openstack.org/63590513:32
*** gcutrini has joined #openstack-infra13:32
fricklerslaweq: done, 23.253.201.90 and 23.253.216.4413:33
slaweqfrickler: thx a lot13:34
*** agopi has quit IRC13:34
*** markvoelker has quit IRC13:35
slaweqfrickler: can it be like that for few hours? I need to leave now for about 1h but will back to it today :)13:36
fricklerslaweq: sure, take your time, if it takes a couple of days, that's fine as well13:37
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory  https://review.openstack.org/63393013:37
*** openstackgerrit has quit IRC13:37
*** quique|rover|eat is now known as quiquell|rover13:39
*** ykarel|away has joined #openstack-infra13:43
*** gcutrini has quit IRC13:43
*** yamamoto has joined #openstack-infra13:46
*** gcutrini has joined #openstack-infra13:46
*** openstackgerrit has joined #openstack-infra13:51
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for peakmem_tracker crash bug 1815604  https://review.openstack.org/63633113:51
openstackbug 1815604 in OpenStack-Gate "peakmem_tracker crashes with "Failure listing processes locking memory: [Errno 3] No such process"" [Undecided,Confirmed] https://launchpad.net/bugs/181560413:51
*** ykarel|away is now known as ykarel13:51
*** smarcet has joined #openstack-infra13:51
smarcetfungi: morning, afaik dev01 is working fine, would u please change dns entry when u get a chance to repoint openstackid-dev to openstackid-dev01 ? thx :)13:52
*** yamamoto has quit IRC13:56
*** e0ne has joined #openstack-infra13:56
*** gcutrini has quit IRC14:01
*** agopi has joined #openstack-infra14:02
*** gcutrini has joined #openstack-infra14:02
*** sdake has quit IRC14:03
*** yamamoto has joined #openstack-infra14:06
*** yamamoto has quit IRC14:06
*** gcutrini has quit IRC14:08
*** gcutrini has joined #openstack-infra14:09
*** ginopc has quit IRC14:09
*** mriedem is now known as mriedem_snow14:10
*** zul has joined #openstack-infra14:10
*** smarcet has quit IRC14:11
openstackgerritMerged openstack-infra/system-config master: Rewrite #nXXX to #LXXX in incoming gitea urls  https://review.openstack.org/63537914:12
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: prevent status update loop in background  https://review.openstack.org/63634314:15
*** smarcet has joined #openstack-infra14:16
openstackgerritJakub Libosvar proposed openstack-infra/project-config master: Move networking-ansible jobs to its project  https://review.openstack.org/63583614:22
*** priteau has quit IRC14:26
*** sshnaidm has quit IRC14:30
*** sshnaidm has joined #openstack-infra14:30
openstackgerritFelix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API  https://review.openstack.org/63571414:30
*** sthussey has joined #openstack-infra14:31
*** markvoelker has joined #openstack-infra14:32
*** sdake has joined #openstack-infra14:35
*** rkukura has quit IRC14:36
*** sdake has quit IRC14:37
*** sdake_ has joined #openstack-infra14:37
*** smarcet has quit IRC14:38
*** rkukura has joined #openstack-infra14:40
*** e0ne has quit IRC14:44
*** smarcet has joined #openstack-infra14:46
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul CLI: allow access via REST  https://review.openstack.org/63631514:47
*** janki has quit IRC14:47
fungi#status log replaced openstackid-dev.openstack.org address records with a cname to openstackid-dev01.openstack.org14:48
openstackstatusfungi: finished logging14:48
fungismarcet: ^ that should be resolving any moment now14:48
smarcetfungi: thx u sire :)14:48
fungialready resolving for me14:49
fungiwhen i try to log into translate-dev via openstackid-dev i get "Whoops, looks like something went wrong." now after supplying a username and password, though it's possible i'm using incorrect credentials since i haven't tried it in a long time14:50
smarcetfungi: i will take a look14:51
*** udesale has quit IRC14:52
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul CLI: allow access via REST  https://review.openstack.org/63631514:54
*** gcutrini_ has joined #openstack-infra14:55
*** snapiri has quit IRC14:56
*** gcutrini has quit IRC14:56
*** gcutrini_ is now known as gcutrini14:56
*** ginopc has joined #openstack-infra14:57
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory  https://review.openstack.org/63393014:58
*** eharney has joined #openstack-infra15:00
roman_gHello. Could I get reviews on https://review.openstack.org/#/c/635381/ - Replace deprecated docs job in docs-on-readthedocs template (Change docs-on-readthedocs project template from using deprecated build-openstack-sphinx-docs job for check and gate to openstack-tox-docs job.). Thank you.15:00
smarcetfungi: its a permission issue15:01
smarcetfungi: will need to update puppet again , sorry15:01
*** udesale has joined #openstack-infra15:02
*** markvoelker has quit IRC15:05
*** ianychoi has quit IRC15:07
*** ianychoi has joined #openstack-infra15:07
fungino worries15:07
*** e0ne has joined #openstack-infra15:09
*** kjackal has quit IRC15:09
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid master: Updated Permissions for deployment  https://review.openstack.org/63635115:10
smarcetfungi: https://review.openstack.org/#/c/636351/1 please review at will that will fix the permissions issue :)15:11
*** sreejithp has joined #openstack-infra15:11
*** smarcet has quit IRC15:14
*** yamamoto has joined #openstack-infra15:16
*** kjackal has joined #openstack-infra15:16
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add opendev/base-jobs to openstack-zuul-linters  https://review.openstack.org/63627415:16
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Replace deprecated docs job in docs-on-readthedocs template  https://review.openstack.org/63538115:16
*** smarcet has joined #openstack-infra15:25
*** yamamoto has quit IRC15:25
*** priteau has joined #openstack-infra15:30
clarkb#status log Set new github admin account to owner on the openstack-infra org.15:31
openstackstatusclarkb: finished logging15:31
clarkbwe may need to do that for our other orgs ^15:31
*** e0ne has quit IRC15:33
*** armax has joined #openstack-infra15:40
*** e0ne has joined #openstack-infra15:41
*** ykarel is now known as ykarel|away15:43
*** luizbag has joined #openstack-infra15:43
openstackgerritSlawek Kaplonski proposed openstack-infra/project-config master: Add new python 3 Neutron jobs to grafana dashboard  https://review.openstack.org/63635915:44
openstackgerritMerged opendev/base-jobs master: Add docs  https://review.openstack.org/63600715:46
*** diablo_rojo has joined #openstack-infra15:47
fungiclarkb: yes, we do from what i can tell. also some of us who removed our accounts from admin groups are still listed as owner people on those orgs15:51
*** mriedem_snow is now known as mriedem15:52
*** gcutrini has quit IRC15:54
*** roman_g has quit IRC15:57
*** roman_g has joined #openstack-infra15:59
*** markvoelker has joined #openstack-infra16:03
*** lpetrut has quit IRC16:03
*** yboaron_ has quit IRC16:04
*** jamesmcarthur has joined #openstack-infra16:06
AJaegerteam, opendev/base-jobs tests are passing now, see https://review.openstack.org/636265 . thanks for reviews to get this working16:06
openstackgerritMerged openstack-infra/puppet-openstackid master: Updated Permissions for deployment  https://review.openstack.org/63635116:09
*** dpawlik has joined #openstack-infra16:14
*** quiquell|rover is now known as quiquell|off16:17
*** ccamacho has quit IRC16:23
*** ccamacho has joined #openstack-infra16:24
*** manjeets has joined #openstack-infra16:27
*** markvoelker has quit IRC16:35
*** udesale has quit IRC16:37
corvusclarkb: did you want to leave a vote on https://review.openstack.org/634829 ?16:37
clarkbcorvus: ya didn't have a chance to rereview with the skopeo note. Will look now16:38
corvusclarkb: we also need https://review.openstack.org/635857 merged to move forward on opendev16:40
clarkbcorvus: interesting that we have to push the image multiple times to apply multiple tags. I guess ti dedups based on layer shas though so its cheap16:42
corvusclarkb: yes.  everything about docker is "interesting".16:42
clarkbcorvus: where are you planning to source skopeo from on the executors? the ppa or building from source or something else?16:44
corvusclarkb: there already there.  projectatomic ppa.  that was the puppet link i sent you yesterday.16:44
corvusthey're already there.16:45
corvusclarkb: http://38.108.68.64/openstack-infra/system-config/src/branch/master/manifests/site.pp#L80716:45
*** ykarel|away has quit IRC16:49
clarkbcorvus: mordred think I found a bug in https://review.openstack.org/#/c/635857/4 possibly a forgotten git add16:52
*** ccamacho has quit IRC16:53
*** ramishra has quit IRC16:57
*** rossella_s has quit IRC16:58
*** pgaxatte has quit IRC17:00
*** sdake_ has quit IRC17:01
*** sdake has joined #openstack-infra17:02
*** hamzy has quit IRC17:06
*** ginopc has quit IRC17:08
*** rpittau has quit IRC17:08
*** aojea has quit IRC17:14
openstackgerritFabien Boucher proposed openstack-infra/zuul master: [WIP] URLTrigger driver time based - artifact change jobs triggering driver  https://review.openstack.org/63556717:15
*** raissa has joined #openstack-infra17:23
*** jpich has quit IRC17:24
*** e0ne has quit IRC17:27
*** rossella_s has joined #openstack-infra17:27
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Properly handle TaskManagerStopped exception  https://review.openstack.org/63639317:29
*** jamesmcarthur has quit IRC17:31
*** jamesmcarthur_ has joined #openstack-infra17:32
*** markvoelker has joined #openstack-infra17:32
*** armax has quit IRC17:32
*** gcutrini has joined #openstack-infra17:33
*** armax has joined #openstack-infra17:33
*** tobiash has quit IRC17:34
*** gfidente is now known as gfidente|afk17:34
*** auristor has quit IRC17:34
*** tobiash has joined #openstack-infra17:35
*** whoami-rajat has quit IRC17:36
*** gcutrini has quit IRC17:39
*** spsurya has quit IRC17:40
*** jamesmcarthur_ has quit IRC17:42
*** e0ne has joined #openstack-infra17:43
*** yamamoto has joined #openstack-infra17:44
*** sdake has quit IRC17:44
*** mriedem is now known as mriedem_afk17:45
*** sdake has joined #openstack-infra17:46
*** gyee has joined #openstack-infra17:47
*** yamamoto has quit IRC17:48
clarkbFriendly reminder we'll haev our weekly meeting Today in just over an hour. Starts at 19:00 UTC in #openstack-meeting. If you get a chance to do so before the meeting please give https://review.openstack.org/#/c/587283/ (letsencrypt spec) a read so that we can talk about it17:50
*** jamesmcarthur has joined #openstack-infra17:51
*** armax has quit IRC17:53
*** dpawlik has quit IRC17:53
*** jamesmcarthur has quit IRC17:56
*** derekh has quit IRC17:56
*** jamesmcarthur has joined #openstack-infra17:58
*** jamesmcarthur has quit IRC17:58
*** dtantsur is now known as dtantsur|afk17:58
*** kopecmartin is now known as kopecmartin|off17:58
*** raissa has quit IRC18:00
*** jamesmcarthur has joined #openstack-infra18:01
*** markvoelker has quit IRC18:06
*** dpawlik has joined #openstack-infra18:06
*** agopi is now known as agopi|food18:06
*** panda is now known as panda|off18:06
*** jamesmcarthur has quit IRC18:07
*** jamesmcarthur has joined #openstack-infra18:08
*** dpawlik has quit IRC18:10
openstackgerritJeremy Stanley proposed openstack-infra/zuul master: Track GitHub connection event queue length stats  https://review.openstack.org/63640418:10
*** yamamoto has joined #openstack-infra18:11
*** jamesmcarthur has quit IRC18:12
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Stop running k8s-on-openstack nested  https://review.openstack.org/63585718:17
*** jpena is now known as jpena|off18:23
*** dpawlik has joined #openstack-infra18:23
*** agopi|food is now known as agopi18:30
*** armax has joined #openstack-infra18:31
*** eharney has quit IRC18:33
*** panda|off has quit IRC18:34
*** gcerami_ has joined #openstack-infra18:37
*** trown is now known as trown|lunch18:41
openstackgerritMerged openstack-infra/zuul-jobs master: Add intermediate registry push/pull roles  https://review.openstack.org/63482918:41
*** gcutrini has joined #openstack-infra18:41
*** ociuhandu_ has joined #openstack-infra18:43
*** hamzy has joined #openstack-infra18:43
*** sdake has quit IRC18:44
*** smarcet has quit IRC18:44
*** wolverineav has joined #openstack-infra18:45
*** ociuhandu has quit IRC18:47
*** priteau has quit IRC18:47
*** ociuhandu_ has quit IRC18:48
*** sdake has joined #openstack-infra18:51
*** ociuhandu has joined #openstack-infra18:55
*** jamesmcarthur has joined #openstack-infra18:56
*** krasmussen has joined #openstack-infra18:58
*** jamesmcarthur has quit IRC18:58
*** slaweq has quit IRC18:59
*** ociuhandu has quit IRC18:59
*** ociuhandu has joined #openstack-infra19:00
clarkbMeeting time over in #openstack-meeting19:00
*** wolverineav has quit IRC19:00
*** markvoelker has joined #openstack-infra19:03
*** jiapei_ has joined #openstack-infra19:03
*** wolverineav has joined #openstack-infra19:04
*** jrosser_ has joined #openstack-infra19:04
*** ociuhandu has quit IRC19:04
*** adrianreza has quit IRC19:04
*** jrosser has quit IRC19:04
*** dtantsur|afk has quit IRC19:04
*** stephenfin has quit IRC19:04
*** jroll has quit IRC19:04
*** yankcrime has quit IRC19:04
*** geguileo has quit IRC19:04
*** jiapei has quit IRC19:04
*** mugsie has quit IRC19:04
*** fdegir has quit IRC19:04
*** zul has quit IRC19:04
*** tinwood has quit IRC19:04
*** jiapei_ is now known as jiapei19:04
*** jrosser_ is now known as jrosser19:04
*** tinwood has joined #openstack-infra19:04
*** mugsie has joined #openstack-infra19:05
*** fdegir has joined #openstack-infra19:05
*** stephenfin has joined #openstack-infra19:06
*** jroll has joined #openstack-infra19:06
*** ociuhandu has joined #openstack-infra19:06
*** jamesmcarthur has joined #openstack-infra19:07
*** dtantsur has joined #openstack-infra19:08
*** luizbag has quit IRC19:10
*** gcerami_ is now known as panda19:14
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Add dockerized test setup  https://review.openstack.org/63642419:18
*** dmsimard has quit IRC19:22
*** gcutrini has quit IRC19:25
*** jamesmcarthur has quit IRC19:30
*** dmsimard has joined #openstack-infra19:32
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Improve event logging in githubconnection  https://review.openstack.org/63642919:33
*** markvoelker has quit IRC19:36
*** sdake has quit IRC19:42
*** wolverineav has quit IRC19:43
*** wolverineav has joined #openstack-infra19:44
*** trown|lunch is now known as trown19:44
*** wolverineav has quit IRC19:46
*** wolverineav has joined #openstack-infra19:47
*** eharney has joined #openstack-infra19:48
*** wolverineav has quit IRC19:49
*** wolverineav has joined #openstack-infra19:50
nhicherhello, is there any documentation to create 'Non-Interactive Users' account for https://review.openstack.org/ ?19:54
*** wolverineav has quit IRC19:55
funginhicher: why do you need a non-interactive user on review.openstack.org?19:56
fungicreating/setting them needs system administrator involvement19:56
*** wolverineav has joined #openstack-infra19:57
*** jamesmcarthur has joined #openstack-infra19:59
*** zul has joined #openstack-infra20:00
nhicherfungi: I'm working to setup a zuul instance, and my understanding is I need a user account for that20:00
nhicherfungi: to be able to honor depends-on20:01
nhicherfungi: https://zuul-ci.org/docs/zuul/admin/drivers/gerrit.html#connection-configuration20:04
ildikovfungi: following up from the meeting, what is the timeline for the namespace changes?20:04
*** mriedem_afk is now known as mriedem20:05
ildikovAnd which announcement did I miss? :)20:06
clarkbildikov: no announcement yet. That was the earlier part of the meeting. Deciding when to communicate stuff20:06
clarkbildikov: fungi had mentioned that people tend to ignore annoubncements so I suggested we reach out ot people like you directly when ready to make sure the announcement is digested properly :)20:06
ildikovAh ok, I thought I already missed it :)20:06
clarkbildikov: so nothing to worry about now. We'll get in touch when there is soemthing :)20:06
ildikovPerfect!20:07
clarkbnhicher: all of our gerrit accounts should be created via launchpad/ubuntuone openid currently20:07
clarkbnhicher: including those for CI systems20:07
ildikovHappy to help making sure the information travels fast and recognized by people who need to act on it20:07
clarkbnhicher: https://docs.openstack.org/infra/system-config/third_party.html#creating-a-service-account20:07
*** dpawlik has quit IRC20:08
odyssey4meHi folks - in glean, is there anything which actually mounts config drive, or does it expect to already have the mount in place?20:11
funginhicher: you need a gerrit account (assuming you're talking about having it act as a third-party ci system) but i don't see in the documentation you linked that it needs to be a non-interactive user20:11
fungiwe expect third-party ci systems to use normal gerrit accounts20:12
clarkbodyssey4me: glean mounts the config drive for you20:12
clarkbodyssey4me: it might expect that the `mount` executable exist20:12
odyssey4meclarkb apologies - I just found that statement in the README... I'm just having issues with a saved image (outside of openstack CI) where glean's not actually doing the mount20:13
clarkbodyssey4me: is the config drive appropriately labeled? it has to be CONFIG-2 or something20:13
*** adriant has joined #openstack-infra20:15
odyssey4meclarkb yep, it would appear so20:15
clarkbmordred: I +2'd the ansible fix as well as your opendev dns changes. The gitea deployment changes already have many +2's. I'll let you apply the +A as appropriate to trigger things so you don't have to do it all at once :)20:15
clarkbcorvus: ^ fyi20:15
mordredclarkb: coolio20:16
clarkbodyssey4me: do you have boot console logs to show if glean is running at all?20:16
clarkbodyssey4me: glean/init/glean.sh in the repo is the script that should be run that will mount the config drive then run glean20:17
*** slaweq has joined #openstack-infra20:19
nhicherclarkb: ok, I will follow this doc, thanks20:20
*** wolverineav has quit IRC20:20
nhicherfungi: it's because I work on software-factory, and we create non-interactive user for that20:20
nhicherneed20:21
*** wolverineav has joined #openstack-infra20:21
funginhicher: yeah, nothing about hooking a zuul to a gerrit necessarily requires a non-interactive user account20:22
fungiany kind of gerrit account will work as long as it has the necessary permissions granted to it20:22
odyssey4meclarkb hmm, running 'glean.sh; glean -i eth0 --debug' seems to work alright20:22
nhicherfungi: ok, thanks =)20:22
clarkbodyssey4me: I would double check that that is actually running then. Make sure you unit is enabled and that it runs late enough to see those devices?20:23
ianwodyssey4me: what platform you on?20:23
odyssey4meclarkb well, the service isn't running - it shows as loaded, but inactive and dead20:24
odyssey4meianw bionic20:24
ianwhrm, that might be right it's a oneshot20:25
*** e0ne has quit IRC20:25
odyssey4meianw however, this is an instance built with diskimage-builder, then saved as an image20:25
odyssey4mewell, built - then had stuff done to it, then saved as an image20:25
ianwby stuff, had it already booted?20:26
ianwor you mean like mounting it and fiddling20:26
odyssey4meyep - an instance booted, and worked - things were installed onto the instance, then the instance saved as an image20:26
odyssey4mesorry - long day, I'm not communicating very well20:26
*** wolverineav has quit IRC20:26
ianwodyssey4me: np ... so in that case it has the http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/e2fac10/controller/logs/screen-dstat.txt.gz20:27
ianwbah, bad paste20:27
ianwthe /etc/network/interfaces.d file already from the first boot?  or was that cleared out?20:28
odyssey4meianw that was cleared out20:28
odyssey4mewhere's the systemd service file meant to be?20:28
ianwyou should have a /usr/lib/systemd/system/glean@.service20:29
odyssey4meok, so it's a oneshot20:30
odyssey4menow I'm trying to figure out how to see what happened at boot - through an emergency console with no network access :/20:30
ianwwelcome to glean debugging!20:31
roman_gThanks to Jeremy Stanley for review/merge 635381.20:31
odyssey4mewould this log anywhere? the journal for the service doesn't appear to have anything from the most recent boot20:31
roman_g(don't know his IRC nick)20:31
ianwroman_g: that's fungi :)20:31
*** jamesmcarthur has quit IRC20:31
fungiyou're welcome roman_g!20:32
roman_g=)20:32
*** markvoelker has joined #openstack-infra20:33
ianwodyssey4me: it should be in the journal ... if you can see it20:34
*** xek__ has joined #openstack-infra20:35
ianwthe service should be triggered by udev when the network device appears; it then checks for the network.d file and if not present should write it out from configdrive20:35
*** xek_ has quit IRC20:37
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Adds _JAVA_OPTIONS to tox passenv  https://review.openstack.org/62852520:39
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Improve event logging in githubconnection  https://review.openstack.org/63642920:47
*** e0ne has joined #openstack-infra20:48
*** jamesmcarthur has joined #openstack-infra20:52
*** jamesmcarthur has quit IRC20:54
openstackgerritMerged openstack-infra/system-config master: Upgrade to LimeSurvey 3.15.6  https://review.openstack.org/57990220:54
*** dpawlik has joined #openstack-infra20:59
*** wolverineav has joined #openstack-infra21:00
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Improve event logging in githubconnection  https://review.openstack.org/63642921:01
odyssey4methanks for the pointers ianw - it looks to me like glean isn't starting at boot, but I'll need to review it again in the morning... time for me to !computer21:02
*** slaweq_ has joined #openstack-infra21:04
ianwodyssey4me: this wouldn't be the first time systemd dependencies have caused problems :/21:05
*** dpawlik has quit IRC21:06
*** slaweq has quit IRC21:06
*** dtantsur has quit IRC21:06
*** fdegir has quit IRC21:06
*** tinwood has quit IRC21:06
*** mugsie has quit IRC21:06
*** dtantsur has joined #openstack-infra21:06
*** markvoelker has quit IRC21:06
*** mugsie has joined #openstack-infra21:06
ianwif you want to post the image at some point, i can at least try it and see if it replicates21:06
*** tinwood has joined #openstack-infra21:06
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Improve event logging in githubconnection  https://review.openstack.org/63642921:15
*** dpawlik has joined #openstack-infra21:20
*** dpawlik has quit IRC21:24
*** sdake has joined #openstack-infra21:26
*** e0ne has quit IRC21:28
*** wolverineav has quit IRC21:30
*** xek__ has quit IRC21:31
*** slaweq_ is now known as slaweq21:32
*** wolverineav has joined #openstack-infra21:34
*** jtomasek has quit IRC21:35
*** trown is now known as trown|outtypewww21:36
*** agopi_ has joined #openstack-infra21:37
*** agopi has quit IRC21:39
*** roman_g has quit IRC21:40
*** agopi_ is now known as agopi21:41
*** wolverineav has quit IRC21:41
*** wolverineav has joined #openstack-infra21:41
*** gcutrini has joined #openstack-infra21:41
*** gcutrini has quit IRC21:44
*** hamzy has quit IRC21:46
*** sdake has quit IRC21:50
slaweqhi infra team21:51
clarkbslaweq: hello21:51
slaweqtoday frickler hold for me 2 nodes: 23.253.216.44 and 23.253.201.90 for debugging21:51
slaweqYou can clean them now, I found what I needed, thx a lot :)21:51
clarkbwill do, thank you for letting us know21:52
slaweqclarkb: thx21:52
*** weshay has joined #openstack-infra21:52
weshayanyone seeing 19-02-12 21:40:50.348996 | primary | The error appears to have been in '/home/zuul/workspace/logs/zuul-variables.yaml': line 142, column 1, but may21:53
weshay2019-02-12 21:40:50.349133 | primary | be elsewhere in the file depending on the exact syntax problem.21:53
clarkbweshay: no, but I'ev just got back from lunch. Have a link?21:54
weshaysorry.. http://logs.openstack.org/06/636406/6/check/tripleo-ci-centos-7-containers-multinode/a3e8b60/job-output.txt.gz#_2019-02-12_21_40_50_34899621:55
weshaysorry may just be that patch.. my dashboard just lit up21:55
clarkbslaweq: both have been marked for deletion21:55
slaweqclarkb: thx for cleaning :)21:56
weshayhrm..21:56
clarkbweshay: http://logs.openstack.org/06/636406/6/check/tripleo-ci-centos-7-containers-multinode/a3e8b60/logs/zuul-variables.yaml is the broken file21:56
clarkbweshay: those quotes may be making it unhappy?21:57
weshayya.. agree21:57
*** zaneb has joined #openstack-infra21:58
*** gcutrini has joined #openstack-infra21:58
weshaysorry clarkb, it's just that patch21:59
openstackgerritMerged openstack-infra/zuul master: Improve event logging in githubconnection  https://review.openstack.org/63642922:02
*** markvoelker has joined #openstack-infra22:03
ianwhttps://github.com/Neilpang/acme.sh/wiki/DNS-alias-mode ... that's another interesting idea22:10
ianwbasically set _acme-challenge.openstack.org -> _acme-challenge.opendev.org ?22:11
clarkboh interesting. If that works that would greatly simplify this. We might need to coordinate with the others we share that domain with though22:13
clarkbI think they are using letsencrypt and that might cause interesting problems between the two groups :)22:13
ianwi'm doing a bit of an audit on the acme.sh rackspace plugin pull request, it seems very close to satisfying the requirements listed in the acme api dev guide22:17
*** gcutrini has quit IRC22:18
ianwif we can get that merged, it would be another option up our sleeve22:18
clarkbcorvus: moving here for planning zuul scheduler restart. puppet should update the zuul intall in the next 20 minutes or so22:18
clarkbIn this case I think we can just restart the scheduler right?22:18
clarkband are there other chagnes to make to get the github3 logs?22:19
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Fix gitea playbooks  https://review.openstack.org/63645222:21
corvusclarkb: the github3 log will be a logging config file change22:21
corvusmaybe tobiash can paste his snippet? :)22:22
openstackgerritMerged openstack-infra/system-config master: Stop running k8s-on-openstack nested  https://review.openstack.org/63585722:22
corvusclarkb, fungi, mordred: https://review.openstack.org/636452 addresses all the errors i currently see in the run all cron log22:22
* tobiash looks for his laptop22:22
clarkbthanks22:22
clarkblooks like we want to modify puppet-zuul/files/logging.conf22:23
corvusi'm going to restart zuul-web now, for a completely unrelated reason, but i don't think there's any interference between these 2 tasks22:25
clarkbso I don't forget. Almost a day later. ze08 swap usage http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64537&rra_id=all compared to ze05 http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64465&rra_id=all22:25
tobiashcorvus, clarkb: http://paste.openstack.org/show/744981/22:26
clarkbze08 is still swapping but far less. and ze05 seems like a reasonable approximation for everything else22:26
corvus#status log restarted zuul-web at commit 6d6c69f93e9755b3b812c85ffceb1b830bd75d6f22:26
openstackstatuscorvus: finished logging22:27
*** jamesmcarthur has joined #openstack-infra22:27
clarkbtobiash: thanks22:27
openstackgerritClark Boylan proposed openstack-infra/puppet-zuul master: Log github3 at DEBUG level  https://review.openstack.org/63645422:29
clarkbcorvus: tobiash ^ I think something like that should do it22:29
corvusclarkb: so the thing about puppet-zuul is that changes everyones logging config.22:32
corvushowever, +2 from me nonetheless.22:32
clarkbcorvus: ya we don't seem to have a good way to make that optional :/22:33
corvusfwiw, it's only going to affect people using zuulv3.22:34
*** slaweq has quit IRC22:34
*** lathiat has quit IRC22:34
mrhillsmano/22:34
mrhillsmanif i wanted an openstack environment running rocky that is non-devstack can i get that via infra?22:35
clarkbcorvus: zuulv3 + puppet which I think might be just us :/22:35
mrhillsmanmulti-node22:35
clarkbmrhillsman: you may need to describe this thing more, but we don't run any openstack clouds. We provide test resources that you can install openstack on (which many people do to test osa, kolla, etc)22:35
*** markvoelker has quit IRC22:36
mrhillsmantest resources are only VMs?22:36
clarkbmrhillsman: currently we have openstack VMs and an experimental k8s cluster (that probably isn't ready for real workload)22:37
clarkbmrhillsman: we could run on baremetal if we had access to such resources but no one has given us baremetal test resources22:38
mrhillsmanis nested virt available now?22:38
clarkbnot universally and it breaks fairly often in the clouds that offer it22:38
pabelangerclarkb: is there an ongoing swap issue with zuul? just ready backscroll on ze0822:38
clarkbpabelanger: its been a thing since end of november ish? it doesn't seem to impact functionality much but it is something we've tried to understand better when we can22:39
clarkbpabelanger: we switched ze08 to jemalloc since tobiash saw improvements from that and it seems like we have improvements too but still swapping22:39
mrhillsmanthx clarkb22:39
clarkbmrhillsman: nested virt is a game of catchup where guest kernels will break the hypervisor side then hypervisor side kernels update22:40
clarkbmrhillsman: unfortunately we don't have much insight to either side and sort of have to ride it out22:40
pabelangerclarkb: ack, thanks22:40
clarkbmrhillsman: what are you trying to do with this rocky cloud? and why does it matter if devstack is involved?22:40
mrhillsmanopenlab gets various requests one of which is an openstack cloud22:41
*** markmcd has quit IRC22:42
mrhillsmanwe can do that right now but not via VMs of course22:42
mrhillsmanwhich pulls us out of some of the existing automation22:42
mrhillsmanso trying to see if we have baremetal nodes (i know we would have to maintain the nodepool ip/node mappings) could we automate standing up an openstack22:43
mrhillsmanvia zuul22:43
clarkbmrhillsman: logan- is deploying the limestone test cloud from our zuul22:43
mrhillsmanbut then there are some additional caveats because the baremetal nodes are not perpetual22:43
clarkbso that is possible22:43
*** agopi has quit IRC22:43
pabelangerwe are too (ansible-network), on packet.net22:44
clarkband pabelanger is working with john studarus to do similar with ^ ya22:44
pabelangerhaven't looped it into nodepool yet22:44
mrhillsmanforgive me if i mess it up22:44
mrhillsmanthe ssh keys requirement or whatnot22:45
mrhillsmanis that relevant for static nodes/baremetal?22:45
mrhillsmanlike i have baremetal that will not always have the same IP22:45
clarkbmrhillsman: yes, zuul need to be able to connect to the test instances. The typical connection method is ssh though other ansible connection drivers work (winrm is used for windows for example)22:45
mrhillsmanare there any concerns with host verification?22:46
pabelangermrhillsman: https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/4c/4c91f44a55cd37a5c80ddf7e147ba0252465195c/post/packet-ci-cloud-deploy/0aa4dbb/ is an example of deploying OSA, from zuul post pipeline.22:46
clarkbnodepool scans hostkeys and then zuul enforces they match22:46
mrhillsmanyeah hostkeys, sorry22:47
*** sreejithp has quit IRC22:47
mrhillsmanbut for the static driver with nodes that will not always have the same IP is that relevant?22:48
clarkbif they don't have the same IP they are not very static :)22:48
mrhillsmani cannot pin an IP to a node, i ask for them, get them for a time, then they go away22:48
mrhillsmanhehe, touche22:48
clarkbhowever if you use dns and keep that updated and ensure the hostkey doesn't chagne it should mostly work22:48
mrhillsman++22:48
clarkbif you ask for them via an API you could also potentially add a new driver that spoke that api for you22:49
mrhillsmanok thx for all the info, will have to go think this through22:49
mrhillsmanthx pabelanger for that link as well22:50
*** ekultails has quit IRC22:50
*** sthussey has quit IRC22:51
*** armax has quit IRC22:53
*** wolverineav has quit IRC22:54
*** armax has joined #openstack-infra22:54
*** wolverineav has joined #openstack-infra22:55
*** wolverineav has quit IRC22:58
*** wolverineav has joined #openstack-infra22:59
clarkbcorvus: zuul is updated on zuul01 now. Do we want to do a restart with just that in place first or do you think those logging changes aren't useful without the github3 logs?22:59
clarkbAny other infra-root want to review https://review.openstack.org/#/c/636454/ ? adds github3 debug logging to zuul logs to aid in debugging github driver slowness23:00
ianw#status log for testing purposes i have registered acme-opendev.org and setup openstackci account rax clouddns with it.  this is for testing rax api integration without me worrying about wiping out openstack.org by accident23:01
openstackstatusianw: finished logging23:01
corvusianw: wait, why are you doing stuff with rax clouddns?23:03
corvusi clearly misunderstood what happened in the meeting23:03
*** gfidente|afk has quit IRC23:04
ianwcorvus: right now, i'm looking at acme.sh and it's incomplete rax dns implementation and seeing if i can get the pull request into a mergable state23:05
corvusianw: okay, but there is no future with rax dns for us.  whatever we do with opendev, it well certainly not be rax.23:05
corvusclarkb: i think restarting without the logging change will give us more information, but i'm 95% sure the next question after that will need the logging change for an answer23:08
clarkbcorvus: ok I'll wait on that change to get in then23:09
corvusianw: we're making really good progress on opendev stuff, and most of our new certs are in the opendev domain.  it has a dns system which is not (and never has been) based on rax dns.23:09
*** kjackal has quit IRC23:09
corvusianw: it's described here: http://specs.openstack.org/openstack-infra/infra-specs/specs/project-hosting.html#dns-servers23:09
corvusianw: and here: https://review.openstack.org/63356923:10
corvusianw: so any work that you do that involves rax dns will not be usable by us23:10
corvusthat's what i tried to say in the meeting23:10
corvusi thought we all understood that23:10
clarkbI think the concern is managing certs for the old names so that redirects work23:10
*** jamesmcarthur has quit IRC23:11
clarkband that is still an issue if we use a single server for that?23:11
corvussure, but how long are we going to do that?  a year or two?23:11
corvusbuy a bunch of 2 year certs at once and forget about it23:11
clarkbya we could basically draw the deprecation line and buy certs for that time period23:11
clarkbso its one time human cost23:12
corvusyeah, it's not worth spending any time on, especially since it doesn't solve the problem we have now or will have in the future23:12
*** whoami-rajat has joined #openstack-infra23:12
*** rascasoft has quit IRC23:13
corvusi'd really like us to be using letsencrypt... like... 6 months ago.  i just don't want to spend any time on stuff that isn't going to happen.23:13
*** tkajinam has joined #openstack-infra23:16
*** rascasoft has joined #openstack-infra23:16
corvusianw: also, please don't give the openstack ci rax creds to *anyone*23:19
ianwwhy would i do that?23:19
openstackgerritMerged openstack-infra/system-config master: Fix gitea playbooks  https://review.openstack.org/63645223:19
corvusianw: you said "and setup openstackci account rax clouddns with it"23:19
fungii read that as "don't supply our rax credentials in some arbitrary external service"23:20
ianwright, i didn't want someone to log into openstackci and see a domain and wonder why it was there, i'm just using it for testing acme.sh integration in essentially a sandbox away from the actual domains23:20
corvusthat sandbox has all of our production services in it23:21
mwhahahanot sure if this is a #zuul question or something for here, but has there been a thought about something like a Conflicts-With (like needed-by/depends-on) that would allow folks to say that this implementation conflicts with these review(s) and might even -2 something if one is merged23:25
ianwright now, all i want to get happening is https://github.com/Neilpang/acme.sh/pull/1999 into a mergable state.  i would like to be able to write in the spec that we have a way to do dns based acme with our status quo23:26
ianwrax dns based acme23:26
corvusianw: it's not our status quo.  *this* is our status quo: https://review.openstack.org/63356923:26
corvuswe've bought 3 opendev.org certs in the past few weeks, if i'm counting right.23:27
*** rascasoft has quit IRC23:27
corvusi don't think we've bought any new openstack.org ones recently.23:27
clarkbmwhahaha: I don't think so, but ya that would be a great #zuul question23:27
mwhahahaclarkb: k thanks, for another day23:28
clarkbmwhahaha: one way you can approximate that today is via an intentional merge conflict though that might dirty your repo content23:28
corvusmwhahaha, clarkb: in most cases actual conflicting implementations have actual merge conflicts23:29
mwhahahayea i'm just working on two competing implementations that they will probably conflict with eachother but it's multiple patches23:29
corvusgerrit tells me about them all the time23:29
mwhahahaso i don't want half of each to get merged23:29
mwhahaha:D23:29
clarkbfungi: fyi https://review.openstack.org/#/c/636454/ is next in the debugging github + zuul23:30
*** ociuhandu has quit IRC23:30
clarkbfungi: we can restart zuul scheduler once that is in to get all the logging23:30
*** ociuhandu has joined #openstack-infra23:30
*** jamesmcarthur has joined #openstack-infra23:32
*** jamesmcarthur has quit IRC23:32
ianwcorvus: i'm not suggesting this in any way for opendev.  i hear your option of "buy a cert for *openstack.org and forget about it" but i would also like us to have an option of being able to have letsencrypt certificates issued to them23:32
*** jamesmcarthur has joined #openstack-infra23:33
*** agopi has joined #openstack-infra23:33
*** markvoelker has joined #openstack-infra23:33
corvusopendev is where we need this.  we don't need this for openstack -- we're leaving that behind.  any investment in that is investment that isn't going into what we need.23:35
fungiclarkb: corvus: thanks, catching back up now. i got sucked into separate rabbit holes involving fried rice and cryptanalysis23:37
fungi(two great tastes that go great together!)23:37
fungiianw: in my opinion, if there's an openstack.org service right now we've pushed back on making https because of the undue complexity in adding an ssl cert, it can wait to get one until we move it to an opendev.org domain name23:40
fungithere's clearly no urgency, and that path also avoids accruing unwarranted tech debt23:40
ianwcorvus: i will update the spec.  it was written with barely any notice of opendev because at the time it was fairly abstract, and i don't think any review comments even really mentioned "don't bother with openstack.org" although that is clearly the current thinking23:42
*** rascasoft has joined #openstack-infra23:42
fungii think that's entirely excusable. they were happening much in parallel so it was hard for either to anticipate the other23:42
corvusianw: good point, good idea, and thanks.  let me know when you do and i'll be happy to give it a look23:43
*** sdake has joined #openstack-infra23:43
tonybI'm playing with the 'sandbox bot' as described in http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002291.html (see https://review.openstack.org/#/c/636279/2/)23:43
tonybI'm kinda at the stage where I need to add some extra pip libraries to make it work (specifically requests)23:44
tonybhttps://review.openstack.org/#/c/636279/2/playbooks/bot-reviewer.yaml23:44
corvustonyb: you're looking at using a zero-node job and running on the executors?23:44
tonybI have a place holder in ^^^ but is there another similar set of tasks I can use as a guide for doign this right?23:45
tonybcorvus: That's beyond my what I know but I *think* so as I'll need access to zuul secrets?23:45
corvustonyb: secrets can be used in any repo, you just have to trust the approvers in that repo23:46
tonybcorvus: Well eventually the bot and job config will live outside the sandbox (probably project-config) for just that reason but for now I don't need that23:47
corvustonyb: so sandbox is probably not a good place for secrets (i'm not sure who the approvers are right now, but it seems like they may change anyway)23:47
tonybcorvus: anyone witha gerrit account and +W in that repo so yeah not there ;P23:47
fungiyep, this was raised in the ml thread, so not part of the design23:47
fungisecrets will go "elsewhere" (with the elsewhere tbd, but presumably a config repo for convenience)23:48
corvustonyb: if it's in project-config, it can run on the executors with no nodes, which means it'll be fast.  the executors already have requests installed on them for python3, so that'll be easy too.  (it's a zuul dependency).23:48
*** rh-jelabarre has quit IRC23:48
corvustonyb: the downside, as you know, is that it's not dynamically self-testing.23:49
fungiwhich does make for a bit of a prototyping challenge23:49
*** wolverineav has quit IRC23:49
tonybthis is just me mucking around to get a feel for it / get as much done before I need top reply on y'all for the approvals23:49
*** wolverineav has joined #openstack-infra23:50
* tonyb wonders if at some point we want this to just be a pip installable tool but that's down the track23:50
tonyb... for now should I just add a couple of tasks to do basucally 1) create a venv ; and 2) install the packages I need into that venv .. then run the 'bot' with that python?23:52
*** rkukura has quit IRC23:52
tonybonce I have the prototype mosylt working I can just move the whole lot to a better repo23:52
corvustonyb: i left comments on that change23:53
tonybcorvus: Thanks23:53
corvustonyb: you could do what you describe, however, i think so much of this isn't going to work in the sandbox repo that i'd recommend a different approach23:53
corvustonyb: i'd go ahead and create the account for this bot, get the python working locally with those credentials, then push the whole thing up in a change to project-config23:54
corvustonyb: if you push up a working python script, between all of us, we can probably get the zuul stuff 99% working in review23:54
corvusso hopefully we only need one or two fixup changes afterword23:54
tonybcorvus: Oh okay I can do that.23:55
*** rascasoft has quit IRC23:55
*** jamesmcarthur has quit IRC23:56
*** wolverineav has quit IRC23:56
corvustonyb: and remember both gertty and zuul may be good references for using the gerrit rest api from python23:56
tonybcorvus: Thanks.  Will do23:57
openstackgerritMerged openstack-infra/puppet-zuul master: Log github3 at DEBUG level  https://review.openstack.org/63645423:57
tonybcorvus: so ... How do I get an account created?23:57
corvustonyb: here's zuul leaving file comments: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/driver/gerrit/gerritconnection.py#n84123:57
clarkbtonyb: https://docs.openstack.org/infra/system-config/third_party.html#creating-a-service-account is the documented process for third party ci accounts23:58
clarkbit likely applies here as well23:58
corvusseems reasonable23:58
tonybcorvus: thanks23:59
tonybclarkb: cool, I'll read that and do the needful23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!