Monday, 2020-03-02

*** ociuhandu has quit IRC00:09
*** ociuhandu has joined #openstack-infra00:10
*** ociuhandu has quit IRC00:20
*** ociuhandu has joined #openstack-infra00:21
*** jamesmcarthur has quit IRC00:26
*** ociuhandu has quit IRC00:31
*** ociuhandu has joined #openstack-infra00:32
*** ociuhandu has quit IRC00:42
*** ociuhandu has joined #openstack-infra00:43
*** ociuhandu has quit IRC00:53
*** ociuhandu has joined #openstack-infra00:54
*** factor has joined #openstack-infra01:00
*** ociuhandu has quit IRC01:04
*** ociuhandu has joined #openstack-infra01:05
*** Lucas_Gray has quit IRC01:09
*** ociuhandu has quit IRC01:15
*** ociuhandu has joined #openstack-infra01:16
*** yamamoto has joined #openstack-infra01:19
*** tkajinam has quit IRC01:24
*** tkajinam has joined #openstack-infra01:25
*** ociuhandu has quit IRC01:26
*** ociuhandu has joined #openstack-infra01:27
*** jamesmcarthur has joined #openstack-infra01:35
openstackgerritIan Wienand proposed opendev/system-config master: Remove files02.openstack.org and related puppet  https://review.opendev.org/70963901:38
openstackgerritIan Wienand proposed opendev/system-config master: Remove static site puppet  https://review.opendev.org/71038801:38
*** ociuhandu has quit IRC01:46
*** ociuhandu has joined #openstack-infra01:47
*** ociuhandu has quit IRC01:58
*** ociuhandu has joined #openstack-infra01:58
*** ociuhandu has quit IRC02:09
*** ociuhandu has joined #openstack-infra02:09
*** jamesmcarthur has quit IRC02:12
*** jamesmcarthur has joined #openstack-infra02:17
*** ociuhandu has quit IRC02:20
*** ociuhandu has joined #openstack-infra02:20
*** jamesmcarthur has quit IRC02:21
*** ramishra has joined #openstack-infra02:23
*** rledisez has quit IRC02:26
*** rledisez has joined #openstack-infra02:28
*** rledisez has quit IRC02:30
*** ociuhandu has quit IRC02:31
*** ociuhandu has joined #openstack-infra02:31
*** jamesmcarthur has joined #openstack-infra02:36
*** ociuhandu has quit IRC02:42
*** ociuhandu has joined #openstack-infra02:43
*** jcoufal has joined #openstack-infra02:43
*** jcoufal has quit IRC02:48
*** ociuhandu has quit IRC02:53
*** ociuhandu has joined #openstack-infra02:54
openstackgerritIan Wienand proposed opendev/system-config master: minimal-nodepool : move to nodepool-base  https://review.opendev.org/71069302:58
*** jamesmcarthur has quit IRC03:02
*** ociuhandu has quit IRC03:04
*** ociuhandu has joined #openstack-infra03:05
*** ociuhandu has quit IRC03:15
*** ociuhandu has joined #openstack-infra03:16
*** ociuhandu has quit IRC03:26
*** ociuhandu has joined #openstack-infra03:27
*** ijw has joined #openstack-infra03:33
*** psachin has joined #openstack-infra03:34
*** ociuhandu has quit IRC03:37
*** ociuhandu has joined #openstack-infra03:38
*** ijw_ has joined #openstack-infra03:41
*** ijw has quit IRC03:45
*** hongbin has joined #openstack-infra03:48
*** ociuhandu has quit IRC03:48
*** ociuhandu has joined #openstack-infra03:49
*** ykarel|away is now known as ykarel03:55
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] stashing nodepool-builder container deployment  https://review.opendev.org/71069403:57
*** raukadah is now known as chandankumar03:59
*** ociuhandu has quit IRC04:00
*** ociuhandu has joined #openstack-infra04:01
*** udesale has joined #openstack-infra04:04
*** ociuhandu has quit IRC04:11
*** ociuhandu has joined #openstack-infra04:12
*** ociuhandu has quit IRC04:22
*** ociuhandu has joined #openstack-infra04:23
*** ijw_ has quit IRC04:24
*** ociuhandu has quit IRC04:33
*** ociuhandu has joined #openstack-infra04:34
*** ociuhandu has quit IRC04:44
*** ociuhandu has joined #openstack-infra04:45
*** ociuhandu has quit IRC04:55
*** ociuhandu has joined #openstack-infra04:56
*** ociuhandu has quit IRC05:06
*** ociuhandu has joined #openstack-infra05:07
openstackgerritIan Wienand proposed opendev/puppet-openstackci master: nodepool : add manage_config flag  https://review.opendev.org/71069605:14
*** ociuhandu has quit IRC05:17
*** ociuhandu has joined #openstack-infra05:18
openstackgerritIan Wienand proposed openstack/project-config master: Remove nodepool/scripts  https://review.opendev.org/71069705:23
*** dave-mccowan has quit IRC05:24
*** ijw has joined #openstack-infra05:26
*** hongbin has quit IRC05:27
*** ociuhandu has quit IRC05:28
*** ociuhandu has joined #openstack-infra05:29
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-infra05:35
*** ociuhandu has quit IRC05:39
*** ociuhandu has joined #openstack-infra05:40
*** yamamoto has quit IRC05:49
*** ociuhandu has quit IRC05:59
*** ociuhandu has joined #openstack-infra05:59
*** yamamoto has joined #openstack-infra06:01
*** threestrands has joined #openstack-infra06:04
*** rui-zang has quit IRC06:07
*** ociuhandu has quit IRC06:10
openstackgerritIan Wienand proposed openstack/project-config master: nodepool: Add nb01-test.opendev.org config file  https://review.opendev.org/71070006:10
*** ociuhandu has joined #openstack-infra06:10
openstackgerritIan Wienand proposed opendev/system-config master: nodepool-base: manage configs via ansible  https://review.opendev.org/71070106:12
*** threestrands has quit IRC06:16
*** ociuhandu has quit IRC06:21
*** ociuhandu has joined #openstack-infra06:22
*** matt_kosut has joined #openstack-infra06:28
*** lbragstad has joined #openstack-infra06:28
*** lbragstad_ has quit IRC06:29
*** ociuhandu has quit IRC06:32
*** ociuhandu has joined #openstack-infra06:33
*** lmiccini has joined #openstack-infra06:35
openstackgerritKevin Zhao proposed openstack/project-config master: Reduce Linaro London node to 0 as test needed  https://review.opendev.org/71070606:36
*** lmiccini has quit IRC06:36
kevinzianw: ^^, pls help to review, I temporary reduce the node of the Linaro London to 0. Need to test on Linaro US06:37
ianwnp06:37
*** lmiccini has joined #openstack-infra06:38
*** ociuhandu has quit IRC06:42
*** ociuhandu has joined #openstack-infra06:43
*** dpawlik has joined #openstack-infra06:46
*** yolanda has joined #openstack-infra06:47
*** arif-ali has joined #openstack-infra06:47
*** rcernin has quit IRC06:47
*** ijw has quit IRC06:49
*** ociuhandu has quit IRC06:54
*** ociuhandu has joined #openstack-infra06:54
*** dpawlik has quit IRC06:57
*** lpetrut has joined #openstack-infra07:02
*** lpetrut has quit IRC07:03
openstackgerritMerged openstack/project-config master: Reduce Linaro London node to 0 as test needed  https://review.opendev.org/71070607:03
*** lpetrut has joined #openstack-infra07:03
*** ociuhandu has quit IRC07:04
*** ociuhandu has joined #openstack-infra07:05
*** dpawlik has joined #openstack-infra07:08
*** ociuhandu has quit IRC07:16
*** ociuhandu has joined #openstack-infra07:16
*** matt_kosut has quit IRC07:17
*** ociuhandu has quit IRC07:27
*** ociuhandu has joined #openstack-infra07:27
*** iurygregory has joined #openstack-infra07:28
*** dciabrin has joined #openstack-infra07:31
*** matt_kosut has joined #openstack-infra07:37
*** ociuhandu has quit IRC07:38
*** matt_kosut has quit IRC07:38
*** ykarel is now known as ykarel|lunch07:38
*** ociuhandu has joined #openstack-infra07:38
*** lbragstad_ has joined #openstack-infra07:40
*** matt_kosut has joined #openstack-infra07:41
*** lbragstad has quit IRC07:42
*** ociuhandu has quit IRC07:49
*** ociuhandu has joined #openstack-infra07:50
*** matt_kosut has quit IRC07:51
*** matt_kosut has joined #openstack-infra07:51
*** slaweq has joined #openstack-infra07:53
*** trident has quit IRC07:54
*** trident has joined #openstack-infra07:57
*** tesseract has joined #openstack-infra07:59
*** ociuhandu has quit IRC08:00
*** ociuhandu has joined #openstack-infra08:01
*** ociuhandu has quit IRC08:11
*** ociuhandu has joined #openstack-infra08:12
*** ccamacho has joined #openstack-infra08:13
*** jaosorior has joined #openstack-infra08:16
*** jcapitao has joined #openstack-infra08:17
*** rpittau|afk is now known as rpittau08:18
*** dychen has quit IRC08:21
*** yamamoto has quit IRC08:21
*** ociuhandu has quit IRC08:22
*** tosky has joined #openstack-infra08:23
*** jpena|off is now known as jpena08:23
*** ociuhandu has joined #openstack-infra08:23
*** ysastri has joined #openstack-infra08:23
*** tkajinam has quit IRC08:24
*** mgoddard has quit IRC08:27
*** lee1 has joined #openstack-infra08:32
*** lee1 is now known as lyarwood08:33
*** ociuhandu has quit IRC08:33
*** mgoddard has joined #openstack-infra08:33
*** ociuhandu has joined #openstack-infra08:34
*** ralonsoh has joined #openstack-infra08:34
*** andrewbonney has joined #openstack-infra08:39
*** ociuhandu has quit IRC08:44
*** ociuhandu has joined #openstack-infra08:45
*** ykarel|lunch is now known as ykarel08:50
*** gfidente has joined #openstack-infra08:51
*** lucasagomes has joined #openstack-infra08:53
*** ociuhandu has quit IRC08:55
*** ociuhandu has joined #openstack-infra08:56
*** yamamoto has joined #openstack-infra09:00
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Optimize repos for size when preparing for nodecache  https://review.opendev.org/71072009:03
*** jcapitao has quit IRC09:03
*** jcapitao has joined #openstack-infra09:04
*** yamamoto has quit IRC09:05
*** ociuhandu has quit IRC09:07
*** ociuhandu has joined #openstack-infra09:07
*** hashar has joined #openstack-infra09:13
*** ociuhandu has quit IRC09:17
*** ociuhandu has joined #openstack-infra09:18
*** dmellado has quit IRC09:23
*** dmellado has joined #openstack-infra09:26
*** ociuhandu has quit IRC09:28
*** ociuhandu has joined #openstack-infra09:29
*** apetrich has joined #openstack-infra09:30
*** dmellado has quit IRC09:33
*** dmellado has joined #openstack-infra09:35
*** jaosorior has quit IRC09:35
fricklerprometheanfire: in my local tests I can confirm that the issue is 1670, with https://github.com/pypa/virtualenv/pull/1673 things are passing09:36
AJaegerconfig-core, please review https://review.opendev.org/710652 and https://review.opendev.org/71056609:39
*** tkajinam has joined #openstack-infra09:39
*** ociuhandu has quit IRC09:40
*** ociuhandu has joined #openstack-infra09:40
*** derekh has joined #openstack-infra09:43
openstackgerritAndreas Jaeger proposed openstack/project-config master: Don't use zuul_work_dir in post-stackviz  https://review.opendev.org/71073009:45
*** ociuhandu has quit IRC09:51
*** ociuhandu has joined #openstack-infra09:51
*** hashar has quit IRC09:52
openstackgerritMerged openstack/cookiecutter master: Update the minversion parameter.  https://review.opendev.org/70739709:55
*** sshnaidm is now known as sshnaidm|pto09:59
*** yamamoto has joined #openstack-infra10:00
*** ociuhandu has quit IRC10:02
*** ociuhandu has joined #openstack-infra10:02
*** pkopec has joined #openstack-infra10:03
AJaegerconfig-core, please review https://review.opendev.org/710730 as well10:05
*** yamamoto has quit IRC10:05
*** dtantsur|afk is now known as dtantsur10:08
*** ociuhandu has quit IRC10:12
*** ociuhandu has joined #openstack-infra10:13
*** tkajinam has quit IRC10:18
*** kozhukalov has quit IRC10:21
*** kozhukalov has joined #openstack-infra10:22
*** gshippey has joined #openstack-infra10:23
*** ociuhandu has quit IRC10:23
*** ociuhandu has joined #openstack-infra10:24
*** sshnaidm|pto has quit IRC10:26
*** ociuhandu has quit IRC10:27
*** ociuhandu has joined #openstack-infra10:28
openstackgerritStephen Finucane proposed openstack/project-config master: Retire tempest-lib  https://review.opendev.org/71073610:30
openstackgerritFelix Edel proposed zuul/zuul master: Dequeue changes via github checks API  https://review.opendev.org/70913510:32
*** yamamoto has joined #openstack-infra10:37
openstackgerritMerged openstack/project-config master: Don't use zuul_work_dir in post-stackviz  https://review.opendev.org/71073010:38
*** sshnaidm has joined #openstack-infra10:42
*** yamamoto has quit IRC10:42
*** ociuhandu has quit IRC10:52
*** udesale has quit IRC10:57
openstackgerritStephen Finucane proposed openstack/project-config master: Retire tempest-lib  https://review.opendev.org/71073611:06
*** sshnaidm_ has joined #openstack-infra11:12
*** sshnaidm has quit IRC11:12
*** tetsuro has joined #openstack-infra11:12
*** sshnaidm_ is now known as sshnaidm|pto11:15
*** ociuhandu has joined #openstack-infra11:18
*** sshnaidm|pto has quit IRC11:23
*** sshnaidm has joined #openstack-infra11:23
*** sshnaidm is now known as sshnaidm|pto11:24
noonedeadpunkHey everyone. Sorry, is there still an autohold for https://zuul.opendev.org/t/openstack/build/172093f318964322867ce87a75df044a ? If yes - can you provide me with access to it? SSH key is here https://launchpad.net/~noonedeadpunk/+sshkeys11:24
noonedeadpunkIf not - can we create one?11:24
*** ociuhandu has quit IRC11:26
*** kaisers has quit IRC11:32
*** rpittau is now known as rpittau|bbl11:32
*** kaisers has joined #openstack-infra11:36
fricklernoonedeadpunk: there is a held node, 158.69.73.2511:38
noonedeadpunkfrickler: thanks11:39
fricklernoonedeadpunk: there are also four older holds for various openstack-ansible-os_manila jobs, do you still need those, too?11:39
fricklermaybe also the nodes got deleted and just zuul didn't clean up11:40
noonedeadpunkfrickler: I htink I don't11:40
noonedeadpunkI was asking to drop them iirc11:40
noonedeadpunkso yes, you can drop all of them11:40
fricklernoonedeadpunk: o.k., I'll clean those up, thx11:40
*** jcapitao is now known as jcapitao|lunch11:46
*** tkajinam has joined #openstack-infra11:47
*** tetsuro has quit IRC11:50
*** yamamoto has joined #openstack-infra11:52
*** yamamoto has quit IRC11:57
fricklermeh, 90 minutes until I start getting nodes in the integrated check queue. combined with my inability to produce proper ansible code in less than three attempts. :-/12:01
AJaegerwelcome to the club, frickler ! Unfortunately my code worked in check but changing to post, it all failed (stackviz), and I'm still iterating. It's more than three attempts already ;(12:04
fricklerAJaeger: well, yours was syntactically correct at least. I just triggered another periodic run for it12:06
*** yamamoto has joined #openstack-infra12:08
*** kaisers_ has joined #openstack-infra12:10
*** yolanda has quit IRC12:10
*** yolanda has joined #openstack-infra12:11
kaisers_Hi! I'm looking for help with an 3rdparty CI reporting issue: I've multiple CI instances reporting in multiple projects. One of these seems to have it's report messages to gerrit ignored. What could be the reason for that? The jobs run as expected and afaics the CI connects to gerrit successfully via ssh and sends it's message but these never turn up in the gerrit reviews12:13
kaisers_The other CI instances reporting in other projects use the same ssh credentials and code and work as expected.12:13
fricklerkaisers_: do you have an example where this happens? the most common cause for this is a wrongly formatted account name12:14
*** jcoufal has joined #openstack-infra12:14
kaisers_frickler yep, let me fetch12:14
kaisers_(note: That ci has been running for years, but this came up somewhere after 2020-02-14)12:15
kaisers_most recent run was here: https://review.opendev.org/#/c/697957/12:15
kaisers_i triggered that manually, saw the job complete and checked the report12:16
AJaegerthanks, frickler !12:16
kaisers_Ci seems to be happy with it's report (log snippet at http://paste.openstack.org/show/790184/)12:17
AJaeger\o/12:18
AJaegerfrickler: https://tarballs.opendev.org/openstack/stackviz/dist/ has a "-new" tarball!12:18
fricklerAJaeger: cool12:18
fricklerkaisers_: hmm, I think gerrit may disallow votes on changes that have been merged. now sure why it wouldn't respond with an error in that case, though. do you also have an example with a non-merged patch?12:19
kaisers_frickler No, that worked (at least it did in recent years). Same happens on open changes, let me check for an example...12:20
AJaegerfrickler: and tarballs are the same (just different paths), so it looks fine!12:20
kaisers_frickler https://review.opendev.org/#/c/710639/ ran yesterday, ci reported at 2020-03-02 00:39:46,70812:22
*** kozhukalov has quit IRC12:23
*** kozhukalov has joined #openstack-infra12:24
openstackgerritAndreas Jaeger proposed openstack/project-config master: Move stackviz publish in-tree  https://review.opendev.org/71075612:27
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: Cap alembic  https://review.opendev.org/71075712:28
*** jpena is now known as jpena|lunch12:36
tobiashthat was a shot in the dark but didn't work ^12:41
tobiashops, should have been in #zuul12:41
*** tkajinam has quit IRC12:45
*** dave-mccowan has joined #openstack-infra12:47
*** kozhukalov has quit IRC12:48
*** rledisez has joined #openstack-infra12:52
*** yamamoto has quit IRC12:56
*** udesale has joined #openstack-infra12:56
*** rlandy has joined #openstack-infra12:57
fricklerkaisers_: this is what I see in the gerrit log. the "1" at the end is an error code, it should be "0" for success, but it doesn't tell the actual error http://paste.openstack.org/show/790186/13:03
kaisers_frickler thanks for digging13:05
kaisers_mmh13:05
*** Lucas_Gray has joined #openstack-infra13:07
kaisers_frickler Seems some symbols have changed, i posted the client side message to compare: http://paste.openstack.org/show/790187/13:08
kaisers_dunno if the quotes should be kept13:09
kaisers_and there's a dot at the end of the message in the gerrit side log. Is that intended?13:09
kaisers_*ick* i used the earlier run's output, updated comparison at http://paste.openstack.org/show/790188/13:13
fricklerkaisers_: the format looks the same for other submissions that are succeeding. most other CIs however a) explicitly specify the project, b) reference the patch and revision instead of the commit SHA, maybe you could try that, too?13:14
*** jcapitao|lunch is now known as jcapitao13:15
*** Lucas_Gray has quit IRC13:17
fricklerkaisers_: hmm, actually a43f0666 seems to be the head of master for openstack/manila, so I'm not surprised that gerrit cannot map that to a review13:18
kaisers_frickler i can check into changing that, however this is working on the other CI instances (and was on the Manila ci) for years which makes me think it's not the core issue here. Examples for these are e.g.: https://review.opendev.org/#/c/710037/ or https://review.opendev.org/#/c/709669/ from today)13:18
kaisers_aaaah13:18
kaisers_That would explain a lot, thanks for pointing that out. I'll check into the change references...13:21
tobiashinfra-root: fyi, currently all py35 jobs are failing when tox creates the venv due to https://gitlab.com/python-devs/importlib_resources/issues/8313:21
fricklertobiash: hmm, could that actually be the same as https://github.com/pypa/virtualenv/issues/1670 ? although so far I've only seen this for py2713:23
*** yamamoto has joined #openstack-infra13:29
*** jpena|lunch is now known as jpena13:32
*** rpittau|bbl is now known as rpittau13:34
*** psachin has quit IRC13:35
*** yamamoto has quit IRC13:35
dtantsuroh, was about to ask here about this failure13:35
dtantsurkills stable/rocky for us13:35
*** aedc has joined #openstack-infra13:37
fricklerdtantsur: which one? seems those are actually different failures according to my local testing13:38
tobiashfrickler: I don't know, but the py35 issue hits us since yesterday and the py27 issue is a week old13:39
dtantsurfrickler: check this out: https://ae5941028d68fbb15248-6e5bdb3c517f39f442b10ad5d86d4a09.ssl.cf5.rackcdn.com/710766/1/check/openstack-tox-pep8/a861ed3/job-output.txt13:39
fricklerdtantsur: o.k., that's the importlib_resources bug. I can workaround that by downgrading to importlib-resources==1.0.0 in the tox venv seemingly.13:41
fricklertobiash: ^^13:41
*** toabctl has joined #openstack-infra13:41
*** yamamoto has joined #openstack-infra13:42
*** aedc_ has joined #openstack-infra13:42
*** nicolasbock has joined #openstack-infra13:42
tobiashfrickler: where does tox get installed? In the job in ensure-tox it seems it's already installed?13:42
*** aedc has quit IRC13:42
fricklerinfra-root: /me needs to step away for a bit (like 2h). https://review.opendev.org/710729 is my poc patch to fix py27, feel free to iterate or build a fix directly into our images13:43
fricklertobiash: yes, wo have a dib module that pre-installs it13:43
tobiashah ok13:44
*** tosky_ has joined #openstack-infra13:47
*** tosky has quit IRC13:47
*** tosky_ is now known as tosky13:47
*** Lucas_Gray has joined #openstack-infra13:48
*** verdurin has quit IRC13:53
*** ykarel is now known as ykarel|away14:05
*** verdurin has joined #openstack-infra14:06
*** Lucas_Gray has quit IRC14:07
*** dciabrin_ has joined #openstack-infra14:10
*** yamamoto has quit IRC14:10
*** dciabrin has quit IRC14:15
*** Goneri has joined #openstack-infra14:16
*** aedc_ has quit IRC14:17
*** aedc has joined #openstack-infra14:19
*** yamamoto has joined #openstack-infra14:22
*** yamamoto has quit IRC14:22
*** yamamoto has joined #openstack-infra14:23
*** aedc has quit IRC14:23
*** aedc has joined #openstack-infra14:23
*** bdodd has joined #openstack-infra14:25
*** yamamoto has quit IRC14:28
fungiinfra-root: just a reminder, i'm basically not around for the next few days (busy with the open cloud workshop at boston university)14:30
fungifolks here really like the idea of opendev and also in patterning some of their operations on how we run our systems, so maybe some opportunities for new collaborators14:31
corvusfungi: \o/14:31
*** eharney has joined #openstack-infra14:31
*** gfidente has quit IRC14:32
*** cdearborn has joined #openstack-infra14:42
*** zbr is now known as zbr|pto14:43
*** aedc_ has joined #openstack-infra14:53
*** aedc has quit IRC14:53
*** beekneemech is now known as bnemec15:00
*** ykarel|away is now known as ykarel15:01
*** gfidente has joined #openstack-infra15:01
*** kaisers_ is now known as kaisers_away15:04
prometheanfirefungi: do you have a review I can monitor?15:05
fungiprometheanfire: a review for what?15:06
*** kaisers_away is now known as kaisers_15:07
prometheanfirefungi: meant that for frickler :D15:07
*** kaisers_ is now known as kaisers_away15:07
prometheanfirefrickler: have a review for the virtualenv issue?15:07
*** kaisers_away is now known as kaisers_15:08
*** kaisers_ is now known as kaisers_away15:08
corvusprometheanfire: https://review.opendev.org/710729  ?15:09
tobiashcorvus: commented ^15:17
*** lpetrut has quit IRC15:18
corvusthx15:18
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources  https://review.opendev.org/71079415:19
*** ociuhandu has joined #openstack-infra15:21
*** jamesmcarthur has joined #openstack-infra15:21
corvusmordred: do you think we could run "pip install --user --upgrade virtualenv==... importlib-resources==..." in the base job and fix both problems?15:22
corvusthat would run on all python versions, even the ones that didn't require it15:23
*** yamamoto has joined #openstack-infra15:23
*** yamamoto has quit IRC15:23
prometheanfirecorvus: thanks15:24
*** yamamoto has joined #openstack-infra15:24
*** kaisers_away is now known as kaisers_15:27
*** kaisers_ is now known as kaisers_away15:27
*** kaisers_away is now known as kaisers_15:27
*** jamesmcarthur has quit IRC15:27
*** kaisers_ is now known as kaisers_away15:28
*** jamesmcarthur has joined #openstack-infra15:28
*** yamamoto has quit IRC15:28
*** goneri_ has joined #openstack-infra15:28
*** goneri_ has quit IRC15:28
*** aedc_ has quit IRC15:32
*** aedc_ has joined #openstack-infra15:32
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources  https://review.opendev.org/71079415:34
*** kaisers_away is now known as kaisers_15:35
*** kaisers_ is now known as kaisers_away15:35
*** chandankumar is now known as raukadah15:36
*** matt_kosut has quit IRC15:38
*** kaisers_away is now known as kaisers_15:39
*** kaisers_ is now known as kaisers_away15:39
*** sshnaidm|pto has quit IRC15:41
*** kaisers_away is now known as kaisers_15:43
*** kaisers_ is now known as kaisers_away15:44
*** aedc_ has quit IRC15:45
*** kaisers_away is now known as kaisers_15:46
*** jamesmcarthur has quit IRC15:46
*** kaisers_ is now known as kaisers_away15:46
*** aedc has joined #openstack-infra15:48
*** kaisers_away is now known as kaisers_15:51
*** kaisers_ is now known as kaisers_away15:51
*** kaisers_away is now known as kaisers_15:52
*** kaisers_ is now known as kaisers_away15:52
*** kaisers_away is now known as kaisers_15:54
*** kaisers_ is now known as kaisers_away15:54
*** lpetrut has joined #openstack-infra15:55
*** tbachman has joined #openstack-infra15:55
*** sshnaidm|pto has joined #openstack-infra15:55
*** nicolasbock has quit IRC15:57
*** dmellado has quit IRC15:57
*** kaisers_away is now known as kaisers_15:58
*** kaisers_ is now known as kaisers_away15:59
*** kaisers_away is now known as kaisers_15:59
*** kaisers_ is now known as kaisers_away15:59
*** dmellado has joined #openstack-infra16:00
*** jamesmcarthur has joined #openstack-infra16:04
*** kaisers_away is now known as kaisers_16:04
clarkbcorvus: I think that would be ok as presumably the bug will be fixed then we can stop doing it for all versions at some point16:04
*** kaisers_ is now known as kaisers_away16:04
*** jamesmcarthur has quit IRC16:04
*** jamesmcarthur has joined #openstack-infra16:05
*** matt_kosut has joined #openstack-infra16:05
*** lpetrut has quit IRC16:06
*** ociuhandu has quit IRC16:07
*** ociuhandu has joined #openstack-infra16:08
*** ociuhandu has quit IRC16:09
*** ociuhandu has joined #openstack-infra16:10
*** kaisers_away is now known as kaisers_16:11
*** kaisers_ is now known as kaisers_away16:11
clarkbtobiash: I don't think upgrade is necessary if you set a specific version to install instead16:12
*** jcapitao is now known as jcapitao_afk16:12
fricklercorvus: prometheanfire: next iteration on https://review.opendev.org/710729 , I'd also support the idea to move both this and the importlib cap into the base job16:12
*** mattw4 has joined #openstack-infra16:12
clarkbtobiash: confirmed at least with virtualenv installation16:12
tobiashclarkb: ah ok16:13
prometheanfirefrickler: yep, watching that16:13
prometheanfireI think moving it to base may be warranted16:13
*** kaisers_away is now known as kaisers_16:14
*** kaisers_ is now known as kaisers_away16:15
*** gyee has joined #openstack-infra16:15
frickleram I right in understanding that there is no way to promote a patch within the integrated check queue? we're at over 2h there now16:15
fricklerkaisers_away: it would would be cool if you could stop this nickname-change-spam16:15
*** kaisers_away is now known as kaisers_16:15
*** kaisers_ is now known as kaisers_away16:16
corvuskaisers_, kaisers_away: https://bisqwit.iki.fi/jutut/away.html may be helpful16:16
*** kaisers_away is now known as kaisers_16:16
*** kaisers_ is now known as kaisers_away16:16
*** dmellado has quit IRC16:18
corvusfrickler: correct, promote is not useful in check16:18
fungipromote is relevant to dependent pipeline queues16:19
funginot independent pipelines16:19
*** ysastri has quit IRC16:19
funginor to supercedent pipelines, i suppose16:20
*** dmellado has joined #openstack-infra16:20
clarkbcatching up on things from last week: I have confirmed no new cert expiry emails for review.o.o.16:20
fungiclarkb: i saw the same. awesome work everyone16:20
clarkband looks like ianw cleaned up qa.openstack.org's dns records16:21
fricklerbut wouldn't that be a useful feature? like when I want this urgent gate fix patch to be checked before the 50 other integrated queue patches waiting for nodes in the check queue?16:21
corvusfrickler: if you want to speed up 710729 you could do so by dequeueing 71024716:21
corvusfrickler: if that's the case, put it in gate?16:21
fungifrickler: i think we would need a way to reorder the event queue to make that happen like you're describing16:22
fungiand possibly also reorder node requests in zk?16:22
fungimy attention is split so i'm not able to noodle on it effectively16:22
corvusthe node request priority is the main relevant factor for how long we wait for a job to start16:23
*** ykarel is now known as ykarel|away16:23
fricklerhmm, manually putting it into gate tentatively without +Wing might be a solution, yes16:23
fungibut yeah, without insisting on "clean check" you can just approve a change and throw it straight against the gate to see if it will stick16:23
fungior use zuul enqueue to do the same manually16:24
corvusit sounds like maybe what you want is to tweak priority of individual changes;  there isn't a facility for that, but i don't think adding support for that in zuul would be too hard16:24
* frickler needs to afk again for a bit16:24
fungiand then in the gate you can of course promote it within any shared queue (though that still doesn't reprioritize it relative to other dependent queues)16:24
*** lmiccini has quit IRC16:32
AJaegerconfig-core, please review https://review.opendev.org/710652 and https://review.opendev.org/710566 , two cleanup changes (retire faafo, remove legacy file)16:37
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources  https://review.opendev.org/71079416:38
*** kaisers_away is now known as kaisers_16:38
*** igordc has joined #openstack-infra16:39
kaisers_frickler corvus all sry, new installation and i didn't get to all the defaults so far16:39
*** diablo_rojo has quit IRC16:40
*** diablo_rojo has joined #openstack-infra16:40
corvuskaisers_: no worries, thanks for looking into it :)16:40
kaisers_should be off now16:41
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources in reqs  https://review.opendev.org/71082616:44
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources in reqs  https://review.opendev.org/71082616:46
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources both places  https://review.opendev.org/71082816:46
*** dmellado has quit IRC16:46
*** jcapitao_afk is now known as jcapitao16:48
*** dmellado has joined #openstack-infra16:49
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources both places  https://review.opendev.org/71082816:49
*** ociuhandu has quit IRC16:51
*** ociuhandu has joined #openstack-infra16:51
johnsomHi infra folks. Point me in the right direction if this isn't the right channel to ask in. In the zuul-jobs "tox" task, we are seeing a new p27->p3 related failure. Our queens jobs (python27) have started failing with:  TASK [tox : Run tox without tests] ImportError: cannot import name 'ContextManager'16:54
johnsomIt appears that somehow python3 is being used for that task now.16:54
johnsomhttps://zuul.opendev.org/t/openstack/build/71a96dc7f7fe4e87a689bec702f84bd5/log/job-output.txt#163116:54
johnsomThis is the example patch: https://review.opendev.org/#/c/709569/16:55
clarkbjohnsom: virtuaelnv and tox are running under python3 there. The errors happens creating a python2 environment. The upstream bug is here https://gitlab.com/python-devs/importlib_resources/issues/8316:55
johnsomSomeone also reported that rocky is now broken as well16:55
clarkbthere is no python2 to python3 transition. Its a failure in the tool which is running under python316:56
johnsomAh, neat. Yeah, but that wasn't running under python3 until Ussuri right? Queens was pure python27.16:56
clarkbjohnsom: there are a couple changes like https://review.opendev.org/710794 and https://review.opendev.org/710729 that are being evaluated to work around this by downgrading importlib-resources16:56
clarkbjohnsom: you're conflating the version of python the tests run under with the version of python used to create a virtualenv16:57
clarkbthey are independent16:57
clarkbyou can use python2 virtualenv to make python2 or python3 environments and python3 virtualenv to make python2 or python3 virtualenvs16:57
clarkbthen you run your tests within the desired environment16:58
johnsomNo, I know the difference, but still we didn't create the virtualenv on python3 back then.  Doesn't matter though.16:58
johnsomit was python2 and python2 back in the queens timeframe16:58
clarkbright that depends on the image, its a separate thing and I think that transitioned about 5 months ago?16:58
clarkbsomething like that16:58
clarkbunrelated to this change other than that importlib-resources is unhappy on python316:59
fungiin november we switched to python3 being the default python for bionic nodes16:59
*** rpittau is now known as rpittau|afk16:59
fungiand installing tox with python3 there16:59
johnsomOk, so what I hear is we need to wait on a fix. I didn't see a mailing list for this, how best should we track this to stop the recheck madness?16:59
fungibut it also started showing up as a similar issue on xenial nodes in january, there was a new thread on the ml about it but i don't remember a resolution17:00
fungihttp://lists.openstack.org/pipermail/openstack-discuss/2020-January/011917.html17:00
clarkbjohnsom: I'm still trying to catch up on it myself. I know prometheanfire was looking at it over the weekend and frickler has been investigating workarounds17:00
clarkbok cool there is an email I need ot read :)17:00
*** dmellado has quit IRC17:01
clarkboh thats january's thread17:01
fungiyeah17:01
fungii don't remember any resolution to it though17:01
johnsomYeah, I think those all got fixed to by python2 on python2 again. This is definitely a new/different issue.17:01
clarkbyes it is a new issue.17:01
fungiit also refers to the thread about the behavior change on our bionic images in november17:01
clarkbhttps://gitlab.com/python-devs/importlib_resources/issues/83 is the bug17:01
fungiyeah, from what i understand the issue over the weekend is that newest tox can no longer work with python2.7 or 3.517:02
clarkbfungi: well its specifically a dep of virtualenv17:02
fungiyeah17:02
clarkbthat requires something that is new in python3.617:03
fungier, right, not newest tox but new release of one of tox's deps17:03
clarkbI'll start a new thread for this issue now that I think I've got a bit of it paged in17:04
johnsomclarkb Awesome, thanks17:04
*** dmellado has joined #openstack-infra17:07
openstackgerritJames E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources in three places  https://review.opendev.org/71083317:08
*** ociuhandu_ has joined #openstack-infra17:10
clarkbok email sent17:12
*** ociuhandu has quit IRC17:13
*** ociuhandu_ has quit IRC17:14
*** tesseract has quit IRC17:16
*** dmellado has quit IRC17:19
*** ijw has joined #openstack-infra17:19
*** udesale has quit IRC17:24
*** dmellado has joined #openstack-infra17:25
*** pkopec has quit IRC17:28
*** lucasagomes has quit IRC17:30
*** pkopec has joined #openstack-infra17:31
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-infra17:35
clarkbinfra-root I've followed up with roman_g via email with nodepool logs showing the "Detailed node error: No valid host was found. There are not enough hosts available." nova errors.17:37
clarkbmostly just a heads up that the issue persists there and appaers to be cloud side problem (not our issue, but trying to help them debug)17:37
*** jpena is now known as jpena|off17:39
*** lbragstad_ is now known as lbragstad17:42
*** igordc has quit IRC17:52
*** jcapitao is now known as jcapitao_off17:54
*** derekh has quit IRC18:05
*** nicolasbock has joined #openstack-infra18:13
*** igordc has joined #openstack-infra18:15
*** iurygregory has quit IRC18:19
*** ijw_ has joined #openstack-infra18:35
*** KeithMnemonic has joined #openstack-infra18:35
*** ijw has quit IRC18:37
*** ralonsoh has quit IRC18:42
*** ijw has joined #openstack-infra18:47
*** jcapitao_off has quit IRC18:49
*** ijw_ has quit IRC18:50
openstackgerritJames E. Blair proposed zuul/zuul master: Fix py35 by pinning importlib-resources  https://review.opendev.org/71082818:54
*** apetrich has quit IRC18:56
openstackgerritMerged openstack/openstack-zuul-jobs master: Remove legacy run-jsbuild.sh  https://review.opendev.org/71056618:57
openstackgerritMerged openstack/project-config master: Retire openstack/faafo  https://review.opendev.org/71065218:58
openstackgerritClark Boylan proposed opendev/base-jobs master: Pin importlib-resources to fix virtualenv and tox  https://review.opendev.org/71085119:02
clarkbcorvus: infra-root fyi ^19:02
Shrewsclarkb: i assume "become: true" is the same as "become: yes"  (we seem to use "yes" everywhere else there)19:04
clarkbShrews: ya, I took it from corvus zuul changes but aiui they are equivalent19:04
*** dtantsur is now known as dtantsur|afk19:04
Shrewsis there a need to do that for py2? seems like we could skip that part19:04
clarkbShrews: yes aiui the bug applies to python3 and python2 installed virtualenv19:05
clarkbso centos 7 nodes need the else case there19:05
Shrewsah19:05
*** apetrich has joined #openstack-infra19:05
clarkbhttps://github.com/pypa/virtualenv/blob/20.0.7/setup.cfg#L48 applies to python2 and 319:05
clarkbso we'll get newer importlib-resources on both and it should break on both19:06
*** sreejithp has joined #openstack-infra19:14
*** ijw has quit IRC19:18
*** ijw_ has joined #openstack-infra19:18
openstackgerritClark Boylan proposed opendev/base-jobs master: Pin importlib-resources to fix virtualenv and tox  https://review.opendev.org/71085119:24
clarkbAJaeger: Shrews ^ thank you for the reviews. I had to update it due to the unbalanced ' in comment :/19:24
clarkbI also switched become: true to become: yes for consistency19:25
*** gfidente is now known as gfidente|afk19:32
clarkbjohnsom: https://review.opendev.org/710851 will land in about 5 minutes. Can you retry some of your unhappy jobs once that is merged?19:35
johnsomclarkb Yep, NP19:36
clarkbifnra-root it just occurred to me that 710851 does the pip install before updating mirrors19:36
clarkbshould I push a new patchset to move it after mirror config?19:36
clarkbinfra-root ^19:36
openstackgerritClark Boylan proposed opendev/base-jobs master: Pin importlib-resources to fix virtualenv and tox  https://review.opendev.org/71085119:38
clarkbok decided to go ahead and address that early rather than risk failures talking to pypi19:38
clarkbjohnsom: ^ fyi will be a little longer19:39
clarkbmordred: AJaeger Shrews ^ fyi19:39
johnsomAck19:39
*** jamesmcarthur has quit IRC19:40
*** jamesmcarthur has joined #openstack-infra19:40
*** pkopec has quit IRC19:40
*** pkopec has joined #openstack-infra19:41
mordredclarkb: we need to update that base job to move start-zuul-console earlier ... we keep growing tasks before it - and it makes the logs ugly at the top19:43
mordrednot in that patch, obvs19:43
clarkbmordred: let me push that up next19:43
clarkbmordred: also I think we can remove your earlier temporary workaround19:43
clarkbI'll get those up shortly (as I already have the code open and paged in)19:44
mordredcool19:44
clarkbactually those might be the same change :)19:44
mordred:)19:44
*** jamesmcarthur has quit IRC19:45
EmilienMclarkb: quick question on https://review.opendev.org/#/c/710851/3/playbooks/base/pre.yaml19:45
EmilienMI randomly reviewied it tbh19:45
EmilienMand hi all btw :)19:46
openstackgerritClark Boylan proposed opendev/base-jobs master: Remove temporary workaround for virtualenv install behavior  https://review.opendev.org/71085819:47
clarkbEmilienM: responded19:49
EmilienMclarkb: cool; i was just curious19:50
EmilienMi'm not a fan of shelling ansible tasks in general but you seem to have a valid reason19:50
clarkbEmilienM: the docs don't document the behavior either so I am wary of relying on it19:51
clarkbinstead I'll do what I really mean explicitly :)19:51
mordredEmilienM: I prefer shelling ansible tasks - but I think we all know I'm crazy19:52
clarkbah its documented in a note not on the actual parameter19:52
clarkbit uses the ansible python version19:52
clarkbwhich is actually likely to be wrong in our case19:52
EmilienMmordred: yeah let's delete all these modules and write shell :D19:52
mordredEmilienM: to be fair- it's really good at doing that :)19:52
clarkbbecause if you set older ansible in the zuul config for a job then we fall back to python2 iirc19:53
EmilienMhttps://github.com/EmilienM/Handsible19:53
mordredEmilienM: hah19:53
EmilienMi still need to finish that project19:53
clarkbyup we have non zero number of jobs that set ansible_version: to older ansible19:56
clarkbI believe this fix would not fix them if using the pip module behavior19:56
clarkb(its possible those jobs don't need the fix though)19:56
*** tbachman has quit IRC19:56
clarkbI'm going to find lunch now. I expect that chagne is totally safe to land given the failed_when: false but I'm happy to hit +A if others review it20:08
*** tbachman has joined #openstack-infra20:09
openstackgerritMerged zuul/zuul master: Fix py35 by pinning importlib-resources  https://review.opendev.org/71082820:11
*** tbachman has quit IRC20:13
*** derekh has joined #openstack-infra20:14
Shrewsclarkb: I +2'd it, but I'll let you +A when you return from lunch20:17
*** tbachman has joined #openstack-infra20:19
*** ijw has joined #openstack-infra20:20
*** derekh has quit IRC20:23
*** ijw_ has quit IRC20:23
*** ijw_ has joined #openstack-infra20:23
*** eharney has quit IRC20:24
clarkbk20:25
*** ijw has quit IRC20:25
*** matt_kosut has quit IRC20:37
*** nicolasbock has quit IRC20:38
*** matt_kosut has joined #openstack-infra20:39
*** ijw_ has quit IRC20:39
*** ijw has joined #openstack-infra20:40
clarkbinfra-root I have approved the importlib-resources pin change in base-jobs20:41
mordredclarkb: kk20:41
clarkbI'll be monitoring it but please call out any unexpected behaviors if you see them20:41
*** tbachman has quit IRC20:43
*** michael-beaver has joined #openstack-infra20:52
openstackgerritMerged opendev/base-jobs master: Pin importlib-resources to fix virtualenv and tox  https://review.opendev.org/71085120:53
clarkbjohnsom: ^ ok any jobs starting after now should use the fix20:53
clarkbjohnsom: want to start your jobs and let us know if they are happy now?20:53
johnsomI just saw that20:53
johnsomFiring up two. Should be about 5-10 minutes for a result20:55
*** tbachman has joined #openstack-infra20:56
clarkbhttp://paste.openstack.org/show/790220/ it works in the general case20:58
*** jamesmcarthur has joined #openstack-infra20:59
clarkbwhether or not it proeprly fixes all the failures yet to be determined20:59
*** njohnston_ is now known as njohnston21:02
clarkbjohnsom: should i be looking at the tox jobs for 709569?21:04
johnsomclarkb Yes21:04
*** sreejithp_ has joined #openstack-infra21:05
johnsom891 Has a different use case where we use a virtualenv as part of DIB21:05
clarkbjohnsom: for that one you may need tofix that in your elements if it is part of the chroot21:05
clarkblooks like pep8 at least is happy21:06
johnsomclarkb yep21:06
johnsomclarkb functional tests passed on 569, so yes, looks like the tox case is resolved21:06
clarkbcool I'll follow up to the email thred and point out that nested virtualenvs (as in the case of a virtualenv in dib chroot) may need separate fixing21:07
*** sreejithp has quit IRC21:07
*** Tengu has quit IRC21:13
*** tobiash has quit IRC21:13
*** tobiash has joined #openstack-infra21:15
*** Tengu has joined #openstack-infra21:15
*** mattw4 has quit IRC21:18
openstackgerritJames E. Blair proposed opendev/system-config master: DNM: Test: Use explicit provides/requires for container jobs  https://review.opendev.org/71053221:18
*** eharney has joined #openstack-infra21:25
*** ahosam has joined #openstack-infra21:42
openstackgerritClark Boylan proposed opendev/system-config master: WIP Upgrade gitea to master  https://review.opendev.org/70580821:47
clarkbmordred: ^ I rebased that change to see if we can learn anything new about deploying master21:47
*** mattw4 has joined #openstack-infra21:48
*** tbachman has quit IRC21:50
*** ahosam has quit IRC21:51
*** imacdonn has quit IRC22:03
*** imacdonn has joined #openstack-infra22:04
mordredcool22:05
openstackgerritMerged zuul/zuul master: Refactor branch protection test infrastructure  https://review.opendev.org/70983422:07
*** slaweq has quit IRC22:09
*** slaweq has joined #openstack-infra22:11
*** slaweq has quit IRC22:16
*** mattw4 has quit IRC22:17
openstackgerritMerged zuul/zuul master: Test canMerge also with checks api  https://review.opendev.org/70983522:19
*** mattw4 has joined #openstack-infra22:22
*** tkajinam has joined #openstack-infra22:28
openstackgerritMerged zuul/zuul master: Deprecate -d switch for running in foreground  https://review.opendev.org/70518522:31
*** kaisers1 has joined #openstack-infra22:33
*** slaweq has joined #openstack-infra22:33
*** kaisers has quit IRC22:35
*** igordc has quit IRC22:38
*** jcoufal has quit IRC22:38
*** slaweq has quit IRC22:39
*** slaweq has joined #openstack-infra22:41
*** sreejithp_ has quit IRC22:44
*** dchen has joined #openstack-infra22:45
*** slaweq has quit IRC22:46
*** bdodd has quit IRC22:53
*** bdodd has joined #openstack-infra22:54
*** auristor has quit IRC22:57
*** jcoufal has joined #openstack-infra22:58
*** auristor has joined #openstack-infra22:58
*** rcernin has joined #openstack-infra23:01
*** jcoufal has quit IRC23:11
*** slaweq has joined #openstack-infra23:11
*** jamesmcarthur has quit IRC23:16
*** slaweq has quit IRC23:16
*** gfidente|afk has quit IRC23:21
*** michael-beaver has quit IRC23:22
*** eharney has quit IRC23:25
mordredclarkb: the gitea master patch worked!23:30
clarkbcool so I guess we just have to double check templates and probably don't want to merge something that says "master" as that moves (and templates may update)23:30
clarkbbut if we wanted to we could set it to a specific revision and then use that maybe?23:30
clarkbthere is a bug in the docker container log collection we should fix first too23:31
clarkbI'm looking at that next23:31
*** Tengu has quit IRC23:35
*** Tengu has joined #openstack-infra23:36
openstackgerritClark Boylan proposed opendev/system-config master: Collect docker logs as root  https://review.opendev.org/71088523:38
clarkbmordred: ^ I think that should fix it23:38
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: config: add support for %(var)s syntax  https://review.opendev.org/71088623:39
ianwmordred / clarkb : if you have a second for https://review.opendev.org/#/c/710697/ and https://review.opendev.org/#/c/710700/1 it adds a sample config file to project-config for nodepool via ansible gate testing23:43
ianwwe *can* get project-config speculatively deployed for gate testing; it will involve copying it from the executor to the nodepool testing host and having it know to look for that.  it's just not my first priority to do right now23:44
clarkbianw: I think project-config is a trusted project?23:47
clarkbwhcih means it won't be updated in the zuul git contexts23:47
ianwhrm ... yeah23:48
openstackgerritIan Wienand proposed opendev/system-config master: [wip] add ansible nodepool-builder  https://review.opendev.org/71088823:51
ianwclarkb: ^ that's the change that will use it.  i put up some stuff yesterday that tried to keep a common role for the puppet & ansible hosts, but it requires puppet hacking and i think ^ is just much more workable23:52
*** tosky has quit IRC23:57
openstackgerritMerged openstack/project-config master: Remove nodepool/scripts  https://review.opendev.org/71069723:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!