Tuesday, 2018-11-06

*** ansmith has joined #openstack-infra00:00
*** eernst has joined #openstack-infra00:07
*** eernst has quit IRC00:08
*** eernst has joined #openstack-infra00:18
*** jamesmcarthur has joined #openstack-infra00:20
*** eernst has quit IRC00:20
*** jamesmcarthur has quit IRC00:24
*** gyee has quit IRC00:25
*** jamesmcarthur has joined #openstack-infra00:32
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: [wip] validate-host: retry network tests and include unbound logs  https://review.openstack.org/61569800:35
*** jamesmcarthur has quit IRC00:36
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: [wip] validate-host: retry network tests and include unbound logs  https://review.openstack.org/61569800:39
fungiianw: it seems gary_perkins is either up very late or very early, but either way we should be able to retry the armci cloud config without insecure: true00:53
*** Swami has quit IRC00:56
ianwfungi: and done! LGTM :)00:56
ianwfungi: i think it's ready to go?  if you want to merge it, i can look at bringing up a mirror when it's all settled down with the cloud-launcher00:57
*** felipemonteiro has joined #openstack-infra00:59
fungiagreed, i just spent a moment to give it one more time over first00:59
fungiand approved00:59
clarkbfollow up on the zuul usage stuff from before. tripleo is down to ~47.7% usage (which includes the same time period as before and the time since then) so I think the changes they are making are working01:01
openstackgerritIan Wienand proposed openstack-infra/project-config master: [dnm] testing new network validation  https://review.openstack.org/61570301:01
clarkbits still the biggest consumer by far, but is now letting others use the resources which is nice01:01
*** longkb has joined #openstack-infra01:03
clarkbalso yay more cloud regions01:04
*** jamesmcarthur has joined #openstack-infra01:08
*** rlandy has quit IRC01:09
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: [wip] validate-host: retry network tests and include unbound logs  https://review.openstack.org/61569801:13
*** felipemonteiro has quit IRC01:15
openstackgerritMerged openstack-infra/system-config master: Add Arm64 CI cloud  https://review.openstack.org/60243601:21
fungiianw: ^ it's in01:22
*** rkukura has quit IRC01:36
*** rh-jelabarre has joined #openstack-infra01:37
*** xinliang has joined #openstack-infra01:39
*** xinliang has quit IRC01:42
ianw\o/01:43
*** jamesmcarthur has quit IRC01:44
*** hongbin has joined #openstack-infra01:56
*** annp has quit IRC01:58
*** longkb has quit IRC01:58
*** longkb has joined #openstack-infra01:58
*** annp has joined #openstack-infra01:58
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: [wip] validate-host: retry network tests and include unbound logs  https://review.openstack.org/61569802:06
openstackgerrit98k proposed openstack-dev/hacking master: Don't quote {posargs} in tox.ini  https://review.openstack.org/60919202:09
*** ianychoi has quit IRC02:10
*** ianychoi has joined #openstack-infra02:10
*** hongbin has quit IRC02:11
openstackgerrit98k proposed openstack-infra/python-storyboardclient master: Don't quote {posargs} in tox.ini  https://review.openstack.org/60917602:14
*** nicolasbock has quit IRC02:17
*** masayukig[m] has quit IRC02:17
*** toabctl has quit IRC02:19
*** toabctl has joined #openstack-infra02:23
*** emerson has quit IRC02:23
*** hongbin has joined #openstack-infra02:29
dmsimardianw: can you explain with context ? not picking it up from the backscroll02:30
*** diablo_rojo has quit IRC02:33
*** mrsoul has joined #openstack-infra02:46
ianwdmsimard: oh, just wondering about the best way to get long-form things into the ansible results, particularly for display in ara02:54
dmsimardwhat kind of long-form things ?02:55
ianwi guess ara_record ... a log file02:55
ianwanyway, i'm just dumping it in the json with with open(f) as f: f.read() and it's ok02:55
ianwbut it could probably be better :)02:55
dmsimardianw: https://review.openstack.org/#/c/580256/02:55
ianwahh02:56
dmsimardclarkb: I might help the tripleo guys a little bit02:57
dmsimardianw: the thing I like about ara_record is the notion of type02:57
dmsimardif I had the time I would add in things like markdown and rst02:58
dmsimardianw: but wait02:59
dmsimardianw: why store the file in ara at all ?02:59
dmsimardcould you compute the link at which the file would end up and link to the file instead ?02:59
ianwdmsimard: probably it could become a normal artifact, but we only want to collect it on the odd error path03:02
*** agopi_ has joined #openstack-infra03:03
*** agopi has quit IRC03:06
*** eernst has joined #openstack-infra03:06
*** lbragstad has quit IRC03:16
*** eernst has quit IRC03:23
*** yamamoto has joined #openstack-infra03:30
*** bhavikdbavishi has joined #openstack-infra03:33
*** bhavikdbavishi has quit IRC03:35
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: validate-host: retry network tests and include unbound logs  https://review.openstack.org/61569803:38
ianwclarkb: ^ i think that's ok03:38
*** cburgess has quit IRC03:41
*** cburgess has joined #openstack-infra03:41
*** jamesmcarthur has joined #openstack-infra03:45
*** jamesmcarthur has quit IRC03:49
*** ykarel has joined #openstack-infra04:20
*** janki has joined #openstack-infra04:28
*** yamamoto has quit IRC04:34
*** yamamoto has joined #openstack-infra04:50
*** armax has quit IRC04:54
*** udesale has joined #openstack-infra04:55
*** armax has joined #openstack-infra04:55
*** udesale has quit IRC04:58
*** udesale has joined #openstack-infra04:58
*** udesale has quit IRC04:58
ianwok, i wonder what's up with all-clouds.yaml, i'm not seeing the new armci in there and it's been  awhile05:02
*** udesale has joined #openstack-infra05:05
ianwahh ... AnsibleUndefinedVariable: 'dict object' has no attribute openstackci_arm64ci_project_name05:06
*** hongbin has quit IRC05:09
ianwok ... typos in hiera file.  should be ok next run05:11
*** jtomasek has joined #openstack-infra05:14
*** agopi_ has quit IRC05:22
*** felipemonteiro has joined #openstack-infra05:28
*** ramishra has joined #openstack-infra05:29
*** rascasoft has quit IRC05:32
*** rh-jelabarre has quit IRC05:32
*** rascasoft has joined #openstack-infra05:35
*** udesale has quit IRC05:40
*** carl_cai has joined #openstack-infra05:52
ianwyay, that got it, watching cloud launcher now to get the control plane image in05:53
*** udesale has joined #openstack-infra05:53
openstackgerritMerged openstack-infra/zuul master: Merger: automatically add new hosts to the known_hosts file  https://review.openstack.org/60845306:03
*** kjackal has joined #openstack-infra06:10
*** jamesmcarthur has joined #openstack-infra06:16
*** jamesmcarthur has quit IRC06:20
*** e0ne has joined #openstack-infra06:24
*** jtomasek has quit IRC06:39
*** felipemonteiro has quit IRC06:43
*** e0ne has quit IRC06:44
*** e0ne has joined #openstack-infra06:48
*** quiquell|off is now known as quiquell06:49
*** longkb has quit IRC06:52
*** longkb has joined #openstack-infra06:53
*** udesale has quit IRC06:54
*** dayou has quit IRC07:07
*** e0ne has quit IRC07:07
*** dayou has joined #openstack-infra07:07
*** maciejjozefczyk has joined #openstack-infra07:13
*** AJaeger_ has quit IRC07:15
*** dpawlik has joined #openstack-infra07:17
*** AJaeger has joined #openstack-infra07:26
*** udesale has joined #openstack-infra07:35
*** pcaruana has joined #openstack-infra07:36
*** adriancz has joined #openstack-infra07:42
*** aojea has joined #openstack-infra07:45
*** quiquell is now known as quiquell|brb07:50
*** florianf has joined #openstack-infra08:00
*** ccamacho has joined #openstack-infra08:03
*** dayou_ has joined #openstack-infra08:03
*** dayou has quit IRC08:04
*** kopecmartin|off is now known as kopecmartin08:06
*** tdasilva has quit IRC08:07
*** ralonsoh has joined #openstack-infra08:07
*** ginopc has quit IRC08:11
*** quiquell|brb is now known as quiquell08:14
*** ginopc has joined #openstack-infra08:14
*** ramishra has quit IRC08:15
*** ykarel_ has joined #openstack-infra08:19
*** ykarel has quit IRC08:22
*** ramishra has joined #openstack-infra08:24
*** dpawlik has quit IRC08:28
*** dpawlik has joined #openstack-infra08:29
*** rtjure has joined #openstack-infra08:33
*** dpawlik has quit IRC08:33
*** dpawlik has joined #openstack-infra08:35
*** priteau has joined #openstack-infra08:39
*** jtomasek has joined #openstack-infra08:41
*** dpawlik has quit IRC08:42
*** dpawlik has joined #openstack-infra08:43
*** jpena|off is now known as jpena08:47
*** jpich has joined #openstack-infra08:57
*** betherly has joined #openstack-infra09:01
*** markmcd has quit IRC09:05
ianwmordred: cloud-launcher is broken :/  can you take a look @ https://storyboard.openstack.org/#!/story/2004267 ?  the first problem, removing the snat router, seems to be fixed by devel version of ansible.  now it's having issues with Lon1 in citycloud for openstackzuul09:11
*** ykarel__ has joined #openstack-infra09:11
ianwmordred: i'm out of time, but if we want to pin different versions of ansible on bridge I'd suggest we take my change to do that with variables in https://review.openstack.org/61489409:11
*** ykarel_ has quit IRC09:12
ianwfungi / gary_perkins : ^ this is why armci cloud does not have a mirror :)09:12
*** ykarel__ is now known as ykarel09:19
*** priteau has quit IRC09:20
*** gfidente has joined #openstack-infra09:20
*** Emine has joined #openstack-infra09:20
*** markmcd has joined #openstack-infra09:21
*** maciejjozefczyk has quit IRC09:22
*** rossella_s has quit IRC09:23
*** rossella_s has joined #openstack-infra09:23
*** priteau has joined #openstack-infra09:23
*** annp has quit IRC09:24
*** derekh has joined #openstack-infra09:39
*** sshnaidm|afk is now known as sshnaidm|rover09:45
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Improve windows logging  https://review.openstack.org/61580409:53
*** lbragstad has joined #openstack-infra10:00
openstackgerritMerged openstack-infra/irc-meetings master: use python 3 to build the site  https://review.openstack.org/61526910:02
*** arxcruz has quit IRC10:03
*** electrofelix has joined #openstack-infra10:10
*** ykarel is now known as ykarel|lunch10:12
*** rcarrillocruz has joined #openstack-infra10:14
*** jamesmcarthur has joined #openstack-infra10:16
*** jamesmcarthur has quit IRC10:21
*** gfidente has quit IRC10:21
*** yamamoto has quit IRC10:24
openstackgerritIlya Etingof proposed openstack-infra/git-review master: Improve exit code implementation  https://review.openstack.org/48026710:25
*** ykarel|lunch is now known as ykarel10:33
*** arxcruz has joined #openstack-infra10:36
*** bhavikdbavishi has joined #openstack-infra10:37
openstackgerritMerged openstack-infra/git-review master: Don't set topic when submitting no-topic patches  https://review.openstack.org/61098810:40
*** dayou_ has quit IRC10:43
*** yamamoto has joined #openstack-infra10:53
*** lbragstad has quit IRC11:02
*** lbragstad has joined #openstack-infra11:04
*** bhavikdbavishi has quit IRC11:05
*** florianf is now known as florianf|lunch11:06
*** yamamoto has quit IRC11:06
openstackgerritSorin Sbarnea proposed openstack/gertty master: Fix all pyflakes errors  https://review.openstack.org/58187111:08
*** shrasool has joined #openstack-infra11:10
*** e0ne has joined #openstack-infra11:11
openstackgerritSorin Sbarnea proposed openstack/gertty master: Adds linting zuul job  https://review.openstack.org/58187111:18
*** gfidente has joined #openstack-infra11:18
openstackgerritSorin Sbarnea proposed openstack/gertty master: Adds linting zuul job  https://review.openstack.org/58187111:21
openstackgerritMerged openstack-infra/zuul master: Fix issue in Github connection with large diffs  https://review.openstack.org/61298911:30
*** ramishra has quit IRC11:30
*** longkb has quit IRC11:30
*** carl_cai has quit IRC11:31
*** dtantsur|afk is now known as dtantsur11:33
*** yamamoto has joined #openstack-infra11:36
*** kaiokmo has joined #openstack-infra11:39
*** shrasool has quit IRC11:47
*** janki has quit IRC12:00
*** florianf|lunch is now known as florianf12:08
*** shrasool has joined #openstack-infra12:10
*** shrasool has quit IRC12:10
*** pbourke has quit IRC12:11
*** pbourke has joined #openstack-infra12:11
*** dpawlik has quit IRC12:12
*** carl_cai has joined #openstack-infra12:12
*** dpawlik has joined #openstack-infra12:12
*** dpawlik has quit IRC12:16
*** yamamoto has quit IRC12:24
*** dayou has joined #openstack-infra12:25
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Improve exit code implementation  https://review.openstack.org/48026712:26
*** gfidente has quit IRC12:26
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Fix unused warn function argument.  https://review.openstack.org/45692912:27
*** markvoelker has joined #openstack-infra12:27
*** ginopc has quit IRC12:27
*** ginopc has joined #openstack-infra12:28
*** e0ne has quit IRC12:29
*** dave-mccowan has joined #openstack-infra12:29
*** yamamoto has joined #openstack-infra12:29
*** shrasool has joined #openstack-infra12:30
*** jamesmcarthur has joined #openstack-infra12:30
*** yamamoto has quit IRC12:34
*** yamamoto has joined #openstack-infra12:34
*** yamamoto has quit IRC12:35
*** jamesmcarthur has quit IRC12:35
*** yamamoto has joined #openstack-infra12:35
*** jpena is now known as jpena|lunch12:38
*** ansmith has quit IRC12:39
openstackgerritFrode Nordahl proposed openstack-infra/project-config master: Add new charm repositories  https://review.openstack.org/61585512:43
*** e0ne has joined #openstack-infra12:43
*** gfidente has joined #openstack-infra12:45
*** rtjure has quit IRC12:50
openstackgerritFrode Nordahl proposed openstack-infra/project-config master: Add new charm repositories  https://review.openstack.org/61585512:55
*** Emine has quit IRC12:59
*** yamamoto has quit IRC13:01
*** rlandy has joined #openstack-infra13:02
*** yamamoto has joined #openstack-infra13:02
*** yamamoto has quit IRC13:07
openstackgerritFrode Nordahl proposed openstack-infra/project-config master: Add new charm repositories  https://review.openstack.org/61585513:07
*** dpawlik has joined #openstack-infra13:07
*** dpawlik has quit IRC13:07
*** dpawlik has joined #openstack-infra13:08
odyssey4meinfra-root yesterday clarkb held a node running the openstack-ansible-deploy-aio_metal-centos-7 test for https://review.openstack.org/#/c/615258/ for me, I'd like to get the IP and have my ssh key added please to take a look13:10
odyssey4memy pub ssh key is https://github.com/odyssey4me.keys13:10
*** ykarel_ has joined #openstack-infra13:11
*** ykarel has quit IRC13:14
fungiodyssey4me: no prob, gimme a moment13:15
*** quiquell is now known as quiquell|lunch13:15
*** Emine has joined #openstack-infra13:15
*** quiquell|lunch is now known as quiquell13:17
fungiodyssey4me: ssh root@158.69.64.22913:17
*** emine__ has joined #openstack-infra13:19
*** Emine has quit IRC13:20
openstackgerritAndrey Kurilin proposed openstack-infra/project-config master: [rally] Remove readthedocs jobs  https://review.openstack.org/61586813:21
*** markvoelker has quit IRC13:21
*** ykarel_ is now known as ykarel13:21
*** shrasool has quit IRC13:23
*** kjackal has quit IRC13:23
*** jcoufal has joined #openstack-infra13:23
*** kjackal has joined #openstack-infra13:23
*** quiquell is now known as quiquell|lunch13:26
*** jpena|lunch is now known as jpena13:28
*** shrasool has joined #openstack-infra13:30
*** boden has joined #openstack-infra13:31
*** kgiusti has joined #openstack-infra13:37
*** quiquell|lunch is now known as quiquell13:37
*** tdasilva has joined #openstack-infra13:40
*** ykarel has quit IRC13:42
*** eharney has joined #openstack-infra13:48
fungistephenfin: any chance you've got a spare moment to take a look at https://review.openstack.org/583535 for git-review?13:50
stephenfinfungi: Sure13:52
*** sthussey has joined #openstack-infra13:52
mordredAJaeger: re: https://review.openstack.org/#/c/615543/1/tox.ini@10 - what's the correct way to do things here? what we _want_ to do is have python3 be the python for all envs except for py2713:58
Shrewsmordred: drop support for py27???? ;)14:00
fungimordred: the fix for https://github.com/tox-dev/tox/issues/477 was introduction of an ignore_basepython_conflict option14:02
fungiwhich you'll need anyway if you want to test py35 and py36 on the same system, since otherwise both will use whatever your default python3 is14:03
*** felipemonteiro has joined #openstack-infra14:03
fungihttps://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conflict14:03
*** ykarel has joined #openstack-infra14:03
*** kaisers has quit IRC14:04
mordredfungi: so - like this: https://review.openstack.org/615543 ?14:07
openstackgerritMerged openstack-infra/git-review master: Improve exit code implementation  https://review.openstack.org/48026714:07
*** aluria has joined #openstack-infra14:07
*** ansmith has joined #openstack-infra14:07
fungimordred: yes14:07
*** agopi_ has joined #openstack-infra14:09
fungimordred: the pre-tox-3.1 workaround was to separately define testenv:py27, testenv:py35, testenv:py36 and so on each with its own distinct basepython override14:09
fungiso you'll likely see that in lots of places too14:09
fungibut it's no longer necessary with ignore_basepython_conflict=True14:10
mordredawesome14:10
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Add ignore_basepython_conflicts option to tox.ini  https://review.openstack.org/61588714:10
mordredfungi: https://review.openstack.org/615887 Add ignore_basepython_conflicts option to tox.ini14:10
*** kaisers has joined #openstack-infra14:10
mordredfungi: added one to zuul to handle 3.5/3.614:10
*** mriedem has joined #openstack-infra14:11
fungialmost, see typo marked inline14:11
*** ramishra has joined #openstack-infra14:13
*** emine__ has quit IRC14:15
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Add ignore_basepython_conflict option to tox.ini  https://review.openstack.org/61588714:17
mordredfungi: thanks!14:18
*** shrasool has quit IRC14:18
*** bobh has joined #openstack-infra14:19
*** carl_cai has quit IRC14:21
*** niska has quit IRC14:25
*** roman_g has joined #openstack-infra14:29
*** ginopc has quit IRC14:34
*** ginopc has joined #openstack-infra14:34
*** niska has joined #openstack-infra14:35
odyssey4meinfra-root the nodepool node 158.69.64.229 can now be released from the hold - thanks for that, we found the bug we needed :)14:36
mordred\o/14:36
Shrewsodyssey4me: scheduled for delete now. thx14:42
odyssey4metyvm!14:42
*** felipemonteiro has quit IRC14:43
openstackgerritMerged openstack-infra/zuul master: Don't do live streaming in loops  https://review.openstack.org/61565714:47
*** agopi_ is now known as agopi14:47
openstackgerritMerged openstack-infra/zuul master: Improve windows logging  https://review.openstack.org/61580414:49
openstackgerritSorin Sbarnea proposed openstack/gertty master: Adds linting zuul job  https://review.openstack.org/58187114:55
*** munimeha1 has joined #openstack-infra14:58
*** rh-jelabarre has joined #openstack-infra14:59
openstackgerritSorin Sbarnea proposed openstack/gertty master: Adds linting zuul job  https://review.openstack.org/58187115:01
openstackgerritFrode Nordahl proposed openstack-infra/project-config master: Add new charm repositories  https://review.openstack.org/61585515:01
*** armax has joined #openstack-infra15:03
*** e0ne has quit IRC15:06
*** ykarel is now known as ykarel|away15:22
*** ginux has joined #openstack-infra15:25
*** ginux is now known as Guest481515:25
AJaegerdiablo_rojo_phon: could you put https://review.openstack.org/#/c/615636/ on your review queue, please? An infra change for elections repo.15:27
*** Guest4815 has quit IRC15:28
*** ginopc has quit IRC15:28
*** dpawlik has quit IRC15:33
*** felipemonteiro has joined #openstack-infra15:34
*** ramishra has quit IRC15:40
*** ykarel|away has quit IRC15:43
*** yamamoto has joined #openstack-infra15:44
*** kopecmartin is now known as kopecmartin|off15:46
*** felipemonteiro has quit IRC15:46
*** yamamoto has quit IRC15:49
*** priteau has quit IRC15:49
openstackgerritNguyen Hai Truong proposed openstack-dev/hacking master: Add python 3.6 unit test job  https://review.openstack.org/61591515:51
clarkbmordred have you had a chance to look at ianw's cloud launcher thing? I wonder if it is related to the weird citycloud keystone setup15:52
mordredclarkb: it might be - I just got off a phone call, looking now15:53
mordredclarkb: oh - no - easy fix for the Lon1 issue15:56
fungistill seeing some odd tcp open rate patterns on the cacti graph for etherpad, and the background system cpu activity seems to have returned early yesterday:15:56
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=441&rra_id=all15:56
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=440&rra_id=all15:56
clarkbfungi: those links show graphite.o.o for me15:57
fungioh, funny15:57
fungii guess some of it is in post variables?15:57
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add Lon1 and Sto2 to openstackzuul cloud entry  https://review.openstack.org/61591815:57
mordredclarkb: ^^15:57
fungior! i was looking at the wrong graph, yes ;)15:58
fungiyep, totally looking at the wrong graphs15:58
clarkbmordred: you need to add the auth_url fix too15:58
mordredclarkb: ++ one sec15:58
mordredclarkb: oh - wait - actually - we released 0.19.0 of openstacksdk15:59
clarkbmordred: does that have the auth_url template built in? if so then likely just need to check that 0.19.0 is installed on bridge.o.o16:00
mordredclarkb: fix is in that - so we can actually remove the workaround16:00
clarkband now I need to join a call16:00
mordredclarkb: http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/config/vendors/citycloud.json?h=0.19.016:00
clarkbalso while I'm noticing it, tripleo seems to have grown a few non voting gate jobs?16:01
*** emerson has joined #openstack-infra16:01
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remove citycloud auth_url workaround  https://review.openstack.org/61591916:01
clarkbEmilienM: mwhahaha ^ fyi paunch, tripleo heat templates, and tripleo upgrade all haev at least one non voting gate job16:01
mwhahahai just cleaned them up :/16:02
EmilienMis it the scenarios?16:02
* mwhahaha looks16:02
stephenfinfungi: Reviewed https://review.openstack.org/#/c/583535/8 One comment left that maybe you could weigh in on16:02
clarkbmordred: openstacksdk==0.17.1  # git sha f28a1b0 is what is installed on bridge.o.o16:02
fungithanks stephenfin16:02
fungi!16:02
*** cloudnull has quit IRC16:02
*** d34dh0r53 has quit IRC16:02
*** eglute has quit IRC16:02
mwhahahaEmilienM: it's containered upgrades (for tripleo-ugrades)16:03
mordredclarkb: ok. well, we should certainly upgrade that :)16:03
mwhahahait's also the stable branches16:03
EmilienMmwhahaha: let me fix that16:03
mwhahahai need to cleanup the scenarios on the stable branches16:03
mwhahahaEmilienM: also containerized upgrades for paunch16:04
EmilienMk16:05
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Revert "Revert "Cleanup down ports""  https://review.openstack.org/61437016:06
*** panda is now known as panda|ko16:06
*** jamesmcarthur has joined #openstack-infra16:07
*** quiquell is now known as quiquell|off16:08
*** eglute has joined #openstack-infra16:08
*** d34dh0r53 has joined #openstack-infra16:08
*** kjackal has quit IRC16:16
*** imacdonn has quit IRC16:17
*** imacdonn has joined #openstack-infra16:17
*** hamzy_ has quit IRC16:19
*** gyee has joined #openstack-infra16:21
dmsimardIs ask.openstack.org supposed to be down ? Noticed it when following a google link16:21
funginot as far as i know. taking a look now16:23
dmsimardLooked quickly at the server and it doesn't seem unhealthy, I don't know how the app works though16:23
fungi[Tue Nov 06 06:53:42.278436 2018] [mpm_event:notice] [pid 24561:tid 140624258938752] AH00493: SIGUSR1 received.  Doing graceful restart16:24
fungi[Tue Nov 06 06:53:42.469710 2018] [core:notice] [pid 24561] AH00060: seg fault or similar nasty error detected in the parent process16:24
dmsimardapache segfault ?16:24
fungiduring log rotation, looks like16:24
fungii've started apache again16:24
dmsimardlooks back online, yay16:24
Tim_okAny pointers to a doc/github where the staring point to the zuul triggered third party CI builds setup is explained w/an example?16:26
*** ccamacho has quit IRC16:27
fungiTim_ok: help on https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-3rd-party-ci.html would be appreciated16:28
openstackgerritMerged openstack-infra/zuul master: Resume paused job with skipped children  https://review.openstack.org/61549316:30
*** ginopc has joined #openstack-infra16:31
*** udesale has quit IRC16:32
*** panda|ko is now known as panda|off16:34
*** andymccr has quit IRC16:36
openstackgerritsebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x  https://review.openstack.org/61193616:37
*** quiquell|off is now known as quiquell16:38
*** quiquell is now known as quiquell|off16:38
*** auristor has quit IRC16:42
*** hamzy_ has joined #openstack-infra16:44
fungicorvus: how have we disabled abandoned mailing lists in the past? just sent a final message to the list noting that it's being shut down, and then run rmlist on the server?16:49
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Only set basepython once  https://review.openstack.org/61594216:49
fungisaved a copy of the config_list output?16:49
*** auristor has joined #openstack-infra16:50
*** chandankumar is now known as chkumar|pto16:59
clarkbmordred: I think modern pip doesn't update all your deps when you pip install -U, so our ansible (and whatever else installs) don't pull in latest openstacksdk16:59
clarkbmordred: do we want to encode pulling in latest sdk or just update the citycloud clouds.yaml to work with older sdk too (I kinda like working with older sdk too)17:00
mordredclarkb: yah. I think we have to set state: latest on our pip install of openstacksdk17:00
fungiright, the default upgrade strategy changed from always to as-needed17:00
mordredclarkb: I don't see any benefit to having old sdk installed - but also, it doesn't come in as a depend of ansible- we install it directly17:00
clarkbah17:00
fungiso if your dependency set claims you need a newer version then pip will upgrade those dependencies, but only those dependencies17:01
clarkbas for supporting older sdk a lot of people use our clouds .yaml and modify as necessary for various clouds17:01
clarkband it might be more friendly to them if our config didn't expect latest sdk17:01
mordredthat doesn't make any sense to me - why would they copy our list of cloud accounts?17:01
clarkbmordred: not the entire list, but if someone wants to use rax they see how we do it17:02
clarkband so on17:02
clarkbbecause it turns out that things are still just enough inside baseball to confuse people (largely around keystone17:03
mordredright. but that's literally the entire point of the vendor config files in sdk - if we maintain our clouds.yaml as a thing for people to cargo cult, we're maintaining a second library - and if there are things that are general we should just update the vendor config in sdk, no?17:03
clarkbwell what we need to do is make keystone api versioning stop being impossible to figure out17:04
clarkb(our config for packethost is wrong btw we get a bunch of warnings from openstackclient)17:04
clarkbin theory once v2 is properly gone then this is a none issue because you use v3 everywhere17:04
mordredright. that's because openstackclient re-implements half of keystoneauth and we haven't fixed it yet17:04
clarkb'Ignoring domain related config project_domain_name for openstackci-packethostbecause identity API version is 2.0' etc17:04
mordredyeah17:05
mordredosc and sdk both work around keystone issues in an opposite way17:05
mordredI've got an idea of how to improve part of that in sdk I'm going to try out tomorrow17:05
mordredbut from what I can see, there isn't any information related to public clouds in our clouds.yaml file that is salient17:05
clarkbwell the cityclodu thing is if using older openstacksdk17:06
mordred(basically trying to figure out what info there would be valuable and how we can not make our specific clouds.yaml file valuable to other people)17:06
mordredsure. but that info is also in newer openstacksdk vendor files ... Im just saying that our specific config shouldn't be an alternate source of information from the sdk repo such that we avoid upgrading things or updating our files because we're maintaining an alternate view of the universe17:07
*** ginopc has quit IRC17:07
mordredour config should be our config and should serve us - and if there are ways in which things need to be clearer to people we should fix the actual docs or the actual source of truth17:08
clarkbyup I agree, but at the same time no one with an older sdk install is going to undersatnd how to use citycloud17:08
mordredright. they should upgrade openstacksdk17:08
clarkbI think sdk/oscc/keystone should (can) do more to address that17:08
mordredas it is designed right now17:08
mordredrunning with old openstacksdk == being broken17:08
clarkbthe proposal to put the clouds.yaml stuff at a well know location is a good step in that direction17:08
mordredthat can and should be made better17:08
mordredyeah17:08
clarkbthen you don't depend on sdk version (once support is added) and just talk to the cloud for that info17:09
mordredI'm super happy mnaser stuck a vendor file up there -I'm gonna try to get that iplemented RSN17:09
mordred*exactly*17:09
mordredbut the thing to put into clouds.yaml is here: http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/config/vendors/citycloud.json - and will totally work for people on older sdk ... so the question is - how can we get people to cargo cult from there instead of from our clouds.yaml file?17:10
*** rkukura has joined #openstack-infra17:11
clarkbmordred: in this case its too late for openstacksdk to say "hey this thing is weird check foo" but maybe citycloud could do that if talkign to the old keystone url via some error/exception17:11
clarkbmordred: that may be more trouble than its worth but basically have the old config error in such a way that it is obvious to the user that they need to go do something to fix it and not wait for the cloud to fix itself17:12
clarkbcurrent error is "that cloud region doesn't exist anymore"17:12
clarkbwhcih isn't true. Should be "please go update your config to this in order to see these cloud regions"17:13
mordredyeah. that would actually be much easier if we split the local vendor files into their own repo/library like with os-service-types17:13
mordredcause then we could do a freshness check on them17:13
mordredbut I think getting vendors to publish the files themselves is likely more better if we can do it17:14
clarkbyes I agree17:14
mnasermordred: you're welcome :D17:19
*** felipemonteiro has joined #openstack-infra17:19
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add Lon1 and Sto2 to openstackzuul cloud entry  https://review.openstack.org/61591817:20
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remove citycloud auth_url workaround  https://review.openstack.org/61591917:20
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Install latest openstacksdk on bridge  https://review.openstack.org/61595117:20
mordredclarkb: there's a set of thigns that should all work - and allow us to decide on removing the auth_url template separately :)17:20
coreycbAJaeger: hi, would you be able to comment on the last question posed here? https://review.openstack.org/#/c/610708/5/goals/stein/python3-first.rst@4517:22
*** ccamacho has joined #openstack-infra17:22
*** e0ne has joined #openstack-infra17:32
*** admcleod has quit IRC17:32
*** ccamacho has quit IRC17:33
*** jpich has quit IRC17:34
clarkbmordred: stack lgtm. I left my thoughts on the auth_url stuff in the last change but +2'd it17:34
clarkbinfra-root can you review https://review.openstack.org/615918 in particular to fix the cloud launcher17:38
*** boden has quit IRC17:40
Shrewsclarkb: openstacksdk 0.19.0 is released, so should that auth_url entry be there?17:41
ssbarneaclarkb mordred : can you please review https://review.openstack.org/#/c/613383/ ? -- i really hate to do horizontal scrolling on zuul. is only a CSS change.17:41
clarkbShrews: the whole stack has the set of changes that you want. (look at the child changes). Basically we still have 0.17.x installled. The following changes install 0.19.0 and then clean up the configs17:41
Shrewsclarkb: oh, oops17:41
clarkbShrews: we need that first one to fix the problem then followups clean things up17:42
clarkbreminder that the infra meeting is an hour earlier for those of you in the northern hemisphere that observe DST17:48
clarkbjust over an hour from now17:48
*** armax has quit IRC17:51
fungibut still at 19:00 for those of you who keep your clocks sanely set to utc!17:51
persia(hint for those who don't have "UTC" as a timezone choice: Iceland sanely keeps clocks set to UTC, so it is safe to set the clock to "Iceland time")17:52
clarkband I think it is an hour later for those of you in the southern hemisphere that observe DST? but timezone math is weird17:52
persiaclarkb: Yes, but the time change isn't particularly synchronised by hemisphere.  For example, Australia changed time 7th October this year.17:52
persiaWhereas Brazil only did it last weekend.17:53
clarkbpersia: ya and europe is earlier than USA too17:53
*** felipemonteiro has quit IRC17:53
clarkband not all US states observe DST (hawaii and arizona don't and maybe others now too?)17:53
clarkbthis is why we use UTC for scheduling around here :)17:54
persiaAnyway, for those particularly interested, I understand tzdata is currently up-to-date.17:54
fungiand apparently next year the eu votes on whether to finally ditch dst?17:54
*** diablo_rojo has joined #openstack-infra17:55
persiaRussia seems to be doing quite well in the aftermath of timezone rationalisation.17:55
openstackgerritMerged openstack-infra/os-loganalyze master: Assures that wrapping on PRE occurs on any kind of characters  https://review.openstack.org/61338317:55
*** sshnaidm|rover is now known as sshnaidm|afk17:56
*** derekh has quit IRC18:01
*** eandersson has joined #openstack-infra18:01
eanderssonHow do you get the nicely formated logs for a service, where you can filter based on log level?18:01
*** e0ne has quit IRC18:03
*** apetrich has quit IRC18:03
clarkbeandersson: we run a wsgi application on the log server that filters the logs and renders them that way on the fly. os-loganalyze is the tool18:03
eanderssonIs this something I need to configure in devstack, or maybe a project config?18:04
*** jpena is now known as jpena|off18:05
clarkbeandersson: it is configured on the web server serving your log files18:05
*** dtantsur is now known as dtantsur|afk18:06
*** e0ne has joined #openstack-infra18:06
mordredalthough - I believe the logs-in-swift code switches it around to run os-loganalyze on the logs as a pre-publication filter, yeah?18:12
clarkbmordred: ya we would publish multiple copies of things18:13
*** rkukura has quit IRC18:13
*** jtomasek has quit IRC18:18
*** bhavikdbavishi has joined #openstack-infra18:19
*** armax has joined #openstack-infra18:20
*** dpawlik has joined #openstack-infra18:21
*** gfidente is now known as gfidente|afk18:21
corvusfungi: i don't think we have done that for mailing lists in the past, but i think that's a fine idea.18:22
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for one git worker  https://review.openstack.org/61596618:22
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all git workers  https://review.openstack.org/61596718:22
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for git load balancer  https://review.openstack.org/61596818:22
AJaegercoreycb: can do...18:23
*** rpioso is now known as rpioso|afk18:24
coreycbAJaeger: ty18:24
*** e0ne has quit IRC18:24
*** rfolco|ruck is now known as rfolco|doctor18:30
*** markvoelker has joined #openstack-infra18:31
fungicorvus: mainly thinking ahead to december 3 when we shut down a bunch of openstack mailing lists in favor of using openstack-discuss18:31
AJaegercoreycb: not the answer you expected - but I left a lengthy explanation18:31
*** gyee has quit IRC18:34
corvusfungi: oh, sorry, i don't think we should run rmlist; i think we should just disable it.18:34
coreycbAJaeger: that's fine. the main question I wanted to get +/-1 on was enabling of 3 sets of py3 unit tests at the same time18:35
corvusfungi: disable subscription, posting, etc.  rmlist might affect accessibility of archives.18:35
*** armax has quit IRC18:35
coreycbAJaeger: which, I don't know if that's what you -1'd in particular. any chance you could clear that bit up?18:36
eanderssonclarkb, I was hoping this would enable that https://review.openstack.org/#/c/615688/18:40
fungicorvus: okay, that's why i'm asking. i couldn't figure out how to "disable" a list without using rmlist (which still leaves the archives behind unless you ask it not to)18:41
*** Swami has joined #openstack-infra18:42
*** ociuhandu has joined #openstack-infra18:42
corvusfungi: i imagine it will be removed from the index of lists.  if we're okay with that, maybe rmlist is all right.18:43
AJaegercorvus: see my last sentence18:43
corvusfungi: but otherwise, we can almost certainly make it so no one can subscribe or post to the list.18:43
fungiyou can set emergency moderation mode on so even subscribers end up in the moderation queue, you can set it so subscription requests require list admin approval, et cetera18:44
*** ralonsoh has quit IRC18:44
corvusAJaeger: was that intended for coreycb?18:44
corvusfungi: probably worth trying it out on a test instance or something to see what rmlist looks like.18:45
fungibut if you want, say, the mta to reject posts rather than having mailman drop them or start participating in spam backscatter, your options seem to be rmlist or adding overrides in the exim config18:45
corvusfungi: if it's still sufficiently discoverable, rmlist is probably preferable.18:45
*** dpawlik has quit IRC18:45
AJaegercorvus: indeed - sorry18:45
*** ociuhandu has quit IRC18:45
AJaegercoreycb: see my last sentence, I added it again on IRC18:46
fungii've had a request to retire the openstack-mentors ml (it's had one legitimate post in the past year and is basically just a spam magnet)18:46
fungithough i'm still confirming with the active mentors that's okay18:47
coreycbAJaeger: are you on the infra team?18:47
clarkbeandersson: ah so you want the senlin logs that infra hosts to be rendered that way?18:47
eanderssonYea18:47
clarkbeandersson: I misunderstood and thought you were looking for your own hosting18:47
corvusfungi: could be a good test case18:48
*** armax has joined #openstack-infra18:48
clarkbeandersson: https://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/os_loganalyze/filter.py#n24 needs to be updated to match your log files if it is in a format that is understood. The systemd log format is understood18:48
eanderssonTrying to clean up devstack in a few projects, and I thought it would be a neat addition18:48
coreycbAJaeger: I really just want a hard +/-1 from infra so I can work on this or just move on18:48
fungicoreycb: you need to get that from the tc, not from the infra team18:49
fungigoal texts are a governance matter for the tc18:50
coreycbfungi: well yes but I think infra needs to give input on what can be handled by infra18:50
AJaegercoreycb: but infra is either the PTL - and that's clarkb - or a couple of individuals with strong opininons but not all the same one ;)18:51
eanderssonSo looking at that clarkb we just need to add something like sl to > r'ceil|key|sah|des|tr)'  # openstack logs18:51
fungicoreycb: there are lots of ways openstack could choose to reduce resource consumption while testing against three versions of python interpreters. the implementation isn't something the infra team is supporting one way or the other18:51
coreycbAJaeger: that's the problem, too many chiefs and different answers :)18:51
clarkbeandersson: if that is the string that identifies your log files, yes18:51
fungicoreycb: you're choosing to add chiefs. the change you've proposed is a governance change, and you need to get the tc to decide. they may choose (and often do) to take input from a number of other stakeholders in the process of doing so18:52
*** hamzy_ is now known as hamzy18:53
clarkbas I've told others (including the TC) before I really don't think we should worry so much about what airship or kolla or nova etc are doing with their testing as long as tripleo uses the bulk of the resources. tripleo is improving but you won't drastically incrase resource consumption by adding some jobs to the 3%18:53
clarkbthe tail is incredibly long our per project and job resource usage18:53
coreycbfungi: i don't mean to offend with that it's just that there are so many suggestions that go in different directions18:53
clarkbadding an extra python job shouldn't have a major impact18:54
AJaegermy -1 was not for the three jobs - but -1 was for piggypacking it to that proposal. I asked for a separate document...18:54
* AJaeger would not block adding this to a few projects....18:54
eanderssonThanks clarkb18:54
fungiyes, i haven't weighed in with my tc hat on (yet) but similarly think we shouldn't go "moving goalposts" in goals once they've been approved and announced18:55
*** bobh has quit IRC18:55
fungia new document (t cycle goal, tc resolution, something) would be preferable18:56
coreycbAJaeger: if you could add that comment to the governance proposal that might be useful. i've had an email thread, and a governance proposal update, and some have told me to go back to the email thread is enough for +1 to move on with this, others have suggested another governance update. so not sure where to go.18:56
AJaegercoreycb: I did.18:56
coreycbAJaeger: i know, but it differes from what other say18:57
AJaegercoreycb: "So, I give a -1: Either do it without this goal - or make it a separate goal but don't piggypack on this one."18:57
openstackgerritErik Olof Gunnar Andersson proposed openstack-infra/os-loganalyze master: Add support for Senlin  https://review.openstack.org/61597718:58
AJaegercoreycb: and I'm not in the TC - that's what the TC has to decide on. I trust them to take my input and make the right decision for OpenStack - I have no problems to get overruled18:58
clarkbinfra meeting over in #openstack-meeting in jsut a moment18:59
coreycbAJaeger: ok thanks18:59
*** shardy has quit IRC19:00
*** rkukura has joined #openstack-infra19:03
coreycbdhellmann: AJaeger doesn't seem completely opposed to adding 3 x py3 unit tests, at least to a subset of projects. clarkb also doesn't seem to be opposed either. do you think we could approve this on my original mail thread rather than a governance proposal? perhaps I could just clear up my plans and get +1's on the ML?19:03
clarkbfwiw I do still think py35 should be deleted along with xenial jobs on master19:04
coreycbclarkb: stop it :)19:04
coreycbclarkb: kidding of course19:05
clarkbthats part of normal support from openstack19:05
clarkb(independent of the python3 first goal)19:05
coreycbclarkb: there were projects opposing that in particular keystone i believe and they pointed at a statement in the python3-first goal that said the min version of py35 wouldn't be changed in stein.19:06
openstackgerritMerged openstack-infra/system-config master: Add Lon1 and Sto2 to openstackzuul cloud entry  https://review.openstack.org/61591819:06
dhellmanncoreycb : yeah, a ML thread makes sense19:07
dhellmannI'd like for us to just take things 1 step at a time19:07
clarkbcoreycb: ya I see that as a bug in the python3 goal that should be fixed :)19:07
dhellmannI don't know yet which version of python 3 RHEL 8 will have. I am willing to gamble it is at least 3.5, but not 3.6 yet19:07
clarkbwe've never continued to support old python before whether 2.6 or 3.x19:07
coreycbclarkb: fair enough :)19:07
coreycbdhellmann: ok i'll re-raise the ML thread and seek out +1's, thanks.19:08
dhellmann++19:08
*** agopi has quit IRC19:09
*** diablo_rojo has quit IRC19:12
fungiinfra-root: heads up we just got a ticket from rackspace saying the host ze09 is on got rebooted19:12
fungii can't ssh into it at the moment19:13
*** diablo_rojo has joined #openstack-infra19:13
*** electrofelix has quit IRC19:14
fungioh, actually says it alerted them and they're investigating, but not that they've rebooted it (yet)19:14
fungiso i suppose that's expected19:14
*** rpioso|afk is now known as rpioso19:15
*** mriedem is now known as mriedem_vote19:18
AJaegerdiablo_rojo: could you put https://review.openstack.org/#/c/615636/ on your review queue, please? An infra change for elections repo.19:19
openstackgerritIan Wienand proposed openstack-infra/system-config master: Install latest openstacksdk on bridge  https://review.openstack.org/61595119:21
ianwmordred: ^ that adds a skip for ANSIBLE0010 (packages shouldn't install from latest) which I feel like is inappropriate for our environment19:22
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip master: [debug] Fix openstack_pip provider for pip 18  https://review.openstack.org/60602119:25
diablo_rojoAJaeger, yep! Its in the queue :)19:25
AJaegerthanks, diablo_rojo19:25
*** rlandy is now known as rlandy|vote19:26
fungiianw: normal solution to ANSIBLE0010 seems to be to use a variable for it, then default it to latest19:29
fungior at least that's what the openstack-ansible crew have been doing19:30
ianwfungi: is that a solution ... ?19:30
fungigives us a place to optionally override package versions in a role if we, say, want to reuse the same role on two servers with different version numbers (e.g. a -dev server and production?)19:31
fungiof course ignoring ANSIBLE0010 doesn't prevent us from still doing that where we want to19:32
ianwfungi: ok, well that's essentially what i did (for other reasons) with https://review.openstack.org/#/c/614894 ... want to look at that, and may be we can do similar?19:33
ianwwe actually need that, or similar, ATM, because the cloud-launcher currently requires bug fixes in a non-released ansible19:34
fungii guess that could do latest instead of omit if we wanted?19:34
ianwyou mean "instead of omit the rule"?  yes, i think we can tackle it all together maybe for consistency19:36
openstackgerritMerged openstack-infra/zuul master: Parallelize prime_installation_map  https://review.openstack.org/60787619:43
openstackgerritMerged openstack-infra/zuul master: Add ignore_basepython_conflict option to tox.ini  https://review.openstack.org/61588719:43
*** yamamoto has joined #openstack-infra19:45
*** david-lyle has joined #openstack-infra19:46
*** kjackal has joined #openstack-infra19:48
*** dklyle has quit IRC19:49
*** eernst has joined #openstack-infra19:49
cmurphyianw: did you have a change to dig into the puppet-pip on centos issue we talked about a while ago?19:50
*** yamamoto has quit IRC19:51
ianwcmurphy: ahhh, yeah, sorry i got distracted19:51
cmurphythis is an example of it happening http://logs.openstack.org/21/606021/14/check/puppet-beaker-rspec-centos-7-infra/37ecc04/job-output.txt.gz#_2018-11-05_20_14_20_972971 it clearly downloads pbr from the rax mirror but then later can't find it and tries to get it from pypi which is intentionally blocked in the test19:51
openstackgerritIan Wienand proposed openstack-infra/system-config master: bridge.o.o: Use latest openstacksdk  https://review.openstack.org/61598219:51
ianwfungi: ^ how's that look?19:51
cmurphyso i worry this has a bigger impact than just the puppet module since it could mean pip is always bypassing the mirror on centos19:52
ianwcmurphy: hrrm "Running setup.py (path:/tmp/pip-install-6FwWi8/os-service-types/setup.py)" ... is os-service-types involved here?19:53
fungicmurphy: i think that's the effect of using an extra index url instead of overriding the index url... could that be the cause19:54
fungi?19:54
cmurphyianw: os-service-types is involved because it's installing shade19:55
cmurphyfungi: hmm maybe?19:55
fungize09 did eventually reboot, up 26 minutes now19:55
cmurphyfungi: hmm no it is overriding extra-index-url http://logs.openstack.org/21/606021/14/check/puppet-beaker-rspec-centos-7-infra/37ecc04/job-output.txt.gz#_2018-11-05_20_05_34_54173319:55
fungiyeah, we normally override the primary index-url to our pypi cache, and then set an extra-index-url for our wheelhouse19:56
fungiso that looks right19:57
*** bhavikdbavishi has quit IRC19:57
fungiindex-url = http://mirror.iad.rax.openstack.org/pypi/simple19:57
fungiextra-index-url = http://mirror.iad.rax.openstack.org/wheel/centos-7.5.1804-x86_6419:57
fungiit shouldn't try to hit pipy.org at all if that configuration is in use (unless i misunderstand how pip works, or its behavior has changed yet again)19:58
*** rlandy|vote is now known as rlandy19:58
fungier, pypi.org19:58
cmurphyso far pip 18 and pip 10 exhibit this, latest patchset will see if pip 9 is the same19:59
ianwpip behaviour change seems likely :/20:01
ianwfungi: nix review on those pinning changes ... i've realised it's not quite right as "version=latest" isn't what we want, it's "state=latest"20:01
ianwi'll have to redo them20:01
fungiianw: yeah, "state" is what openstackansible is using for them as well20:02
ianwfungi: i wonder though if "latest" means we always re-install, i'm assuming it's not smart enough to check the version before doing anything20:03
fungii do not know20:03
ianw"present" == "install" , "latest" == "install -U"20:05
ianwso either way, it's fully invoking pip20:05
*** shrasool has joined #openstack-infra20:10
*** jamesmcarthur has quit IRC20:10
openstackgerritFabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver  https://review.openstack.org/60440420:11
*** bobh has joined #openstack-infra20:11
AJaegerconfig-core, could I get a second +2 on https://review.openstack.org/615637 (clarkb already reviewed) - removes now obsolete publish-static jobs from project-config20:13
AJaeger, please?20:13
*** rfolco|doctor is now known as rfolco|off20:16
*** mriedem_vote is now known as mriedem20:20
*** jtomasek has joined #openstack-infra20:21
*** aojea has quit IRC20:24
*** jtomasek has quit IRC20:24
openstackgerritIan Wienand proposed openstack-infra/system-config master: Pin bridge.o.o to ansible 2.7.0, add devel testing job  https://review.openstack.org/61489420:26
*** jamesmcarthur has joined #openstack-infra20:27
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for wiki.openstack.org  https://review.openstack.org/61565920:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for logstash.openstack.org  https://review.openstack.org/61566020:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for subunit workers  https://review.openstack.org/61566120:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for elasticsearch.openstack.org  https://review.openstack.org/61566520:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for one git worker  https://review.openstack.org/61596620:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all git workers  https://review.openstack.org/61596720:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for git load balancer  https://review.openstack.org/61596820:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for mirror-update.o.o  https://review.openstack.org/61599120:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for rax mirror  https://review.openstack.org/61599220:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all mirrors  https://review.openstack.org/61599320:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for files.openstack.org  https://review.openstack.org/61599420:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for on zookeeper instance  https://review.openstack.org/61599520:28
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all zookeeper instances  https://review.openstack.org/61599620:28
cmurphysame problem with pip 9 http://logs.openstack.org/21/606021/16/check/puppet-beaker-rspec-centos-7-infra/195c5ea/job-output.txt.gz20:32
openstackgerritMerged openstack-infra/storyboard-webclient master: Handle project names in the new Story modal controller  https://review.openstack.org/61277820:33
*** jamesmcarthur has quit IRC20:34
*** mmethot has joined #openstack-infra20:40
ianwcmurphy: so it's always been a problem?  or just with this new matching?20:43
*** munimeha1 has quit IRC20:46
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for one nameserver  https://review.openstack.org/61599820:50
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all nameservers  https://review.openstack.org/61599920:50
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for master nameserver  https://review.openstack.org/61600020:50
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid-dev  https://review.openstack.org/61600120:50
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid  https://review.openstack.org/61600220:50
cmurphyianw: i don't think it's always been a problem, it worked at some point20:51
mordredianw: ANSIBLE0010 ... sigh20:58
mordredianw: thanks for the followup on that20:59
AJaegerianw, mordred, could I get a second +2 on https://review.openstack.org/615637 (clarkb already reviewed) - removes now obsolete publish-static jobs from project-config, please?20:59
ianwmordred: per conversions with fungi, having thoughts on how we can pin it from variables ... working on it now21:00
mordredianw: nod. although - honestly, I like your comment in the file ... we do frequently run various things CD and I do think, depending on the thing, that that's ok21:09
clarkbthis is one of many of the nasible lint rules I disagree with21:10
clarkbthe other being you have to use the git module which only does like 10% of what git can do21:11
openstackgerritJeremy Stanley proposed openstack-infra/nodepool master: Correct heading levels for Kubernetes config docs  https://review.openstack.org/61600721:11
fungiclarkb: ansiblelint should totally be renamed youredoingitwrong21:13
openstackgerritIan Wienand proposed openstack-infra/system-config master: Pin bridge.o.o to ansible 2.7.0, add devel testing job  https://review.openstack.org/61489421:13
openstackgerritIan Wienand proposed openstack-infra/system-config master: bridge.o.o: Use latest openstacksdk  https://review.openstack.org/61598221:13
clarkbfor that 10% its great, but it shouldn't be considered a sin to shell/command out to git itself21:15
ianwi don't like that, afaics, there appears to be no contextal information around the reasoning, e.g. https://github.com/ansible/ansible-lint/commit/f5fcee4eb3bea623e625de2ede02a3e2686e5e1e21:16
ianwthe entire changelog is "Add two new rules and fix tests appropriately"21:16
ianwi know i go a bit overboard on change comments ... but come on ...21:16
clarkbianw: ya, fwiw I find that openstack in general is good about writing commit messages. I think we've learned over the years that we are doing this with others and communication is both hard and important21:17
fungii too had no luck finding any explanation of that rule other than its extremely brief title21:20
*** bobh has quit IRC21:20
*** boden has joined #openstack-infra21:21
clarkbianw: I always appreciate your overboard comments21:21
clarkbianw: it looks like lon1 has the floating ip router detach problem now with cloud launcher?21:23
clarkbwhy are we trying to delete subnets there ?21:23
ianwclarkb: oh, that's an ansible bug21:24
clarkbthats a scary bug if its trying to delete networking21:24
ianwclarkb: comments in https://storyboard.openstack.org/#!/story/2004267, that's why i'm trying to get this ansible pinning thing working because i guess we need a non-release version21:24
*** jcoufal has quit IRC21:25
clarkbgotcha, hrm21:25
openstackgerritMerged openstack-infra/system-config master: Install latest openstacksdk on bridge  https://review.openstack.org/61595121:25
clarkbso if we use devel ansible it will fix cloud launcher if we also fix openstacksdk/clouds.yaml21:26
ianwclarkb: i think so, at least i did a manual run with ansible devel branch version, and it then got stuck on the clouds.yaml21:27
ianwso now ^^ is in, can probably try again with the manual devel-branch run ... ?21:28
clarkbianw: ya21:28
clarkbI think the change prior to that should've fixed the clouds.yaml issue too for older sdk21:28
clarkbso you can run it now or wait for sdk to update21:28
*** apetrich has joined #openstack-infra21:29
ianwi've got the window up, so will do manual run now21:30
ianwclarkb: while on it, do you think we can add infra-core to ansible-role-cloud-launcher?21:30
ianwlike that change you have to silence the warnings would be nice21:30
clarkbianw: another workaround may be to set enable_snat to True in the module params for citycloud?21:30
clarkbianw: uh if it is an infra project sure (I'm not sure if it was added as an infra project /me checks)21:31
*** pcaruana has quit IRC21:31
clarkbianw: grepping for it in openstack/governance shows no results21:31
clarkbI think its a rcarrillocruz and pabelanger collaboration outside of infra?21:32
clarkbhuh how did I get added to core on that21:32
ianwyeah, that's why i was wondering if we can come up with some sort of agreement21:32
clarkbrcarrillocruz: ^ the infra team is still using this tool. Do you mind if we adopt it?'21:33
ianwclarkb: hrm, yeah the explicit snat:true might work around that too, i guess that seems otherwise pretty safe?21:33
ianwhrm, it still bailed on21:34
ianw"msg": "Region Lon1 is not a valid region name for cloud openstackzuul-citycloud. Valid choices are Kna1,La1. Please note that region names are case sensitive."21:34
ianwi'll wait for the sdk update and try again21:34
clarkbianw: looks like the all-clouds.yaml update hasn't applied yet21:34
clarkbwhen did that merge21:34
clarkbtwo and a half hours ago21:35
ianwoh, ha, i just assumed that was done :)  might be a typo stopping it? ...21:35
clarkb2018-11-06 21:30:15,642 p=25732 u=root |  ERROR! Completely failed to parse inventory source /opt/system-config/inventory/openstack.yaml21:35
rcarrillocruzclarkb: by all means21:36
ianwyep ...21:36
rcarrillocruzmore than happy to21:36
clarkbrcarrillocruz: thanks!21:36
rcarrillocruzyw!21:36
clarkb#status log infra-core added to ansible-role-cloud-launcher-core after getting rcarrillocruz's go ahead21:36
openstackstatusclarkb: finished logging21:36
clarkbianw: openstack plugin: __init__() got an unexpected keyword argument 'interface'21:37
clarkbopenstacksdk hasn't updated yet21:39
clarkbhttps://github.com/ansible/ansible/blob/v2.7.0/lib/ansible/plugins/inventory/openstack.py doesn't show an 'interface' when grepped21:39
clarkbso something is passing it interface and it is getting mad?21:39
clarkbmordred: ^ that look familiar to you?21:40
*** ansmith has quit IRC21:42
mordredclarkb: not immediately, no. I'd like to know what is passing interface to the plugin - that seems super weird21:42
clarkbya unfortunately that is the extent of the error message we seem to get in ansible.log21:42
clarkb(I dno't see a traceback)21:42
mordredclarkb: _blerg_21:43
mordredclarkb: looking21:43
clarkbit comes after stevedore loads all the keystone auth entrypoints21:43
clarkbso could be that interface is happening deeper than the ansible plugin level21:43
*** hamzy has quit IRC21:45
mordredyeah ... but still, interface should be an acceptable parameter21:46
*** bobh has joined #openstack-infra21:46
mordredclarkb: I got tracebacks by doing:21:47
mordredansible -vvvv bridge.openstack.org -m setup21:47
mordredalthough it's not making me happier21:47
mordredoh - wait a second - do we have interface in an auth dict ?21:48
mordredyup. there it is21:48
mordredpatch coming21:48
clarkbya armci cloud added it looks like21:48
fungiyeah, thanks again ianw for working on that!21:49
openstackgerritMonty Taylor proposed openstack-infra/system-config master: interface is not an auth option  https://review.openstack.org/61601521:49
mordredclarkb, fungi: ^^21:49
fungioof21:50
mordredclarkb: I edited the clouds.yaml by hand for now to verify21:50
clarkbwe may need to manually apply that to /etc/openstack/clouds.yaml on bridge.o.o once it merges as it won't update itself on that host21:50
clarkbcool21:50
ianwhrm, i guess the import we do during unit testing is only checking yaml-ish syntax, not correctness ... probably can't do that without actually running things21:51
mordredclarkb: this once again makes me think I want to update the running of the bridge playbook to not use the openstack inventory21:51
mordredianw: we can (and should) make a clouds.yaml linter21:52
clarkbmordred: and just do localhost localconnection and call it a day?21:52
mordredclarkb: yeah - or even execute it pointing to a whole other inventory and ansible.cfg so that it doesn't even try to run it or something21:52
mordredclarkb: I havne't written a patch yet because i'm nt 100% what feels right21:52
clarkbmordred: the nice thing about the current setup is that its consistent and clear21:53
mordredbut there's too much opportunity to get wedged while applying changes to the system21:53
mordredyah21:53
clarkbyes if it breaks we have to manually unwedge it21:53
clarkbbut maybe that is worthwhile? I don't know21:53
*** geekinutah has quit IRC21:57
*** geekinutah has joined #openstack-infra21:57
*** bobh has quit IRC22:02
*** kgiusti has left #openstack-infra22:09
*** rh-jelabarre has quit IRC22:14
*** eernst has quit IRC22:19
ianwmordred: a voluptuous style parser, or you thinking something else?22:26
mordredianw: I was thinking something provided by the sdk codebase that actually parses semantics ... although maybe that could be done via voluptuous22:27
mordredianw: of course, for that to reall work, we should probably fully define and document all of the semantics :)22:28
ianwmordred: yeah ... voluptuous errors would be better than weird "this isn't a dict" type errors you can get from deep inside bits of the sdk when you screw it up currently22:31
mordredyeah22:32
ianwclarkb: not sure if you saw it but https://review.openstack.org/#/c/615698/ implements the discussion on the traceroute failures we discussed yesterday22:32
ianwnamely a retry, and capturing of unbound logs22:32
clarkbianw: I hadn't but I shall now take a look22:33
*** felipemonteiro has joined #openstack-infra22:33
*** dave-mccowan has quit IRC22:35
openstackgerritMerged openstack-infra/system-config master: interface is not an auth option  https://review.openstack.org/61601522:37
clarkbI'll edit the clouds.yaml file on bridge.o.o now so that ^ can apply22:38
*** bobh has joined #openstack-infra22:42
*** bobh has quit IRC22:45
*** bobh has joined #openstack-infra22:46
*** bobh has quit IRC22:50
clarkbianw: looks like all-clouds.yaml has the citycloud fix now. And next ansible pass should add the interface fix22:52
clarkbif you want to give cloud launcher another run22:52
ianwok, bash history recall coming22:54
ianwand running22:55
ianwhrm, ansible-role-cloud-launcher doens't seem to be registered on readthedocs, but seems to think it's publishing there22:57
*** boden has quit IRC22:57
ianwkeystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-b085e685-c63d-430c-8d74-db63e4e9f22f)23:01
ianwhrumph, it doesn't tell you the cloud it was working on23:02
clarkbianw: that might be the interface bug in all-clouds23:02
clarkbsince that would cause broken auth aiui23:02
ianwwell it's a different error to before23:02
*** shrasool has quit IRC23:03
ianwdo we consume ansible-role-cloud-launcher from git or releases?  putting in your fix, and adding the cloud name to the running task would help with this a lot23:03
clarkbianw: system-config/roles.yaml says we consume HEAD23:05
*** felipemonteiro has quit IRC23:07
openstackgerritIan Wienand proposed openstack/ansible-role-cloud-launcher master: Add the cloud name to more task details  https://review.openstack.org/61602723:09
ianwclarkb: ^ i think that would let us know what cloud is failing in the logs23:09
clarkbianw: agreed23:11
openstackgerritMerged openstack/ansible-role-cloud-launcher master: Don't use deprecated when jinja2 syntax  https://review.openstack.org/60055123:18
openstackgerritIan Wienand proposed openstack-infra/project-config master: Fix ansible-role-cloud-launcher docs  https://review.openstack.org/61602923:21
clarkbianw: on the network debugging change I am trying to understand what zuul_debug_info.py s doing now23:27
clarkbwhat is an image_manifest?23:28
clarkbthe other thing is this is in zuul-jobs and is going to change debug info existing users gets23:28
ianwclarkb: it grabs like /etc/dib-builddate and stuff and stuff it into the output23:29
clarkbah23:29
ianwi don't think the changes are incompatible ... it only adds info23:30
clarkbI think the interface thing is only a minor concern since the role itself is the interface and that doesn't change23:30
clarkbya would only be an issue if someone uses that python module directly instead of the role23:30
ianwyeah, personally i think we can rule that out, it's under library/ in a role directory23:31
clarkbyou've convinced me +223:31
ianwclarkb: can we take the more relaxed approach with ansible-role-cloud-launcher and singe approve 616027?  i'm really not sure how else to tell which cloud it's getting hung up on without a bunch of effort23:32
*** rfolco|off has quit IRC23:33
ianwi'm not sure what we could do for testing in that project23:33
clarkbianw: ya I think you can do that since it really only changes the names and not function23:33
ianwi guess it would have to be a devsatck-ish based test23:34
clarkbhrm clouds.yaml broke itself again so ansible isn't running again23:34
clarkbmaybe there was a race between me editing it and ansible switching it back again23:35
clarkbI'll edit it again under that assumption and if it breaks again dig in23:35
openstackgerritMerged openstack/ansible-role-cloud-launcher master: Add the cloud name to more task details  https://review.openstack.org/61602723:41
*** yamamoto has joined #openstack-infra23:48
*** yamamoto has quit IRC23:53
*** rpioso is now known as rpioso|afk23:54
ianwok, running the launcher and it's grabbed those two changes23:56
ianwoutput is much nicer23:56
clarkbsystem-config on bridge updated to include the interface fix so I expect the last manual fix was the last one23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!