Saturday, 2017-09-02

jeblairi thought those were added to try to fix the error00:00
dmsimarder, I messed up my patch00:03
dmsimardlet's try that again00:03
dmsimardah it's missing an extra revert00:04
*** harlowja has quit IRC00:05
jeblairdmsimard: okay.  the other thing is that if we see the one-line parted emitting the error message, but, as you say, the devices are actually usable, then maybe we should stick a "|| /bin/true" on the parted command.00:05
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Revert "Move fix_disk_layout to an Ansible role in setup_host"  https://review.openstack.org/50022200:06
dmsimardok, added the missing revert00:06
jeblairthat makes more sense to me, thanks :)00:06
dmsimardjeblair: more or less00:06
dmsimardjeblair: the mklabel command is not fatal but still exits 1, so it does not do the mkpart when running in a single command00:06
dmsimardso a || true would work only if the commands remain split00:07
dmsimardworst case, if the label *really* did not apply, the following mkpart commands will truly fail00:07
jeblairdmsimard: how do you know that's the case?  i've still never seen any output from a one-line parted failure.00:08
dmsimardjeblair: tried it locally with that script, hang on00:08
clarkbit looks like my tox PR should pass CI so if nothing else that hack doesn't break their test suite and allows tempest to run00:10
clarkb(and maybe we'll get a new tox release soon with it included)00:10
dmsimardjeblair: http://paste.openstack.org/raw/620247/00:11
dmsimardjeblair: that's with https://gist.github.com/dmsimard/bdea83a76d6cc049f808c1110b6c1ca3#file-x-sh00:11
*** Apoorva has quit IRC00:12
dmsimardbut anyway, this is with a loopback device -- best way I've got of trying this, but it might not be a reliable way of reproducing the issue00:12
*** Apoorva has joined #openstack-infra00:12
dmsimardclarkb: could we perhaps hold a rax centos node for trying out different approaches manually ?00:12
jeblairdmsimard: gotcha00:12
clarkbdmsimard: ya we can do that. Also can you set -x00:13
clarkbI'll boot a node manually in rax now00:13
dmsimardclarkb: where, inside the revert ?00:13
dmsimardclarkb: oh, in the test script00:13
dmsimardsure00:13
dmsimardthis is oneline: http://paste.openstack.org/raw/620248/00:14
dmsimardthis is three commands: http://paste.openstack.org/raw/620249/00:15
*** spzala has joined #openstack-infra00:16
*** spzala has quit IRC00:16
*** rama_y has quit IRC00:16
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971900:16
clarkbserver is building, dmsimard can I get a public key for you?00:16
*** Apoorva has quit IRC00:16
dmsimardclarkb: https://github.com/dmsimard.keys00:17
clarkbthanks, once built I'll get you an ip address00:17
clarkbI went with iad because I know I've seen these problems in that region specifically00:17
dmsimardhappens with iad ord and dfw so far as I can tell00:18
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971900:18
clarkbdmsimard: root@104.130.172.20900:21
clarkbdmsimard: that is a centos-7 instance booted with the nodepool image on the same flavor00:21
clarkbso thats curious00:21
clarkbthere is an xvde100:21
clarkbit is not mounted though00:21
clarkband blkid doesn't show it00:22
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Ansiblify the wheel build a little more  https://review.openstack.org/49985700:22
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add wheel mirror jobs  https://review.openstack.org/49984300:22
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Treat wheel builders more like build nodes  https://review.openstack.org/50021700:22
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add periodic pipeline  https://review.openstack.org/50022600:22
jeblairclarkb: i think it comes partitioned; we just repartition so we can swap.00:23
dmsimardclarkb: it's just partitioned -- doing a "parted /dev/xvde print" shows it already has the msdos label and there's just one primary partition00:23
*** salv-orlando has joined #openstack-infra00:24
dmsimard(so perhaps the mklabel is not required since it's already there)00:24
clarkbdmsimard: is there a fielsystem?00:24
clarkbI wonder if the "refresh" of what that partition is could be causing us problems?00:24
*** baoli has joined #openstack-infra00:25
dmsimardclarkb: no filesystem afaict, won't let me mount it00:25
*** jascott1 has quit IRC00:26
dmsimardputting the kids to sleep and I'll check a bit after00:26
clarkbya lsblk seems to confirm00:26
*** baoli_ has joined #openstack-infra00:27
*** salv-orlando has quit IRC00:28
*** EricGonczer_ has joined #openstack-infra00:29
*** jascott1 has joined #openstack-infra00:29
*** baoli has quit IRC00:30
*** SergeyLukjanov has quit IRC00:33
*** SergeyLukjanov has joined #openstack-infra00:33
*** DinaBelova has quit IRC00:33
*** DinaBelova has joined #openstack-infra00:34
*** freerunner has quit IRC00:37
clarkbdmsimard: I've logged off the node feel free to throw your script at it00:37
*** NikitaKonovalov has quit IRC00:37
*** andreykurilin has quit IRC00:37
dmsimardclarkb: can we hold that through the weekend if need be ? I'll let you know if I find anything worthwhile00:37
*** SergeyLukjanov has quit IRC00:37
*** Diabelko has quit IRC00:38
*** gouthamr has quit IRC00:38
*** ruhe has quit IRC00:38
clarkbdmsimard: ya I won't delete it00:38
*** puiterwijk has quit IRC00:38
*** andreykurilin has joined #openstack-infra00:38
*** freerunner has joined #openstack-infra00:38
*** NikitaKonovalov has joined #openstack-infra00:38
*** Diabelko has joined #openstack-infra00:39
*** DinaBelova has quit IRC00:39
*** ruhe has joined #openstack-infra00:39
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Update to zuulv3 variables  https://review.openstack.org/50022900:39
*** puiterwijk has joined #openstack-infra00:39
*** DinaBelova has joined #openstack-infra00:39
*** baoli_ has quit IRC00:40
*** links has joined #openstack-infra00:40
*** SergeyLukjanov has joined #openstack-infra00:40
dmsimardclarkb: cool, that will certainly be useful, thanks00:41
dmsimardclarkb: for the record, we should have probably rechecked that patch a few times after all :)00:41
clarkbit hit the py35 thing00:41
clarkb(so unrelated to the change)00:41
*** harlowja has joined #openstack-infra00:47
*** scottda_ has quit IRC00:47
*** armax has quit IRC00:54
*** bobh has quit IRC00:55
*** gouthamr has joined #openstack-infra00:56
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Test CentOS 7.4 CR  https://review.openstack.org/50023500:57
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Test CentOS 7.4 CR  https://review.openstack.org/50023500:58
*** armax has joined #openstack-infra01:00
*** bobh has joined #openstack-infra01:00
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Update to zuulv3 variables  https://review.openstack.org/50022901:02
*** armax has quit IRC01:04
*** rama_y has joined #openstack-infra01:09
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM: Testing  https://review.openstack.org/50023601:09
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM: Testing  https://review.openstack.org/50023601:11
*** harlowja has quit IRC01:13
*** bobh has quit IRC01:13
pabelangerclarkb: did we want to add tox to upper-constraints, or just hold off until upstream fixes at this point? https://review.openstack.org/500221/01:14
*** zhurong has joined #openstack-infra01:17
*** nicolasbock has joined #openstack-infra01:18
*** EricGonczer_ has quit IRC01:21
openstackgerritIan Wienand proposed openstack-infra/devstack-gate master: Ansiblise ephemeral/swap setup  https://review.openstack.org/49946701:23
*** salv-orlando has joined #openstack-infra01:24
*** salv-orlando has quit IRC01:29
clarkbpabelanger: it would be best if requirements core could review that change01:31
*** mat128 has joined #openstack-infra01:32
*** salv-orlando has joined #openstack-infra01:33
*** yamamoto_ has quit IRC01:35
openstackgerritIan Wienand proposed openstack-infra/devstack-gate master: Ansiblise ephemeral/swap setup  https://review.openstack.org/49946701:36
*** EricGonczer_ has joined #openstack-infra01:36
*** markvoelker has joined #openstack-infra01:37
*** LindaWang has joined #openstack-infra01:41
*** harlowja has joined #openstack-infra01:45
*** harlowja has quit IRC01:47
*** yamahata has quit IRC01:48
*** iyamahat has quit IRC01:50
jeblairwhat's the py35 thing?01:53
clarkbjeblair: devstack py35 job installs tox 2.8 I think because its checking if tox is installed under python301:56
clarkbit is instead installed under python201:56
clarkbeasy fix is pin via constraints and that change has been apporved01:57
jeblairclarkb: ok thanks01:57
openstackgerritIan Wienand proposed openstack-infra/devstack-gate master: Ansiblise ephemeral/swap setup  https://review.openstack.org/49946702:00
*** ijw has quit IRC02:10
*** markvoelker has quit IRC02:11
*** ijw has joined #openstack-infra02:17
*** mriedem has quit IRC02:21
*** xinliang has quit IRC02:24
*** EricGonczer_ has quit IRC02:25
*** gaurangt- has quit IRC02:28
*** vhosakot has quit IRC02:29
*** rama_y has quit IRC02:29
*** thorst_afk has joined #openstack-infra02:32
*** thorst_a_ has joined #openstack-infra02:34
*** yamamoto has joined #openstack-infra02:36
*** thorst_afk has quit IRC02:36
*** xinliang has joined #openstack-infra02:38
*** xinliang has quit IRC02:38
*** xinliang has joined #openstack-infra02:38
*** ykarel has joined #openstack-infra02:42
*** ijw has quit IRC02:44
*** salv-orlando has quit IRC02:47
*** yamamoto has quit IRC02:47
*** salv-orlando has joined #openstack-infra02:47
*** thorst_a_ has quit IRC02:49
*** salv-orlando has quit IRC02:52
*** mat128 has quit IRC02:54
*** squid has quit IRC02:59
*** armax has joined #openstack-infra03:01
*** squid has joined #openstack-infra03:04
*** ykarel has quit IRC03:05
*** armax has quit IRC03:06
*** baoli has joined #openstack-infra03:06
*** markvoelker has joined #openstack-infra03:09
*** ykarel has joined #openstack-infra03:16
*** baoli has quit IRC03:16
*** jamielennox has quit IRC03:21
*** jamielennox has joined #openstack-infra03:22
*** nicolasbock has quit IRC03:28
*** ykarel has quit IRC03:40
*** markvoelker has quit IRC03:42
*** reed has quit IRC03:43
*** reed has joined #openstack-infra03:45
*** salv-orlando has joined #openstack-infra03:48
*** salv-orlando has quit IRC03:53
*** ykarel has joined #openstack-infra03:56
*** armax has joined #openstack-infra04:03
*** armax has quit IRC04:08
*** salv-orlando has joined #openstack-infra04:15
*** claudiub has quit IRC04:17
*** ykarel has quit IRC04:18
*** yamahata has joined #openstack-infra04:30
*** rama_y has joined #openstack-infra04:32
*** HeOS has quit IRC04:33
*** deduped has quit IRC04:36
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo: remove centos-7-nonha-multinode-updates job  https://review.openstack.org/50024904:36
EmilienMpabelanger: ^ cleanup and saving resources04:36
*** jpena|off has quit IRC04:37
*** nhicher has quit IRC04:38
*** links has quit IRC04:38
*** jpena|off has joined #openstack-infra04:39
*** nhicher has joined #openstack-infra04:39
*** markvoelker has joined #openstack-infra04:39
*** lihi has quit IRC05:01
*** ykarel has joined #openstack-infra05:01
*** oanson has quit IRC05:01
*** lihi has joined #openstack-infra05:01
*** oanson has joined #openstack-infra05:02
*** psachin has joined #openstack-infra05:04
*** reed has quit IRC05:04
*** armax has joined #openstack-infra05:05
*** reed has joined #openstack-infra05:06
*** armax has quit IRC05:09
*** markvoelker has quit IRC05:11
*** reed has quit IRC05:13
*** reed has joined #openstack-infra05:20
*** salv-orlando has quit IRC05:27
*** sree_ has joined #openstack-infra05:28
*** salv-orlando has joined #openstack-infra05:28
*** sree_ is now known as Guest415605:28
*** ramishra has joined #openstack-infra05:32
*** salv-orlando has quit IRC05:32
*** psachin has quit IRC05:34
*** rama_y has quit IRC05:51
*** jascott1 has quit IRC06:00
*** jascott1 has joined #openstack-infra06:00
*** tinwood has quit IRC06:02
*** tinwood has joined #openstack-infra06:03
*** jascott1 has quit IRC06:05
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: [WIP] Remove dd from LVM element  https://review.openstack.org/49815306:05
*** armax has joined #openstack-infra06:05
*** armax has quit IRC06:10
*** salv-orlando has joined #openstack-infra06:28
*** deduped has joined #openstack-infra06:34
*** xarses_ has joined #openstack-infra06:35
*** salv-orlando has quit IRC06:37
*** zhurong has quit IRC06:41
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add /label-list to the webapp  https://review.openstack.org/50025406:42
*** e0ne has joined #openstack-infra06:52
*** rcernin has joined #openstack-infra06:56
*** tinwood has quit IRC07:00
*** e0ne has quit IRC07:00
*** tinwood has joined #openstack-infra07:01
*** slaweq has joined #openstack-infra07:01
*** Guest4156 has quit IRC07:02
*** sree_ has joined #openstack-infra07:02
*** sree_ is now known as Guest6376907:03
*** lbragstad has quit IRC07:04
*** lbragstad has joined #openstack-infra07:04
*** slaweq has quit IRC07:05
*** armax has joined #openstack-infra07:06
*** Guest63769 has quit IRC07:07
*** yamamoto has joined #openstack-infra07:08
*** markvoelker has joined #openstack-infra07:09
*** armax has quit IRC07:10
*** gouthamr has quit IRC07:13
*** jascott1 has joined #openstack-infra07:16
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: [WIP] Remove dd from LVM element  https://review.openstack.org/49815307:20
openstackgerritMerged openstack-infra/devstack-gate master: Combine parted script into one command  https://review.openstack.org/50018807:25
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: OSA: Make openSUSE Leap 42.3 an official gate job  https://review.openstack.org/49498507:25
*** salv-orlando has joined #openstack-infra07:33
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: OSA: Make openSUSE Leap 42.3 an official gate job  https://review.openstack.org/49498507:34
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: Remove openSUSE jobs from unsupported OSA projects  https://review.openstack.org/50025507:34
*** salv-orlando has quit IRC07:38
*** markvoelker has quit IRC07:42
*** zhurong has joined #openstack-infra07:42
*** florianf has quit IRC07:43
*** ramishra has quit IRC07:44
*** slaweq has joined #openstack-infra07:53
*** yamamoto has quit IRC08:02
*** armax has joined #openstack-infra08:07
*** slaweq has quit IRC08:09
*** slaweq has joined #openstack-infra08:10
*** armax has quit IRC08:11
*** shardy has quit IRC08:13
*** yamamoto has joined #openstack-infra08:23
*** jtomasek has joined #openstack-infra08:28
*** yamamoto has quit IRC08:30
*** salv-orlando has joined #openstack-infra08:35
*** Jeffrey4l has quit IRC08:38
*** salv-orlando has quit IRC08:39
*** yamamoto has joined #openstack-infra08:39
*** markvoelker has joined #openstack-infra08:39
*** Jeffrey4l has joined #openstack-infra08:41
*** e0ne has joined #openstack-infra08:46
*** yamahata has quit IRC08:46
*** zhurong has quit IRC08:49
*** e0ne has quit IRC08:50
openstackgerritZhenguo Niu proposed openstack-infra/project-config master: Use whole disk image for Mogan job  https://review.openstack.org/50026308:51
*** e0ne has joined #openstack-infra08:55
*** e0ne has quit IRC09:02
*** armax has joined #openstack-infra09:08
*** yamamoto has quit IRC09:08
*** markvoelker has quit IRC09:12
*** armax has quit IRC09:12
*** slaweq has quit IRC09:16
*** yamamoto has joined #openstack-infra09:25
*** ykarel has quit IRC09:32
*** salv-orlando has joined #openstack-infra09:35
*** salv-orlando has quit IRC09:40
*** tosky has joined #openstack-infra09:43
*** yamamoto has quit IRC09:49
*** dingyichen has quit IRC09:54
*** yamamoto has joined #openstack-infra09:55
*** yamamoto has quit IRC10:00
*** LindaWang has quit IRC10:01
*** numans has quit IRC10:09
*** armax has joined #openstack-infra10:09
*** markvoelker has joined #openstack-infra10:09
*** numans has joined #openstack-infra10:11
*** claudiub has joined #openstack-infra10:12
*** armax has quit IRC10:13
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: [WIP] Remove dd from LVM element  https://review.openstack.org/49815310:15
*** ramishra has joined #openstack-infra10:18
*** salv-orlando has joined #openstack-infra10:36
*** yee379 has quit IRC10:38
*** yee37926 has joined #openstack-infra10:38
*** yamamoto has joined #openstack-infra10:41
*** salv-orlando has quit IRC10:41
*** markvoelker has quit IRC10:43
*** rcernin has quit IRC10:46
*** rcernin has joined #openstack-infra10:52
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: [WIP] Remove dd from LVM element  https://review.openstack.org/49815310:53
*** tosky has quit IRC11:00
*** armax has joined #openstack-infra11:09
*** armax has quit IRC11:14
*** psachin has joined #openstack-infra11:17
*** ykarel has joined #openstack-infra11:18
*** thorst_afk has joined #openstack-infra11:23
*** pbourke has quit IRC11:25
*** thorst_afk has quit IRC11:26
*** pbourke has joined #openstack-infra11:27
*** nicolasbock has joined #openstack-infra11:32
*** wolverineav has joined #openstack-infra11:34
*** salv-orlando has joined #openstack-infra11:37
*** nicolasbock has quit IRC11:41
*** salv-orlando has quit IRC11:41
*** jkilpatr has quit IRC11:50
*** nicolasbock has joined #openstack-infra11:54
*** slaweq has joined #openstack-infra12:04
*** slaweq has quit IRC12:07
*** slaweq has joined #openstack-infra12:08
*** slaweq has quit IRC12:11
*** ykarel has quit IRC12:34
*** salv-orlando has joined #openstack-infra12:38
*** markvoelker has joined #openstack-infra12:41
*** salv-orlando has quit IRC12:45
*** nicolasbock has quit IRC12:54
*** salv-orlando has joined #openstack-infra13:02
*** sree has joined #openstack-infra13:04
*** yamamoto has quit IRC13:07
*** yamamoto has joined #openstack-infra13:09
*** yamamoto has quit IRC13:09
*** yamamoto has joined #openstack-infra13:09
*** sree has quit IRC13:09
*** markvoelker has quit IRC13:13
*** hwoarang has quit IRC13:19
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove bindep_command and bindep_fallback references  https://review.openstack.org/49891613:26
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove bindep_command lines  https://review.openstack.org/49958913:27
*** hwoarang has joined #openstack-infra13:32
*** yamamoto has quit IRC13:32
*** yamamoto has joined #openstack-infra13:32
*** abelur_ has quit IRC13:33
*** bobh has joined #openstack-infra13:34
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove bindep_command lines  https://review.openstack.org/49958913:45
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Remove HUDSON_PUBLISH_DOCS environment variable  https://review.openstack.org/50030313:58
openstackgerritMonty Taylor proposed openstack-infra/puppet-jenkins master: Remove HUDSON comment  https://review.openstack.org/50030914:08
jeblairclarkb, dmsimard, pabelanger: one-line parted merged at 7:25, so any tripleo jobs started after 10:30 should be running with the change.14:13
*** bobh has quit IRC14:13
*** hwoarang has quit IRC14:14
clarkbe-r says no recent occurences, but that may mean no recent jobs14:15
*** hwoarang has joined #openstack-infra14:16
*** rama_y has joined #openstack-infra14:18
jeblairfascinating -- building this new devstack job from scratch, i just found that devstack uses hexdump on line 577, and it *installs* hexdump on line 756.  this does not work.14:21
jeblairit's because we set the mysql password in the gate14:26
*** yamamoto has quit IRC14:38
*** EricGonczer_ has joined #openstack-infra14:43
*** yamahata has joined #openstack-infra15:00
*** claudiub has quit IRC15:03
*** sree has joined #openstack-infra15:06
*** armax has joined #openstack-infra15:06
*** sree has quit IRC15:10
*** markvoelker has joined #openstack-infra15:10
*** rama_y has quit IRC15:15
*** david-lyle has quit IRC15:19
*** e0ne has joined #openstack-infra15:19
*** armax has quit IRC15:21
*** gmann has quit IRC15:23
*** mriedem has joined #openstack-infra15:23
*** gmann has joined #openstack-infra15:23
*** jkilpatr has joined #openstack-infra15:28
*** e0ne has quit IRC15:37
*** e0ne has joined #openstack-infra15:37
*** e0ne has quit IRC15:38
*** wolverineav has quit IRC15:38
*** wolverineav has joined #openstack-infra15:38
*** wolverineav has quit IRC15:38
*** e0ne has joined #openstack-infra15:38
*** yamamoto has joined #openstack-infra15:38
*** e0ne has quit IRC15:39
*** e0ne has joined #openstack-infra15:39
*** e0ne has quit IRC15:39
*** eroux has joined #openstack-infra15:40
*** e0ne has joined #openstack-infra15:40
*** e0ne has quit IRC15:40
*** e0ne has joined #openstack-infra15:41
*** psachin has quit IRC15:41
*** e0ne has quit IRC15:41
*** yamamoto has quit IRC15:43
*** mriedem has quit IRC15:43
*** markvoelker has quit IRC15:44
*** EricGonczer_ has quit IRC15:49
*** EricGonczer_ has joined #openstack-infra15:50
*** rkukura has quit IRC15:58
*** gouthamr has joined #openstack-infra16:00
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack  https://review.openstack.org/50020216:06
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module  https://review.openstack.org/49630116:06
*** claudiub has joined #openstack-infra16:08
*** rcernin has quit IRC16:12
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack  https://review.openstack.org/50020216:13
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module  https://review.openstack.org/49630116:14
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack  https://review.openstack.org/50020216:22
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: Add devstack_local_conf ansible module  https://review.openstack.org/49630116:22
*** salv-orl_ has joined #openstack-infra16:30
*** salv-orlando has quit IRC16:32
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Test CentOS 7.4 CR  https://review.openstack.org/50023516:32
*** ramishra has quit IRC16:36
mnasercan two changes depend on each other?16:38
mnaser(in gerrit)16:38
mnasersomething changed in ubuntu which broke our gate, puppet-ceph runs p-o-i jobs in gate so they will never pass, p-o-i change will pass but never merge because it depends on puppet-ceph which is failing16:39
*** markvoelker has joined #openstack-infra16:41
*** yamamoto has joined #openstack-infra16:41
dmsimardmnaser: changes with dependencies can't merge until the parent change has merged16:43
mnaserwell that sorta puts us in a weird dead lock16:43
dmsimardmnaser: only way it to temporarily make the broken jobs non-voting and revert16:43
mnaserbecause we cant merge into p-o-i without depends-on so the tests pass, but that change would never merge because puppet-ceph isn't passing p-o-i jobs16:44
mnaseri'll try to get the p-o-i job passing with depends on and then figure out that part16:44
mnaserxenial shipped 12.2.0 which requires ceph-mgr so all jobs with rbd went poof16:45
dmsimardmnaser: ah that's why, I was wondering why all the ubuntu jobs were failing all of a sudden16:45
dmsimardbtw there is a legit failure in rdo release for gnocchi -> pandas -> matplotlib, already working on a fix16:45
dmsimardnot sure why we didn't catch that16:45
mnaserdmsimard yeah i noticed that with the cr patch16:46
mnaserwhich btw failed because you were running yum-config-manager and not checking for the os version :p16:46
pabelangerdmsimard: mnaser: another option is to disable ceph in poi, merge fix, then enable in poi again. It will be 3 patches and hold off on merging anything new while ceph is disabled16:46
mnaser(fore xenial)16:47
dmsimardmnaser: bah the goal is not to test ubuntu anyway16:47
mnaser:P16:47
dmsimardjust throwaway patches16:47
dmsimardpabelanger: ceph is not as easy to disable as, say, an openstack project though16:47
dmsimardpabelanger: if gnocchi is broken you disable gnocchi, done16:47
dmsimardpabelanger: scenarios with ceph in it is because multiple components in that scenario depend on ceph so you might as well disable the entire scenario16:48
pabelangersure, but the process is the same, you'll need to disable that functionality for a moment, land fixes, renable. That's one way to unwedge16:49
dmsimardI guess16:49
*** salv-orlando has joined #openstack-infra16:49
pabelangermaking job non-voting is the other16:49
*** salv-orlando has quit IRC16:50
*** yamamoto has quit IRC16:50
*** salv-orlando has joined #openstack-infra16:50
dmsimardpabelanger: is there an easy way to see what code reviews might have triggered an e-r query ?16:51
dmsimardso we could recheck them, for example16:51
*** salv-orl_ has quit IRC16:53
pabelangerdmsimard: I usually use build_change from logstash16:53
dmsimardoh, not a bad idea16:53
pabelangercan either filter by that in webui, or use elastic-recheck-query16:54
*** salv-orlando has quit IRC16:55
mnaseri think disabling p-o-i jobs for puppet-ceph would be best, with a few depends-on, that way the project-config change to enable the jobs again merges right after too16:55
mnaserdisable p-o-i for puppet-ceph -> puppet-ceph fixes -> enable p-o-i for puppet-ceph as first cross-repo chain16:56
mnaserand p-o-i changes depending on puppet-ceph fix16:57
mnaseronce i get the actual patch working i'll start doing that chain16:57
*** gouthamr has quit IRC16:58
dmsimardmnaser: disabling jobs requires project-config reviews so just have to keep that in mind :P16:58
mnaseryeah it'll be a cross repo depends-on funtime16:58
mnaser:p16:58
dmsimardpabelanger: oh I found another way, kinda17:06
*** sree has joined #openstack-infra17:06
dmsimardhttps://review.openstack.org/#/q/reviewer:%22Elastic+Recheck%2217:06
*** gouthamr has joined #openstack-infra17:07
pabelangerYa, you can query gerrit too17:07
*** sree has quit IRC17:11
*** markvoelker has quit IRC17:13
*** ykarel has joined #openstack-infra17:16
*** eroux has quit IRC17:24
*** ykarel has quit IRC17:26
*** ykarel has joined #openstack-infra17:27
*** salv-orlando has joined #openstack-infra17:51
*** ykarel has quit IRC17:54
*** salv-orlando has quit IRC17:56
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS file if it exists  https://review.openstack.org/50032018:02
mordredjeblair: nice find18:02
*** bobh has joined #openstack-infra18:04
mordredmnaser: wow - a stable release of ubuntu released a new version of ceph that requires a new thing? or was this from the UCA archive?18:04
mnasermordred ceph-mgr was included in 10.x and required in 12.x, im guessing uca for pike decided to ship 12.x18:05
mnaserand that's how we broke18:05
mordredyah18:05
mordredmnaser: so - here's another question/option - (I don't know how all the things hang together so these may both be bong) ...18:06
mordredmnaser: a) can you just put in a package pin to 10.x (are the 10x packages still in the repo and just superseeded?)18:06
mordredmnaser: b) is there a an older UCA you could pin to while you mkae hte change? like the ocata UCA or something?18:06
mordredmnaser: c) tell me to shut up because I'm bothering you :)18:07
mnasermordred absolutely not, thanks for that input, 1 sounds interesting so i'm finding this out https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L11-L2018:08
mnaseri wonder if the pinning is not working18:08
mnasercause it sure looks like it doesn't (unless uca packages require 10.x)18:08
mnaseri wonder if even with the pin, the newer version 'wins' so to speak18:10
mordredpins should be able to prevent newer things winning ...18:11
mordredmnaser: oh - so - yah - that's just pinning source, I'd imagine upstream ceph would also have 12.0 in their repo right?18:12
mordredoh. duh. debian-jewel is the packages for jewel which means 10.x right?18:12
mnaseryep18:12
mnaserunless the mirror facts are pointing to a different url18:13
* mnaser checks18:13
mnaserhttps://github.com/openstack/puppet-openstack-integration/blob/master/configure_facts.sh#L37 nope18:13
mordredmnaser: also - are there dependencies between apt::pin and installatoin so that you're assured that apt::pin gets processed before you install stuff?18:14
mordredmnaser: you got a log showing the wrong ceph being installed?18:14
mnasermordred http://logs.openstack.org/95/500195/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ba25431/logs/ceph//ceph-osd.0.txt.gz18:15
mnaseryou can see ceph version 12.2.0 there18:15
mnaserbut the apt::pin not happening early is interesting theory18:15
mnaserlet me check18:15
mnaser2017-09-01 21:24:33 +0000 /Stage[main]/Openstack_integration::Repos/Apt::Pin[ceph]/Apt::Setting[pref-ceph]/File[/etc/apt/preferences.d/ceph.pref]/ensure (notice): defined content as '{md5}c8bb49860f66a880e233fd689c312cbd'18:16
mnaserit's happening very early18:16
mnasermordred http://logs.openstack.org/95/500195/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ba25431/logs/dpkg-l.txt.gz18:17
mnaserthe absolute proof the pin isn't working is the ~cloud0 tagging18:17
mordredmnaser: hrm18:20
clarkbmy tox PR merged, they've got a dev package pushed for testing18:23
mordred\o/18:23
mordredmnaser: ok - so - a few things18:23
clarkbif anyone else wants to test it (I did tempest locally)18:23
mnaseryay i fixed xenial and broke centos 7.. we'll get there18:24
mnaser:p18:24
mordredmnaser: a) not that it matters, but your default for ceph_miror is http:// not https:// ...18:24
mordredmnaser: neat! what did you do to fix xenial?18:24
mnasermordred write a manifest to install and configure ceph-mgr and deploy it18:25
mnaser12.2.0 cant run without one, so having it fixed it18:25
mnaserhttps://review.openstack.org/#/c/500317/18:25
mnaserbut i think centos doesn't ship with the ceph-mgr packaged so that could be why it failed..18:25
mordredawesome18:26
mnaserhttp://centos.mirror.vexxhost.com/7/storage/x86_64/ceph-jewel/ doesnt look like it18:26
mnaserok, i lied, ceph-mgr has been introduced in 11.x18:26
mnaserso it was not in 10.x18:26
mordredmnaser: so - you may also want to put a packages => ['ceph', 'ceph-base'] into your apt-pin18:26
mordredsince that pin is, I think, just saying to generally speaking prefer the ceph mirror over others18:27
mnaserbut not override i guess (which is why we're here)18:27
mordredyah18:27
mnaserso effectively i think it's useless in the pike release18:27
mordredhttps://forge.puppet.com/puppetlabs/apt#defined-type-aptpin18:27
mordreddoes not indicate that the name parameter does anything other than name the pin18:27
*** EricGonczer_ has quit IRC18:28
mnasermordred im thinking the best solution is to pin 10.x everywhere18:29
mnaserand then when 12.x comes out for centos as well we can up both of them and both of them will have ceph-mr18:29
mnasermgr*18:29
mordredmnaser: ++18:32
mnasermordred on second thought, it wouldn't provide proper testing for anyone trying to use uca and they'd be in for a surprise .. so i think i'll install ceph-mgr only for debian based with a note to remove that exception when 12.x packages are out18:33
mordredmnaser: that way you can also have the pin in the module since the module doesn't support 12 - and you can test that your ceph-mgr support works by including a bump of the pin in with the patch that adds support18:33
mordredmnaser: nod18:33
mnaserso if i'm $new_user and trying to use puppet modules, stuff doesn't work because it turns out i have to pin to an older release than shipped18:34
mnaserhttps://review.openstack.org/500318 seems reasonable18:35
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove HUDSON_PUBLISH_DOCS  https://review.openstack.org/50032118:37
mordredmnaser: yah - agree18:42
*** slaweq has joined #openstack-infra18:47
*** eranrom_ has joined #openstack-infra18:50
*** salv-orlando has joined #openstack-infra18:52
*** eranrom has quit IRC18:52
*** salv-orlando has quit IRC18:57
*** armax has joined #openstack-infra18:57
*** armax has quit IRC19:02
openstackgerritClark Boylan proposed openstack-infra/devstack-gate master: Create aligned partitions in disk fixing  https://review.openstack.org/49931419:03
clarkbassuming jeblair's parted one liner fixes centos partitioning getting ^ in would be nice to remove those warnings (and potentially make things perform better)19:03
pabelangerya, I've been watch the gate for tripleo, I haven't seen a swap failure yet. However, not many patches today19:06
*** sree has joined #openstack-infra19:07
openstackgerritPaul Belanger proposed openstack-infra/elastic-recheck master: Remove query for bug 1710605  https://review.openstack.org/50032219:09
openstackbug 1710605 in OpenStack-Gate "Generic a mirror outage due to an invalid repo file, since baseurl is empty" [Undecided,New] https://launchpad.net/bugs/171060519:09
*** markvoelker has joined #openstack-infra19:11
*** sree has quit IRC19:12
*** bobh has quit IRC19:12
*** tinwood has quit IRC19:14
*** armax has joined #openstack-infra19:14
*** tinwood has joined #openstack-infra19:15
clarkbhttp://paste.openstack.org/show/620274/ are the results of latest parted optimization patchset19:15
pabelangercool19:18
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Switch puppet-integration-jobs for puppet-ceph to non-voting  https://review.openstack.org/50032319:18
clarkbpabelanger: should we consider generating some tripleo load while things are quiet to gather more data on the partitioning change?19:21
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971919:21
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS file if it exists  https://review.openstack.org/50032019:21
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Update tox/test-requirements  https://review.openstack.org/50032419:21
pabelangerclarkb: I believe dmsimard recheck some tripleo jobs this morning to do just that19:23
openstackgerritMerged openstack-infra/elastic-recheck master: Remove query for bug 1710605  https://review.openstack.org/50032219:24
openstackbug 1710605 in OpenStack-Gate "Generic a mirror outage due to an invalid repo file, since baseurl is empty" [Undecided,New] https://launchpad.net/bugs/171060519:24
pabelangerclarkb: however, I was still under the impression that jeblair fix didn't fix the issue locally for dmsimard19:24
clarkbpabelanger: dmsimard would have to confirm but the testing done was against a loopback device which probably isn't very valid. But I booted a rax instance for like testing but not sure if that has happened yet19:26
pabelangerya19:26
pabelangerclarkb: also, http://logs.openstack.org/be/bed562d25557e3792fbdd39f0265939ad0a829da/post/cinder-upstream-translation-update/8045f91/console.html looks like tox is broken on proposal19:26
pabelangerI'll patch system-config in a bit19:27
smcginnisDo we still have bad tox's out there?19:27
pabelangerlikely on static workers19:28
smcginnisSo we just need to wait for those images to be refreshed?19:28
pabelangerwe don't use DIB for those servers, they live in control plane. We likely need to pin tox in system-config19:29
clarkbsmcginnis: also my patch to tox merged and tox upstream has produced a test package if you'd like to test what should become a new release19:29
smcginnisclarkb: Excellent!19:30
clarkbhttps://github.com/tox-dev/tox/issues/595 detail there19:30
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS file if it exists  https://review.openstack.org/50032019:30
smcginnisNice. I'm trying to take care of a couple other things, but if I have time I'll test it out.19:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output  https://review.openstack.org/49973219:31
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins master: Pin tox to 2.7.0  https://review.openstack.org/50032619:31
pabelangerclarkb: smcginnis ^19:31
pabelangerupstream seems to have confirmed the fix, https://devpi.net/obestwalter/dev/tox/2.8.1.dev8+gda3a86a19:32
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Switch puppet-integration-jobs for puppet-ceph to non-voting  https://review.openstack.org/50032319:32
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971919:33
mnaseris there a specific reason behind why gate-project-config-layout still runs in trusty?19:37
*** jascott1 has quit IRC19:37
mnaserit takes longer in queue usually because of less trusty nodes available19:37
pabelangerzuul.o.o is still trusty19:37
*** jascott1 has joined #openstack-infra19:37
clarkbmnaser: beacuse that is what we run the zuul/jjb stuff on in production and we have seen python version variability in the past19:37
mnaserah gotcha19:38
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add upper-constraints plumbing to base job and site-vars  https://review.openstack.org/50032819:38
mnaseri was just wondering if it was a leftover forgotten thing thats all19:38
pabelangerya, when the gate is quiet, rackspace is used more19:38
pabelangerand takes longer for nodes to comeonline there19:39
pabelangerTime to ready is ~ 4mins19:39
mordredpabelanger: if you have a second, wanna look at https://review.openstack.org/#/c/500320/ and https://review.openstack.org/500328 and tell me what you think?19:40
clarkbmordred: can you review https://review.openstack.org/50032619:40
mordredclarkb: yup19:40
clarkbtyty19:41
mordredclarkb: +A19:41
*** jascott1 has quit IRC19:42
mordredpabelanger: also, if you're not hacking on it right now I'm gonna poke at your 500155 for a minute19:42
pabelangermordred: ya, go for it19:43
*** markvoelker has quit IRC19:44
mnaserwould someone be kind enough to try and help me out on what i want to accomplish here? https://review.openstack.org/#/c/500323/ -- i just need the xenial jobs to be non voting for a little while19:48
mnaserim not really sure what im doing wrong19:48
openstackgerritMerged openstack-infra/puppet-jenkins master: Pin tox to 2.7.0  https://review.openstack.org/50032619:48
smcginnismnaser: I get the same feeling every time I touch that file. :)19:48
clarkbmnaser: drop the name: prefix19:49
mnaserduh.19:49
mnaserthats what i get for putting it in template the first time19:49
mnaserand copy pasting19:49
clarkb(I don't know why templates take a name but pipelines done)19:49
clarkb*dont19:50
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Switch puppet-integration-jobs for puppet-ceph to non-voting  https://review.openstack.org/50032319:50
mnaserattempt #3 fingers crossed19:50
*** salv-orlando has joined #openstack-infra19:53
*** salv-orlando has quit IRC19:57
mnaserhttps://review.openstack.org/#/c/500323/ if i can get some +2s so we can work on getting this merged https://review.openstack.org/#/c/500317/19:59
mnaserthanks clarkb :)20:00
openstackgerritMerged openstack-infra/project-config master: Switch puppet-integration-jobs for puppet-ceph to non-voting  https://review.openstack.org/50032320:08
clarkbit appaers that tripleo jobs kill root ssh for infra-rooters20:10
pabelangerclarkb: ah, I noticed that too. I've been meaning to ask why20:12
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Revert "Switch puppet-integration-jobs for puppet-ceph to non-voting"  https://review.openstack.org/50032920:15
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Update to using zuulv3 variables  https://review.openstack.org/50022920:22
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Refactor run-docs role to use tox role  https://review.openstack.org/50015520:22
mordredpabelanger: ^^20:23
openstackgerritMonty Taylor proposed openstack-infra/shade master: Switch to normal tox-py35 job with constraints  https://review.openstack.org/50033020:28
mordredpabelanger: that job ^^ should show the zuul-jobs upper-constraints patch working20:28
mordred(or not working, I mean, it's too much to ask for for a patch to work the fIRST time)20:29
openstackgerritMonty Taylor proposed openstack-infra/shade master: Switch to normal tox-py35 job with constraints  https://review.openstack.org/50033020:33
*** EricGonczer_ has joined #openstack-infra20:35
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS file if it exists  https://review.openstack.org/50032020:36
*** jascott1 has joined #openstack-infra20:39
*** markvoelker has joined #openstack-infra20:41
*** salv-orlando has joined #openstack-infra20:54
*** jtomasek has quit IRC20:58
*** salv-orlando has quit IRC20:58
*** sree_ has joined #openstack-infra21:08
*** sree_ is now known as Guest8280221:09
*** Guest82802 has quit IRC21:13
*** markvoelker has quit IRC21:14
*** apetrich has quit IRC21:27
*** apetrich has joined #openstack-infra21:27
*** rama_y has joined #openstack-infra21:35
*** bobh has joined #openstack-infra21:36
*** bobh has quit IRC21:38
*** tosky has joined #openstack-infra21:38
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS_FILE file if it exists  https://review.openstack.org/50032021:47
openstackgerritMonty Taylor proposed openstack-infra/shade master: Switch to normal tox-py35 job with constraints  https://review.openstack.org/50033021:47
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add upper-constraints plumbing to base job and site-vars  https://review.openstack.org/50032821:48
*** andreww has joined #openstack-infra21:52
*** salv-orlando has joined #openstack-infra21:54
*** xarses_ has quit IRC21:55
*** EricGonczer_ has quit IRC21:56
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS_FILE file if it exists  https://review.openstack.org/50032021:58
*** salv-orlando has quit IRC21:59
mordredBOOYAH22:02
mordredtotally worked22:02
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS_FILE file if it exists  https://review.openstack.org/50032022:04
*** slaweq has quit IRC22:15
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971922:17
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971922:29
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971922:33
*** bobh has joined #openstack-infra22:49
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971922:50
*** salv-orlando has joined #openstack-infra22:55
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Revert "Switch puppet-integration-jobs for puppet-ceph to non-voting"  https://review.openstack.org/50032922:55
mordredmnaser: you got it working?22:59
*** salv-orlando has quit IRC23:00
*** bobh has quit IRC23:06
*** sree_ has joined #openstack-infra23:09
*** sree_ is now known as Guest4013123:09
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Set puppet-cinder xenial gates to non-voting  https://review.openstack.org/50033423:10
*** markvoelker has joined #openstack-infra23:12
mnasermordred hey sorry i wasnt on irc23:13
mnaserceph-mgr is working23:13
mnaserbut there was another issue23:13
mnaserwhich made scenario002 fail23:13
*** Guest40131 has quit IRC23:13
mnaserhttps://review.openstack.org/#/c/500335/23:13
mnaserwhich should be fixed by this23:13
mnaserso hopefully this will be the last of the issues (it looks like it is) and puppet-openstack modules will be working on pike again23:14
*** slaweq has joined #openstack-infra23:15
*** tosky has quit IRC23:19
*** slaweq has quit IRC23:24
mordredmnaser: \o/23:38
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Log exeptions for module failure cases  https://review.openstack.org/50033623:43
mnaserif anyone is around, appreciate a +2 on https://review.openstack.org/#/c/500334/ to help fix puppet-openstack gates23:44
*** markvoelker has quit IRC23:44
*** armax has quit IRC23:46
*** armax has joined #openstack-infra23:46
*** armax has quit IRC23:46
*** armax has joined #openstack-infra23:47
*** armax has quit IRC23:47
*** armax has joined #openstack-infra23:48
*** armax has quit IRC23:48
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos  https://review.openstack.org/48971923:48
*** armax has joined #openstack-infra23:49
*** armax has quit IRC23:49
*** armax has joined #openstack-infra23:49
mordredmnaser: done23:49
*** armax has quit IRC23:50
mnasermordred tyvm!  i just saw all 4 p-o-i xenial patches green in status.o.o, so woohoo23:50
*** armax has joined #openstack-infra23:50
*** armax has quit IRC23:51
mnaserill nag cores for +2s soon and we should be good to go23:51
*** salv-orlando has joined #openstack-infra23:56
*** rama_y has quit IRC23:56
openstackgerritMerged openstack-infra/project-config master: Set puppet-cinder xenial gates to non-voting  https://review.openstack.org/50033423:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!