Wednesday, 2017-08-02

jeblairfungi, jlvillal: yes, though i'm more confused by why it's complaining at all.  like, how did this start enforcing.... hrm00:00
jlvillaljeblair: Got me.00:00
jeblairjlvillal: did it actually reject that push?  or was it just a warning message?00:00
jlvillaljeblair: Rejected as far as I could tell00:00
*** dingyichen has joined #openstack-infra00:01
jlvillaljeblair: No, I was wrong. I think00:01
jlvillalI think it is a warning, but not the real reason it was rejected.00:01
jeblairjlvillal: agreed00:02
cloudnullfungi: so the host running volume is down00:03
cloudnulllocked up00:03
cloudnullI have to cycle it00:03
jeblairjlvillal: do you maybe want to unabandon 427395 and push your cherry-pick on top of that?00:03
cloudnullvolume ID 975d5bd9-37f5-4cfe-965a-a917056de96100:03
jlvillaljeblair: Yeah, I may have to unabandon about 40 patches...00:03
cloudnullI assume that the two volumes are in a RAID 1 ?00:04
jlvillaljeblair: trying to decide what I want to do. New change-ids or unabandon patches...00:04
fungicloudnull: that would explain things, thanks! and no, the two volumes are separate00:04
cloudnullah ok .00:04
cloudnulli'll try to get it back asap00:04
jeblairjlvillal: right.  that's the alternative.00:04
fungicloudnull: there's no hurry, it's non-impacting at the moment. i could also just create a new volume00:05
clarkbcloudnull: we've avoided raiding cinder volumes in part because there is no indication of whether or not youd' get a volume from different devices or even hosts00:05
fungiif it becomes urgent00:05
clarkbcloudnull: so its a false sense of security00:05
fungiyeah, without an actual anti-affinity api to cinder volume placement, software raid strikes me as silly00:05
clarkbthough I guess our logs filesystem is basically a raid000:05
*** vhosakot has quit IRC00:06
fungimore like a jbod00:06
fungiwe don't stripe writes00:06
*** pahuang has quit IRC00:06
cloudnullfungi: what ever is easiest for everyone.00:07
cloudnullif nuking and re-building is simple and fast by all means go ahead00:07
cloudnullI'm going to try and get the host back online regardless.00:07
fungicloudnull: if i delete the volume, will that be likely to get the instance out of an error state? or am i likely to need to boot a replacement instance too?00:07
cloudnullhowever it's possible this is due to some hardware failure and its finally just giving up00:08
cloudnullI can reset the state of the instance00:08
fungiif you're going to poke at it anyway, then i'll wait00:08
fungiif you decide it's not worth getting it running, lmk and i'll happily rebuild the broken bits on our end00:08
*** jkilpatr has quit IRC00:08
cloudnullit'll be a few00:09
fungino rush, and thanks for whatever you come up with!00:09
cloudnullthese hp nodes take ~20 min to cycle00:09
fungiyeah, tell me about it. i used to deal with their earlier incarnations i expect00:10
fungithough i also used to deal with mainframes that took hours to boot, so by comparison...00:10
fungicould be worse!00:10
fungii need to go do some chores for a few minutes so i can settle in before the tc office hour starts in 45 minutes, but will be back shortly00:12
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Wrap handle_keys with debug statement
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix webapp path parsing
dmsimardmordred: re - pre-release, so what I want is "{name}-pypi-both-upload" and in release too, I guess ?00:13
dmsimardI've been publishing releases to pypi manually after tagging a release00:14
dmsimardbut openstack is already authorized to push on pypi for ara00:14
*** tosky has quit IRC00:15
openstackgerritMerged openstack-infra/infra-manual master: Add zuul v3 migration doc
*** gmann has quit IRC00:16
*** gmann has joined #openstack-infra00:16
clarkbdmsimard: yes that sounds right00:17
clarkbdmsimard: you also need the appropriate artifact build jobs00:17
clarkbdmsimard: there should be examples in the openstack release job group00:17
dmsimardclarkb: thanks! I'll look around.00:18
fungithere were a number of years where we only uploaded wheels for pre-releases but sdists and wheels for full release versions. i think a year or so ago we basically stopped doing the wheel-only uploads to pypi for pre-releases and just upload both regardless now00:18
dmsimardfungi: yeah it turns out it's safe to upload pre-releases to pypi, I didn't out of precaution so that pre-releases would not get installed due to higher version number00:19
dmsimardso knowing that now, I'll leave my fate in the hands of zuul and automated releasing \o/00:19
clarkbdmsimard: it wasn't safe once upon a time but that was many years ago and many pip releases00:19
*** mat128 has quit IRC00:19
fungiolder versions of pip didn't know any better (maybe pip<=1.3?) but it was also the same point where pip learned to find and install wheels00:20
fungihence our workaround, if your pip was new enough to find the wheel, it was also new enough to know to prefer release versions unless you told it pre-releases were okay00:20
*** jamesmcarthur has joined #openstack-infra00:20
clarkbjlvillal: are we essentially force pushing some external history into the ironic tempest plugin repo?00:21
* clarkb wonders how we ended up with so many changes all at once for it00:21
fungiclarkb: he said something about backports?00:21
fungiguessing a mass backport00:21
clarkbthese appear to be against master though00:21
fungioh, i bet these are split out of another repo, and we didn't import the initial repo state from a git filter-branch of those files00:22
funginow that everyone is required to put their tempest plugins in separate repos from other stuff00:23
*** pahuang has joined #openstack-infra00:23
*** jamesmcarthur has quit IRC00:25
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device
*** slaweq has joined #openstack-infra00:30
*** s-shiono has joined #openstack-infra00:31
*** rloo has quit IRC00:31
*** DFFlanders has quit IRC00:33
dmsimardianw: is f26 live yet ?00:35
*** baoli has joined #openstack-infra00:35
dmsimardnot impatient, just wondering :)00:35
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config master: Add pre-release and release automation for ARA
*** slaweq has quit IRC00:35
*** baoli has quit IRC00:35
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config master: Add pre-release and release automation for ARA
*** DFFlanders has joined #openstack-infra00:36
dmsimardclarkb: ^ I think that should do it, thanks for pointing me in the right direction :D00:36
*** baoli has joined #openstack-infra00:36
ianwdmsimard: i haven't started the builds, but we could00:36
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config master: Add pre-release and release automation for ARA
*** sree_ has joined #openstack-infra00:37
dmsimardianw: ok!00:37
*** sree_ is now known as Guest4052300:37
*** thorst_afk has joined #openstack-infra00:40
*** Guest40523 has quit IRC00:41
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Simplify github status descriptions
*** thorst_afk has quit IRC00:43
*** thorst_afk has joined #openstack-infra00:43
*** srwilkers_ has joined #openstack-infra00:47
*** https_GK1wmSU has joined #openstack-infra00:47
*** thorst_afk has quit IRC00:47
*** https_GK1wmSU has left #openstack-infra00:48
*** ramishra has joined #openstack-infra00:51
*** cuongnv has joined #openstack-infra00:51
*** lrossetti_ has quit IRC00:55
*** ijw has quit IRC00:56
*** ijw has joined #openstack-infra00:57
*** jamesdenton has quit IRC00:57
*** zeih has joined #openstack-infra00:57
*** jamesdenton has joined #openstack-infra00:58
*** jkilpatr has joined #openstack-infra01:00
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add Fedora 26 builds
openstackgerritIan Wienand proposed openstack-infra/project-config master: Initial Fedora 26 testing
*** gyee has quit IRC01:00
*** hongbin has joined #openstack-infra01:01
*** zeih has quit IRC01:02
*** EricGonczer_ has joined #openstack-infra01:03
*** samP has quit IRC01:04
*** bobh has joined #openstack-infra01:05
*** yushb has joined #openstack-infra01:10
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Support fedora-26 for nodepool dsvm job
*** EricGonczer_ has quit IRC01:14
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Make github ssl verification configurable
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add max-nodes-per-job tenant setting
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Use zuul attributes in nodeset section
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add a short name to the project in the inventory
*** Apoorva has joined #openstack-infra01:16
*** jkilpatr has quit IRC01:17
*** yamahata has quit IRC01:20
*** Apoorva_ has quit IRC01:20
jeblairzuul/nodepool don't look quite right01:20
*** Apoorva has quit IRC01:21
*** annegentle has joined #openstack-infra01:22
fungijeblair: should we merge 489776 and take nodepool.o.o back out of the emergency disable list, or will that confuse troubleshooting further?01:22
*** https_GK1wmSU has joined #openstack-infra01:22
jeblairfungi: i'm not sure we can merge anything at this point01:23
jeblairzuul and nodepool look like they are busy idling01:23
*** liujiong has joined #openstack-infra01:23
*** https_GK1wmSU has left #openstack-infra01:24
*** EricGonczer_ has joined #openstack-infra01:24
jeblairoh, 4689 merge jobs pending01:26
jeblairzuul *is* processing merge jobs01:26
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Use project object in _uploadPack in gerrit driver
fungiso just... too busy to start any non-merge jobs for a bit01:28
jeblairi think something may be awry with the ironic-tempest-plugin stuff01:28
*** EricGonczer_ has quit IRC01:28
jeblairlooking at zm01, it seems to be merging a *very* long stack of changes01:28
fungiyeah, i get the impression that instead of doing a git filter-branch and importing that at repo creation, they decided to push in all the commit history one change at a time or something01:29
cloudnullfungi: it's dead jim01:29
cloudnullthe node refuses to come back .01:29
fungicloudnull: thanks for trying anyway! i'll work on a replacement, though maybe not until tomrrow01:30
cloudnulland i don't think dcops will prioritize getting the drives swapped01:30
cloudnullso i suspect that this instance is a loss.01:30
*** mriedem has quit IRC01:30
fungicloudnull: is what it is, not hard to replace on our end but thanks for looking!01:30
cloudnullsorry for the hassel01:31
fungino apology needed, really01:31
jeblair4605 merge jobs01:31
jeblairso, erm, i reckon it'll work its way out eventually01:31
fungijlvillal: if you're still around, what's the story with the really long chain of dependent changes for the tempest plugin?01:31
*** ijw has quit IRC01:31
mtreinishfungi: my guess is history preservation01:32
*** ijw has joined #openstack-infra01:32
mtreinishfungi: if you run the scripts to do that history migration without after the repo is already in gerrit, that's a lot of patches01:32
fungithat was my guess as well, though we have some less impactful solutions to that01:33
jeblairfungi: i'm done looking at things, feel free to change nodepool at will; i have to run now01:33
fungihave a good evening jeblair, and thanks!01:34
*** tuanluong has joined #openstack-infra01:38
*** gouthamr has joined #openstack-infra01:42
*** Goneri has quit IRC01:43
*** thorst_afk has joined #openstack-infra01:44
mnaserold patch which would make things prettier if we can get a +2 :)
*** pahuang has quit IRC01:48
*** dingyichen has quit IRC01:50
*** aeng has quit IRC01:50
*** aeng_ has joined #openstack-infra01:50
*** dingyichen has joined #openstack-infra01:50
*** abelur_ has joined #openstack-infra01:51
*** LindaWang has joined #openstack-infra01:51
*** dchen has joined #openstack-infra01:52
*** pahuang has joined #openstack-infra01:52
*** pahuang has quit IRC01:52
fungiyeah, would be nice to get another infra-root to review it01:52
fungii'm not going to single-core approve is as i'm sponsored by the same organization as the author01:53
*** markvoelker has joined #openstack-infra01:54
*** dingyichen has quit IRC01:55
mnaserfungi np :) i left a screenshot with the change applied to hopefully helpout the review01:56
fungii saw! thanks so much for testing that01:57
*** esberglu has quit IRC01:59
*** slagle has quit IRC01:59
*** pahuang has joined #openstack-infra02:00
*** thorst_afk has quit IRC02:00
*** baoli has quit IRC02:02
*** baoli has joined #openstack-infra02:02
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Make v2 grenade job non-voting
*** slaweq has joined #openstack-infra02:07
*** slaweq has quit IRC02:11
*** slagle has joined #openstack-infra02:12
*** gildub has joined #openstack-infra02:15
jlvillalfungi, As mtreinish said, history preservation.02:15
jlvillalfungi, Luckily it is only doing pep8 checks.02:16
*** dingyichen has joined #openstack-infra02:22
*** bobh has quit IRC02:23
*** markvoelker has quit IRC02:23
*** yamamoto has quit IRC02:24
*** ijw has quit IRC02:25
*** slaweq has joined #openstack-infra02:25
clarkbjlvillal: was importing history when the project was created considered? (our project creation process does that for free)02:27
*** yamamoto has joined #openstack-infra02:30
*** slaweq has quit IRC02:30
*** yamamoto has quit IRC02:35
*** yamamoto has joined #openstack-infra02:36
*** esberglu has joined #openstack-infra02:38
*** dave-mcc_ has joined #openstack-infra02:40
*** yamamoto has quit IRC02:41
*** esberglu has quit IRC02:42
*** dave-mccowan has quit IRC02:42
*** baoli has quit IRC02:43
*** panda has quit IRC02:43
*** aeng_ is now known as aeng02:44
*** yamamoto has joined #openstack-infra02:45
*** panda has joined #openstack-infra02:48
*** ramishra has quit IRC02:53
*** nicolasbock has joined #openstack-infra02:57
*** zeih has joined #openstack-infra02:58
*** markvoelker has joined #openstack-infra02:59
openstackgerritMerged openstack-infra/subunit2sql master: Add a unit test with a simple sql injection ala xkcd
jlvillalclarkb, No, but we asked afterwards and people said "too late"...03:02
*** zeih has quit IRC03:02
*** bobh has joined #openstack-infra03:04
*** rkukura has quit IRC03:12
*** dave-mccowan has joined #openstack-infra03:12
*** dave-mcc_ has quit IRC03:14
*** bobh has quit IRC03:18
*** gtrxcb has joined #openstack-infra03:21
*** esikachev has joined #openstack-infra03:23
*** thorst_afk has joined #openstack-infra03:23
*** esikachev has quit IRC03:27
*** thorst_afk has quit IRC03:27
*** aeng has quit IRC03:29
*** hongbin has quit IRC03:30
*** dchen has quit IRC03:31
*** dchen has joined #openstack-infra03:32
*** gildub has quit IRC03:32
*** yamamoto has quit IRC03:33
*** dave-mccowan has quit IRC03:33
*** rodrigopaiva has joined #openstack-infra03:37
*** rodrigopaiva has quit IRC03:41
*** rodrigopaiva has joined #openstack-infra03:46
*** stakeda has joined #openstack-infra03:47
*** links has joined #openstack-infra03:48
*** rodrigopaiva has quit IRC03:48
*** yamamoto has joined #openstack-infra03:49
pabelangerjeblair: fungi: clarkb: mordred: So, apparently we generate .md5 and .sha1 files for tarballs and wheels, but don't do anything with them. EG: upload them to tarballs.o.o. Do we actually want to start uploading them?03:59
pabelangerworking on ansible playbook for tox-tarball job03:59
*** sree has joined #openstack-infra04:00
*** slaweq has joined #openstack-infra04:03
*** yamamoto has quit IRC04:05
openstackgerritIan Wienand proposed openstack-infra/project-config master: Initial Fedora 26 testing
*** gouthamr has quit IRC04:06
*** gouthamr has joined #openstack-infra04:06
clarkbpabelanger: I think we record them in the logs but thsts it04:07
pabelangerclarkb: right, that is what I see too04:07
*** slaweq has quit IRC04:07
*** yamamoto has joined #openstack-infra04:10
*** gongysh has joined #openstack-infra04:13
*** winggundamth has joined #openstack-infra04:16
*** yushb has quit IRC04:18
*** gildub has joined #openstack-infra04:22
*** adisky__ has joined #openstack-infra04:24
*** sree has quit IRC04:28
openstackgerritMerged openstack-infra/nodepool master: Support fedora-26 for nodepool dsvm job
*** ykarel has joined #openstack-infra04:30
*** gildub has quit IRC04:31
*** kornicameister has quit IRC04:31
*** yushb has joined #openstack-infra04:32
*** kornicameister has joined #openstack-infra04:32
openstackgerritIan Wienand proposed openstack-infra/project-config master: Disable swift for Vitrage
*** ijw has joined #openstack-infra04:40
*** gildub has joined #openstack-infra04:42
*** esikachev has joined #openstack-infra04:44
*** ijw has quit IRC04:45
openstackgerritMerged openstack-infra/project-config master: Add pre-release and release automation for ARA
openstackgerritMerged openstack-infra/project-config master: tripleo: move container upgrade jobs to check
*** yamamoto has quit IRC04:50
*** sree has joined #openstack-infra04:51
*** pahuang has quit IRC04:51
*** ijw has joined #openstack-infra04:52
openstackgerritMerged openstack-infra/project-config master: [Zun] Add py35 dsvm non-voting job
*** aeng has joined #openstack-infra04:55
*** dchen has quit IRC04:56
*** dchen has joined #openstack-infra04:56
*** esikachev has quit IRC04:59
*** zeih has joined #openstack-infra04:59
openstackgerritMerged openstack-infra/project-config master: Disable swift for Vitrage
*** ijw has quit IRC05:00
openstackgerritIan Wienand proposed openstack-infra/puppet-jenkins master: Don't install ccache for RPM distros
*** zeih has quit IRC05:03
*** yamamoto has joined #openstack-infra05:07
*** ramishra has joined #openstack-infra05:12
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual master: Add internal link for job naming
*** jtomasek has joined #openstack-infra05:16
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Don't ignore inexistent jobs in config
*** psachin has joined #openstack-infra05:20
*** cuongnv has quit IRC05:23
*** thorst_afk has joined #openstack-infra05:23
*** slaweq has joined #openstack-infra05:25
*** ramishra has quit IRC05:26
*** ramishra has joined #openstack-infra05:28
*** thorst_afk has quit IRC05:28
*** cuongnv has joined #openstack-infra05:29
*** slaweq has quit IRC05:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Simplify github status descriptions
*** shu-mutou is now known as shu-mutou-AWAY05:32
*** yushb has quit IRC05:39
*** esikachev has joined #openstack-infra05:42
openstackgerritMerged openstack-infra/zuul-jobs master: Simplify run tox task
*** pgadiya has joined #openstack-infra05:48
*** lathiat_ has quit IRC05:52
*** lathiat has joined #openstack-infra05:52
*** gouthamr has quit IRC05:53
*** cshastri has joined #openstack-infra05:59
*** yushb has joined #openstack-infra06:00
*** e0ne has joined #openstack-infra06:01
*** e0ne has quit IRC06:03
*** slaweq has joined #openstack-infra06:15
*** rlandy has joined #openstack-infra06:18
*** mpaolino has joined #openstack-infra06:19
*** jaosorior has quit IRC06:25
*** gildub has quit IRC06:27
*** rwsu has quit IRC06:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Rename Node.hold_reason to 'comment'
*** ccamacho has left #openstack-infra06:33
*** jaosorior has joined #openstack-infra06:34
*** pcaruana has joined #openstack-infra06:34
*** markus_z has joined #openstack-infra06:36
*** jamesmcarthur has joined #openstack-infra06:37
*** jamesmcarthur has quit IRC06:41
*** panda has quit IRC06:43
*** rcernin has joined #openstack-infra06:46
*** panda has joined #openstack-infra06:47
*** esberglu has joined #openstack-infra06:48
*** sflanigan has quit IRC06:48
*** kjackal_ has quit IRC06:51
*** markvoelker has quit IRC06:51
*** esberglu has quit IRC06:52
*** andreas_s has joined #openstack-infra06:55
*** sflanigan has joined #openstack-infra06:57
*** sflanigan has joined #openstack-infra06:57
*** namnh has joined #openstack-infra06:58
*** prateek has joined #openstack-infra06:59
*** zeih has joined #openstack-infra07:00
*** zeih has quit IRC07:05
*** slaweq_ has joined #openstack-infra07:05
*** aarefiev_afk is now known as aarefiev07:07
*** sflanigan has quit IRC07:07
*** apetrich has quit IRC07:08
*** slaweq_ has quit IRC07:10
*** zeih has joined #openstack-infra07:12
*** apetrich has joined #openstack-infra07:13
*** https_GK1wmSU has joined #openstack-infra07:14
*** kjackal_ has joined #openstack-infra07:15
*** zeih has quit IRC07:16
*** https_GK1wmSU has left #openstack-infra07:16
*** aviau has quit IRC07:19
*** aviau has joined #openstack-infra07:19
*** sflanigan has joined #openstack-infra07:21
*** sflanigan has joined #openstack-infra07:21
*** thorst_afk has joined #openstack-infra07:24
*** sree has quit IRC07:28
*** sree has joined #openstack-infra07:28
*** sree has quit IRC07:29
*** thorst_afk has quit IRC07:29
*** esikachev has quit IRC07:31
*** sree has joined #openstack-infra07:31
openstackgerritGergely Csatari proposed openstack-infra/project-config master: Removal of installguide-cookiecutter
*** annegentle has quit IRC07:34
*** dizquierdo has joined #openstack-infra07:34
*** hashar has joined #openstack-infra07:35
*** electrofelix has joined #openstack-infra07:35
*** Hal has joined #openstack-infra07:36
*** Hal is now known as Guest8369307:36
openstackgerritMerged openstack-infra/project-config master: Enable Django 1.10 test for Sahara UI only from Pike+
*** Guest44789 has quit IRC07:39
openstackgerritMerged openstack-infra/project-config master: networking-midonet: Make v2 grenade job non-voting
*** gtrxcb has quit IRC07:44
*** aeng has quit IRC07:45
*** ralonsoh has joined #openstack-infra07:46
*** rlandy is now known as rlandy|ci_mtgs07:46
*** gcb_ has quit IRC07:48
*** tesseract has joined #openstack-infra07:48
*** pblaho has joined #openstack-infra07:49
*** ccamacho has joined #openstack-infra07:52
*** alexchadin has joined #openstack-infra07:53
*** egonzalez has joined #openstack-infra07:53
*** cuongnv has quit IRC07:58
*** kjackal__ has joined #openstack-infra08:03
*** kjackal_ has quit IRC08:04
*** kjackal___ has joined #openstack-infra08:04
*** kjackal__ has quit IRC08:08
*** esikachev has joined #openstack-infra08:09
openstackgerritMerged openstack-infra/project-config master: Removal of installguide-cookiecutter
*** apetrich has quit IRC08:12
openstackgerritMerged openstack-infra/project-config master: Add Fedora 26 builds
*** ralonsoh has quit IRC08:12
*** ralonsoh has joined #openstack-infra08:13
openstackgerritMerged openstack-infra/project-config master: Add python and client libray for Rack Scale Design
*** https_GK1wmSU has joined #openstack-infra08:14
*** derekh has joined #openstack-infra08:15
*** ralonsoh has quit IRC08:15
*** apetrich has joined #openstack-infra08:16
*** https_GK1wmSU has left #openstack-infra08:16
*** cuongnv has joined #openstack-infra08:16
*** ralonsoh has joined #openstack-infra08:18
*** pgadiya has quit IRC08:25
*** jpena has joined #openstack-infra08:26
*** efoley has joined #openstack-infra08:27
*** s-shiono has quit IRC08:27
*** efoley has quit IRC08:28
*** efoley_ has joined #openstack-infra08:28
rm_workhey, anyone have any tips on why our devstack logs stopped having color enabled? (octavia)08:29
rm_workis there a common cause for that I can check?08:29
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix github dependent pipeline with merge
*** sree has quit IRC08:30
*** dizquierdo has quit IRC08:32
*** lucas-afk is now known as lucasagomes08:32
*** openstackgerrit has quit IRC08:33
yuvalAJaeger: again :\08:36
yuvalAJaeger: yolanda: Infra cores: same as yesterday and two days ago, gate jobs remain queued08:37
yolandareally? :(08:37
*** yushb has quit IRC08:37
yuvalseems so08:37
yolandaok looking, if that happened at the same hour again, there is a problem for sure08:37
yuvalyolanda: thanks :)08:38
yolandalauncher is active today08:38
*** ralonsoh_ has joined #openstack-infra08:40
*** ralonsoh has quit IRC08:40
yuvalmaybe it was stuck and 'released'08:40
*** pgadiya has joined #openstack-infra08:41
*** wolverineav has joined #openstack-infra08:42
*** wolverineav has quit IRC08:42
*** wolverineav has joined #openstack-infra08:43
*** martinkopec has joined #openstack-infra08:43
*** openstackgerrit has joined #openstack-infra08:47
openstackgerritMerged openstack-infra/tripleo-ci master: Let the pre-upgrade hook stop services in pike+
ianwyuval / yolanda / AJaeger : i don't think things are stuck at the moment.  things are quite busy though because we turned of osic and previously there was a huge merge backlog08:48
yolandayep, in this case, i'm not seeing errors in nodepool08:49
yuvalok, good to hear08:49
ianwfungi will look at restoring the mirror for osic tomorrow (his time) i believe08:50
*** dizquierdo has joined #openstack-infra08:52
*** dchen has quit IRC08:55
*** dingyichen has quit IRC08:55
*** winggundamth has quit IRC08:56
*** slaweq_ has joined #openstack-infra08:59
*** markus_z has quit IRC09:00
*** slaweq_ has quit IRC09:05
*** kjackal_ has joined #openstack-infra09:11
*** kjackal__ has joined #openstack-infra09:13
*** kjackal___ has quit IRC09:13
*** markus_z has joined #openstack-infra09:15
*** lrossetti has joined #openstack-infra09:16
*** kjackal_ has quit IRC09:16
*** dtantsur|afk is now known as dtantsur09:17
*** lrossetti has quit IRC09:19
*** nmagnezi has joined #openstack-infra09:19
*** lrossetti has joined #openstack-infra09:20
*** slaweq_ has joined #openstack-infra09:22
*** priteau has joined #openstack-infra09:24
*** thorst_afk has joined #openstack-infra09:25
*** slaweq_ has quit IRC09:27
*** gcb has joined #openstack-infra09:28
*** esberglu has joined #openstack-infra09:30
*** thorst_afk has quit IRC09:30
openstackgerritJames Page proposed openstack-infra/project-config master: Add Gnocchi charm and associated interfaces
*** sdague has joined #openstack-infra09:32
*** esberglu has quit IRC09:34
openstackgerritJames Page proposed openstack-infra/project-config master: Add Gnocchi charm and associated interfaces
*** sree has joined #openstack-infra09:39
AJaegeryolanda: could you put on your review queue, please?09:40
*** aviau has quit IRC09:41
*** aviau has joined #openstack-infra09:41
*** yushb has joined #openstack-infra09:41
*** esikachev has quit IRC09:42
*** sree has quit IRC09:43
openstackgerritDinesh Bhor proposed openstack-infra/project-config master: Add Python3.5 job to masakari-* projects
*** digv has joined #openstack-infra09:44
digvhi.. is there a way to trigger a job with specific branch of the project by editing layout.yaml?09:44
AJaegerdigv: please explain very specific what you want09:45
digvI want to run patch which comes only on master branch of cinder09:46
AJaegerdigv: if you push a change to gerrit, it runs on the branch you use, so if you run on master, the master tests are run.09:46
openstackgerritJames Page proposed openstack-infra/project-config master: Add Gnocchi charm and associated interfaces
*** tosky has joined #openstack-infra09:48
digvI want to avoid the patch coming on branches other than master, since the ci env setting for other branches are different...09:48
digvso whenever patch based on ocata get proposed on gerrit, I want to avoid it & redirect it to other slave vms in my ci env09:49
*** dizquierdo has quit IRC09:49
openstackgerritJames Page proposed openstack-infra/project-config master: Add Gnocchi charm and associated interfaces
digvhere is conf details from layout.yaml ->
openstackgerritMerged openstack-infra/project-config master: Push translated pages to docs-draft
*** kjackal_ has joined #openstack-infra09:52
*** kjackal__ has quit IRC09:52
*** kjackal__ has joined #openstack-infra09:52
digvis there a way to distinguish the patch depending on the branch & run then run to specific slave09:52
*** e0ne has joined #openstack-infra09:53
AJaegerdigv: you can specify that a specific job only run for changes against some branches. Look at the branch conditions in the layout file09:56
*** kjackal_ has quit IRC09:56
AJaegerdigv: Ah, you're asking for third-party CI? Then best ask on the thirdparty CI operator list.09:56
AJaegerchannel I mean - channel #openstack-third-party-ci09:57
AJaegerdigv: you can define that your jobs should only run if the change is for master09:57
digvAJaeger: How? by only defining "branch: master"?10:01
*** cuongnv has quit IRC10:02
*** namnh has quit IRC10:06
*** hrubi has quit IRC10:06
*** yushb has quit IRC10:06
*** hrubi has joined #openstack-infra10:06
AJaegerdigv: together wit hthe job name10:07
AJaegersee the examples in layout.yaml10:07
*** tuanluong has quit IRC10:07
AJaegersorry, lunch time now...10:07
Dinesh_BhorAJaeger, yolanda: Hi, Please check this when you get time:
*** sambetts|afk is now known as sambetts10:09
*** jaosorior has quit IRC10:10
*** liujiong has quit IRC10:16
*** ramishra has quit IRC10:19
*** ramishra has joined #openstack-infra10:21
*** slaweq_ has joined #openstack-infra10:23
*** esikachev has joined #openstack-infra10:26
digvAJaeger" that helped, thanks10:26
*** slaweq_ has quit IRC10:27
*** jascott1 has quit IRC10:36
*** jascott1 has joined #openstack-infra10:37
*** jascott1 has quit IRC10:41
*** alexchadin has quit IRC10:43
*** zhurong has quit IRC10:43
*** panda has quit IRC10:43
*** panda has joined #openstack-infra10:44
*** rossella_s has quit IRC10:49
*** rossella_s has joined #openstack-infra10:49
*** esikachev has quit IRC10:50
*** markvoelker has joined #openstack-infra10:52
*** esikachev has joined #openstack-infra10:54
*** rossella_s has quit IRC10:54
*** pgadiya has quit IRC10:56
*** markvoelker has quit IRC10:57
*** rossella_s has joined #openstack-infra10:58
*** jaosorior has joined #openstack-infra11:02
*** dizquierdo has joined #openstack-infra11:03
*** pgadiya has joined #openstack-infra11:08
*** kaisers1 has quit IRC11:15
*** gongysh has quit IRC11:15
*** LindaWang has quit IRC11:15
*** digv has quit IRC11:16
*** esberglu has joined #openstack-infra11:18
*** jaypipes has joined #openstack-infra11:18
*** kaisers1 has joined #openstack-infra11:21
*** thorst_afk has joined #openstack-infra11:21
*** jaypipes is now known as jaypipes-jurydut11:22
*** jaypipes-jurydut is now known as jaypipesjuryduty11:22
*** esberglu has quit IRC11:23
*** _ari_|gone is now known as _ari_11:23
*** jkilpatr has joined #openstack-infra11:24
*** rhallisey has joined #openstack-infra11:26
*** gildub has joined #openstack-infra11:27
*** esikachev has quit IRC11:28
*** ldnunes has joined #openstack-infra11:29
AJaegerDinesh_Bhor: could you have others from your team look over it first, please?11:31
*** baoli has joined #openstack-infra11:32
*** baoli has quit IRC11:32
*** baoli has joined #openstack-infra11:32
*** baoli has quit IRC11:33
*** jcoufal has joined #openstack-infra11:34
*** nmagnezi has left #openstack-infra11:36
*** lucasagomes is now known as lucas-hungry11:37
*** kjackal_ has joined #openstack-infra11:37
*** kjackal___ has joined #openstack-infra11:38
*** kjackal__ has quit IRC11:39
*** sree has joined #openstack-infra11:40
*** baoli has joined #openstack-infra11:40
electrofelixfor zuulv3 is it intended to allow multiple sources (gerrit, github, etc) into the same pipeline? or will pipelines from different sources always need to be separate?11:40
*** nicolasbock_ has joined #openstack-infra11:41
electrofelixthought it was going to be the first one as that's where the issue around cross-repo-dependencies mentioned in an email before, but can't track down anything definitive11:41
*** ramishra has quit IRC11:41
*** nicolasbock__ has joined #openstack-infra11:42
*** kjackal_ has quit IRC11:42
*** kjackal___ has quit IRC11:44
*** nicolasbock has quit IRC11:44
*** LindaWang has joined #openstack-infra11:44
*** sree has quit IRC11:45
*** nicolasbock has joined #openstack-infra11:45
*** baoli has quit IRC11:46
*** nicolasbock_ has quit IRC11:46
*** nicolasbock__ has quit IRC11:47
*** kaisers1 has quit IRC11:48
*** markvoelker has joined #openstack-infra11:53
*** dprince has joined #openstack-infra11:54
*** kaisers1 has joined #openstack-infra12:02
*** baoli has joined #openstack-infra12:02
Dinesh_BhorAJaeger: sure, I will request them to give vote12:04
*** rlandy|ci_mtgs has quit IRC12:04
*** jpena is now known as jpena|lunch12:06
openstackgerritMerged openstack-infra/system-config master: Use new OpenStack logo on Gerrit sites
*** kjackal_ has joined #openstack-infra12:07
*** gildub has quit IRC12:07
*** stakeda has quit IRC12:08
openstackgerritMerged openstack-infra/yaml2ical master: Add Adhoc frequency
fungipabelanger: the md5 and sha1 we echo in the logs are to some extent an artifact from before we started making detached signatures and uploading those12:09
funginow that we're signing the files with a private key as a separate step, the checksums aren't as useful12:10
*** jcoufal has quit IRC12:11
fungirm_work: colorization where? when you run devstack locally, or browsing on the site?12:12
*** alexchadin has joined #openstack-infra12:15
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements
*** LindaWang has quit IRC12:18
fungielectrofelix: you might try asking in #zuul as there may be more people in there who know how cross-project deps are supposed to work with mixing different triggers. i got the impression from the decision to switch to dependency urls that it was expected to be a feature, but am unsure how it's supposed to work since i thought pipelines had exclusive triggers12:18
*** markvoelker has quit IRC12:22
*** markvoelker has joined #openstack-infra12:22
*** slaweq_ has joined #openstack-infra12:25
electrofelixfungi: thanks, I keep forgetting there is now a dedicated channel, need to add it to my autojoins12:25
*** ramishra has joined #openstack-infra12:26
*** slaweq_ has quit IRC12:30
*** coreyob has quit IRC12:31
fungielectrofelix: yeah, most of the zuul v3 discussion in here has been focused on migrating the openstack community ci, and so on maintaining feature parity with v2 for our specific use cases (which up to now, non-gerrit services have not been)12:35
*** baoli has quit IRC12:36
*** lucas-hungry is now known as lucasagomes12:39
dhellmanndid anything change in the pipermail setup recently? about every other time I visit a page on lately the CSS fails to load. Reloading the page usually fixes it.12:40
dhellmannthe behavior feels like a loadbalancer that's out of balance12:40
dhellmannI started noticing the issue yesterday. I suppose it could be a browser cache issue on my side, too12:41
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches
*** baoli has joined #openstack-infra12:43
*** sdague_ has joined #openstack-infra12:43
fungihasn't been happening to me (yet anyway)12:44
*** gordc has joined #openstack-infra12:45
fungiwe're still getting slammed with attempts to spam some of the -owner addresses there (>100k of them yesterday) but system load remains low12:46
dhellmannI'll restart my browser and see if it stops12:49
*** baoli has quit IRC12:51
*** Goneri has joined #openstack-infra12:52
*** gouthamr has joined #openstack-infra12:56
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches
mnaseryay the logo update merged12:57
mordreddhellmann: I read that weirdly (still coffeeing) and thought for a brief second you were going to restart your browser to see if the spam flood stopped12:58
dhellmannmordred : that would be something, wouldn't it?12:58
mordreddhellmann: I mean, sadly, it's not unpossible12:59
fungii'm going to restart my browser and see if it starts my day over13:00
fungithat could be a handy trick if i can make it work13:00
*** camunoz has joined #openstack-infra13:00
*** yamamoto has quit IRC13:02
*** xyang1 has joined #openstack-infra13:04
*** jpena|lunch is now known as jpena13:06
*** jcoufal has joined #openstack-infra13:07
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
*** gordc has quit IRC13:09
*** esberglu has joined #openstack-infra13:11
*** abelur_ has quit IRC13:14
*** bobh has joined #openstack-infra13:14
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command
*** yamamoto has joined #openstack-infra13:18
pabelangerfungi: ya. I think for now I'll look to just print the info into logs.13:20
mnaserlooks like infracloud is having issues by any chance?13:20
pabelangerwhat are you seeing?13:21
*** kgiusti has joined #openstack-infra13:21
mnasermirrors unable to connect13:21
mnaseri saw this on 3 builds13:21
pabelangerwonder if the mirror is under load13:22
pabelangerwill osic gone, it will get more jobs scheduled to it13:23
pabelangerlooks doen13:23
pabelangerunable to SSH to too13:24
*** LindaWang has joined #openstack-infra13:24
*** adarazs has left #openstack-infra13:26
pabelangermnaser: do you want to propose a patch to project config setting max-server for infracloud-vanilla to 0. Until we get the mirror back online13:27
mnaserpabelanger sure np13:27
mnaser1 sec13:27
*** sree has joined #openstack-infra13:27
*** sree has quit IRC13:28
*** prateek has quit IRC13:28
*** sree has joined #openstack-infra13:29
*** pgadiya has quit IRC13:29
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Set infracloud-vanilla servers to zero.
mnaserpabelanger ^13:29
mnaseri think i did it right? :p13:29
pabelangermnaser: yup, thanks.13:31
pabelangerjust looking at logs on controller to see if I can see the reason13:31
*** felipemonteiro has joined #openstack-infra13:31
*** baoli has joined #openstack-infra13:32
*** jistr is now known as jistr|mtg13:32
*** xinliang has quit IRC13:32
*** felipemonteiro has quit IRC13:37
pabelangerya looks to be down13:37
pabelangerand mirror lives on that13:37
fungithat's unfortunate13:38
pabelangerjust +3 490025 for now13:38
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
pabelangerwill see if I can get into ilo and reboot13:38
*** rwsu has joined #openstack-infra13:41
*** mriedem has joined #openstack-infra13:41
fungiwant me to manually apply that on nodepool.o.o for now? i have it in emergency disable since the change to stop using osic never got approved13:42
pabelangerfungi: ah, ya please. Looks like your review is in patch conflict now13:43
fungii just went ahead and updated it on the server13:43
fungialso want me to delete all nodes in that region so we don't have more test failures?13:43
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command
*** sdague has quit IRC13:43
pabelangerfungi: might as well, they will fail13:44
*** xinliang has joined #openstack-infra13:44
fungithis way at least zuul will hopefully requeue most of the ones that hadn't finished yet13:44
pabelangerilo looks down too13:45
pabelangerno route13:45
*** alexchadin has quit IRC13:45
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output
pabelangerI'll rebuild the mirror shortly13:48
pabelangerneed to top up on coffee first13:48
fungiyeah, just finished doing the same, starting work on the osic mirror rebuild now13:51
openstackgerritMerged openstack-infra/project-config master: Set infracloud-vanilla servers to zero.
*** aarefiev is now known as aarefiev_afk13:51
*** jistr|mtg is now known as jistr13:52
mnasershould we run rechecks (or will you make a small announcement when we're ready to recheck?)13:52
pabelangermnaser: ya, let me notice now13:53
cloudnullfungi: if you need anything in the osic to help get the mirror online, let me know.13:54
cloudnullI did check in on the old node, its still dead13:54
cloudnulland the cinder volume node has been powered down13:54
Odd_Blokezxiiro: o/ I have a project which would be made much easier with; I was wondering if I could get a high-level indication of whether or not that include syntax is acceptable; if it is, I'm happy to work against my fork of JJB whilst the details of getting it merged are ironed out but I don't want to be depending on something that has no chance of getting13:54
Odd_Blokemerged. :)13:54
*** yamamoto has quit IRC13:55
fungicloudnull: nope, just need to run a few canned commands/scripts to launch the replacement, create and attach the new volume, and update dns13:55
fungicloudnull: but thanks!13:55
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci master: Enable repo for python-os-testr
pabelanger#status notice We have disable infracloud-vanilla due to the compute host running mirror.regionone.infracloud-vanilla.o.o being offline. Please recheck your failed jobs to schedule them to another cloud.13:55
openstackstatuspabelanger: sending notice13:55
zxiiroOdd_Bloke: please join #openstack-jjb I'd defer that question to some of the others since I'm not familiar with that part of the code.13:55
*** sdague has joined #openstack-infra13:56
-openstackstatus- NOTICE: We have disable infracloud-vanilla due to the compute host running mirror.regionone.infracloud-vanilla.o.o being offline. Please recheck your failed jobs to schedule them to another cloud.13:57
*** zhurong has joined #openstack-infra13:57
Odd_Blokezxiiro: Oh, nice, didn't know JJB had its own channel.  Have joined and posted there. :)13:58
openstackstatuspabelanger: finished sending notice13:59
zxiiroOdd_Bloke: yep it was created recently. makes it easier for us to communicate13:59
fungimnaser: looks like gerrit was smart enough to apply the theme change without a restart. hurrah!14:03
*** nicolasbock_ has joined #openstack-infra14:04
*** nicolasbock__ has joined #openstack-infra14:07
fungicloudnull: as suspected, can't delete the server instance, nor detach/delete the volumes from it, while the instance is still in an error state14:07
*** nicolasbock has quit IRC14:07
fungicloudnull: i don't suppose there's an easy trick to cleaning that up?14:07
fungithe api accepts the server delete call but just seems to queue it, because the instance sticks around still in an error state14:08
mnaserfungi: yes i notice, looks nice :)14:08
mnaserfungi do you have admin access to that cloud?14:08
mnaseryou can do a force detach with cinder14:09
fungimnaser: nope, which is why i'm pestering the generous admin14:09
mnaseroh okay :p14:09
*** andreas_s has quit IRC14:09
fungidoesn't look like openstack server remove volume --force is a thing, at least for me14:10
EmilienMpabelanger: really awesome for the buildlogs mirror, thanks14:10
*** nicolasbock_ has quit IRC14:10
*** hongbin has joined #openstack-infra14:10
pabelangerEmilienM: thanks, will help also.14:13
fungicloudnull: i found a sufficient workaround in the interim... renamed the instance and volumes to broken, broken1 and broken2 so i don't have to worry about confusion in our ansible inventory14:14
pabelangerfungi: ^ mind a +3 on NODEPOOL_BUILDLOGS_CENTOS_PROXY variable?14:14
EmilienMpabelanger: I'm already preparing the patches to use the mirror in puppet & tripleo CI14:14
*** slaweq has quit IRC14:14
*** mriedem1 has joined #openstack-infra14:14
pabelangerEmilienM: ack14:15
*** slaweq has joined #openstack-infra14:15
*** esikachev has joined #openstack-infra14:15
pabelangernew mirror IP address for infracloud-vanilla14:16
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
pabelangerworking on DNS changes now14:16
*** mriedem has quit IRC14:16
pabelanger#status log DNS updated to
openstackstatuspabelanger: finished logging14:18
*** jcoufal has quit IRC14:18
pabelangerfungi: mnaser: should hit new server in 60min, current DNS TTL14:19
*** slaweq has quit IRC14:19
pabelangerfungi: mnaser: new nodepool VM should get it out of box14:19
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci master: Default $NODEPOOL_PROVIDER
*** prateek has joined #openstack-infra14:20
*** mriedem1 is now known as mriedem14:20
mnaserpabelanger so should we revert the patch?14:21
pabelangermnaser: ya14:21
openstackgerritMohammed Naser proposed openstack-infra/project-config master: Revert "Set infracloud-vanilla servers to zero."
*** nicolasbock_ has joined #openstack-infra14:21
fungipabelanger: any recollection of which network we should pass the launch script for building in osic? we have a GATEWAY_NET and a GATEWAY_NET_V6 there14:21
fungii suppose i could just try one... i have a 50/50 chance of picking correctly14:22
openstackgerritMerged openstack-infra/project-config master: Add buildlogs.centos to nodepool variables
*** rloo has joined #openstack-infra14:22
*** eharney has joined #openstack-infra14:22
pabelangerfungi: I believe GATEWAY_NET_V614:23
fungidarn, i picked the more generic of the two14:23
*** annegentle has joined #openstack-infra14:23
fungirestarted with your recommendation now14:24
*** nicolasbock__ has quit IRC14:25
pabelangergoing to revert change on nodepool.o.o now for infracloud14:27
fungithe dns change has already propagated for that?14:27
*** slaweq has joined #openstack-infra14:27
fungii thought you said the ttl was an hour14:27
pabelangerfungi: it has a TTL of 60mins, but new server should be fine right?14:27
pabelangersince they wouldn't have cached the request14:28
fungionly if it's not forwarding dns queries to an external recursor14:28
fungiit's caching at the external recursor you have to worry about there14:28
pabelangerRight, was assuming local unbound server would look it up first, before passing to google / opendns14:28
fungi(opendns, google, et cetera)14:28
pabelangerya, okay. I'll hold off for the 60mins14:28
pabelangergate isn't that bad atm14:29
fungilocal unbound server is configured as a caching forwarder, which means if it doesn't find the record in its local cache it forwards the request to another recursove resolver14:29
*** rwsu has quit IRC14:29
fungier, recursive14:29
* mnaser thought unbound was doing recursive lookups14:29
mnaserso i thought thats why it was ok for CI instances14:29
mnaseris there a way to manage gerrit email notifications? i keep a close eye on it all the time so getting all the emails is a bit redundant, i looked thought settings but couldnt find much14:30
mnaser(and while i was there i found out you can create your own menu, pretty cool)14:30
*** jcoufal has joined #openstack-infra14:30
*** rbrndt has joined #openstack-infra14:32
*** slaweq has quit IRC14:32
*** bobh has quit IRC14:32
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix github dependent pipeline with merge
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove getProjectBranches from FakeGithubConnection
fungimnaser: pabelanger: sets up query forwarding14:34
fungiwith forward-addr directives in the forward-zone block14:35
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Update tox-tarball to use tox role
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels
mnaserfungi oh gotcha14:36
*** ykarel has quit IRC14:36
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Use py35 tox envlist for documentation
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
openstackgerritMerged openstack-infra/project-config master: Revert "Set infracloud-vanilla servers to zero."
*** zhurong has quit IRC14:38
mnaseris there a way of hooking into job timeouts (or an example of a project that does it)?  unfortunately a timed out job has no data helpful to do much with14:39
fungimnaser: gerrit's e-mail notification features are pretty inflexible. you can "watch" specific projects for certain events (new changes, new patch sets, all comments, submitted changes, abandoned changes) but any changes you own, comment on or are added to as a reviewer by someone else will get notification for all those event types unless you remove yourself as a reviewer on the individually and any14:39
fungivotes as well (and if you're the change owner you're stuck regardless)14:39
*** ralonsoh_ has quit IRC14:39
*** rwsu has joined #openstack-infra14:39
fungimnaser: i have some semi-complex server-side filters on my personal mta to change which inbox certain gerrit event notifications go into, in an effort to wrangle the madness14:39
mnaserfungi: well, i was going to do that anyways but i figured less work for gerrit rather than me filtering it14:40
*** xarses has joined #openstack-infra14:40
*** gordc has joined #openstack-infra14:41
dtantsurfolks, how do I bother with gerrit problems? there are two accounts associated with my email: The 2nd one (ID 10239) is what I use, the first one is what the election scripts get..14:41
dtantsurending up with an non-sense full name (and potentially other issues)14:42
fungiyeah, i have gerrit set not to watch any projects, and then my exim filter looks like: (some of which may be obsolete/redundant these days as i haven't touched it in years now)14:42
fungimnaser: ^14:42
*** LindaWang1 has joined #openstack-infra14:43
*** panda has quit IRC14:43
mnaserfair enoguh14:43
* mnaser creates filters14:43
fungidtantsur: i can try to work out which account to disable here in a bit14:43
*** panda has joined #openstack-infra14:44
*** LindaWang has quit IRC14:45
*** prateek has quit IRC14:45
*** LindaWang1 is now known as LindaWang14:45
*** jamesmcarthur has joined #openstack-infra14:46
fungipabelanger: when i tried to launch with --network=GATEWAY_NET_V6 it never managed to connect to do the puppet apply steps... when i used --network=GATEWAY_NET it worked, but the instance has no ipv6 address and no eth1 interface attached... am i supposed to pass the launch script multiple networks?14:48
*** bauzas has quit IRC14:50
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels
clarkbfungi: assuming  osic I believe you need two interfaces one for each network14:52
fungiyeah, just trying to work out how to pass that through launch-node.py14:52
fungidoesn't look like the --network option takes repetition nor any sort of iteration over the value provided14:53
*** armax_ has joined #openstack-infra14:53
*** armax has quit IRC14:53
*** armax_ is now known as armax14:53
pabelangerfungi: clarkb: with --network=GATEWAY_NET_V6 did it just get an IPV6 address? or ipv4 too?14:54
fungipabelanger: dunno, the script just kept complaining that it couldn't ssh into the server14:54
clarkbit gets both but I dont think you can fip on that network14:55
pabelangerfungi: I'd try again, and look at openstack server show $FQDN14:55
*** nunchuck has joined #openstack-infra14:55
fungi--network=GATEWAY_NET,GATEWAY_NET_V6 gives me "Network ['GATEWAY_NET,GATEWAY_NET_V6'] is not a valid network in openstackci-osic-cloud1:RegionOne" so i'm testing with --network=GATEWAY_NET --network=GATEWAY_NET_V6 now but suspect it's just taking the second one14:55
clarkbso you need the other network too to fip (and other network may be our normal private thing in neutron)14:55
fungiWhile testing ssh access: [Errno -2] Name or service not known14:55
*** Douhet has quit IRC14:55
*** yamamoto has joined #openstack-infra14:55
fungiis all i'm getting, same as when i only used --network=GATEWAY_NET_V614:56
pabelanger--config-drive flag too?14:56
fungiwondering if someone booted the previous instance bespoke instead of with our script14:56
pabelangerso cloud-init is setup14:56
fungino idea... the readme doesn't say anything about needing --config-drive14:56
*** Douhet has joined #openstack-infra14:57
pabelangerYa, usually for rax we don't14:57
pabelangerbut other clouds I've needed to use it14:57
pabelangerso, I just always pass it now14:57
dmsimardI ran a pre-release tag to test the release jobs for ARA and it looks like it adds an extra ".0" digit to the versioned file on tarballs.o.o ?14:57
dmsimardSo the signing and pypi upload jobs failed on
dmsimardThey are trying to get but the file is (extra zero at the end)14:58
ttxfungi: looking up ptgbot status. I found the website, but the bot doesn't seem to be running ? Also if we do an index page, would you rather have it live in puppet-ptgbot or the ptgbot repository ? Probably puppet-ptgbot since it needs to change at each event14:58
fungidmsimard: you have to make sure your tags follow pbr's notion of semver+pep440 version string normalization14:58
fungittx: i have a change proposed for the index page for a few weeks already, just waiting on infra reviewers to have bandwidth to review it14:59
clarkbfungi: I think we largely replaced the script since osic wa first booted14:59
ttxok, will review it14:59
fungittx: i can look into what's going on with the service in a bit, but backlogged on several other problems this morning14:59
*** felipemonteiro has joined #openstack-infra15:00
* ttx adds puppet-ptgbot to watch lists15:00
fungittx: the change in question should be using topic:ptgbot if memory serves15:00
*** yamamoto has quit IRC15:00
*** slaweq has joined #openstack-infra15:01
dmsimardfungi: ara uses pbr and I have no problem with it, I think. Tag is 0.14.0rc315:01
*** ykarel has joined #openstack-infra15:01
*** marst has joined #openstack-infra15:01
*** links has quit IRC15:01
ttxfound it @
*** felipemonteiro_ has joined #openstack-infra15:04
*** felipemonteiro has quit IRC15:04
*** felipemonteiro__ has joined #openstack-infra15:04
*** jamesmcarthur has quit IRC15:05
*** jistr is now known as jistr|mtg15:06
*** jamesmcarthur has joined #openstack-infra15:06
*** felipemonteiro__ has quit IRC15:06
*** felipemonteiro__ has joined #openstack-infra15:07
*** felipemonteiro_ has quit IRC15:08
clarkbfungi: pabelanger I want to say you can't attach floating IPs to the ipv6 external network and you can't direct attach a VM to the external ipv4 network. So that forced us to have two interfaces one for ipv6 with private v4 and a second with private v4 that we could attach fips to15:08
clarkbfungi: pabelanger also we may have added ipv6 interface later as osic ipv6 didn't exist when we first deployed into that region so we may not have booted it with two networks but just one and added a network later15:09
fungidmsimard: right, is the pep440 normalized release candidate version string. that's what you should use in your tag15:09
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for missing host mapping bug 1708039
openstackbug 1708039 in grenade "gate-grenade-dsvm-neutron-multinode-ubuntu-xenial fails with "No host-to-cell mapping found for selected host"" [Undecided,New]
fungidmsimard: 0.14.0 is the release it's working toward, and 0rc3 is the prerelease version. prerelease version components are separate from the 3 semver version components (because the semver spec doesn't incorporate the notion of prereleases) but pep440 normalization needs version components to start with a digit15:11
fungidmsimard: best way to double-check in the future is before pushing the tag try running `tox -e venv python sdist` and look at what version number it gives the tarball15:12
fungiif it doesn't match, git tag -d and reapply the correct string15:12
*** thorst_afk has quit IRC15:14
*** esikachev has quit IRC15:14
*** dprince has quit IRC15:14
fungiclarkb: pabelanger: trying with --network=GATEWAY_NET --config-drive now to see where that goes15:15
dmsimardfungi: so I guess what I really want is 0.14rc3 then ?15:15
*** e0ne has quit IRC15:16
clarkbfungi: I think that may fail due to not being able to direct attach to the external ipv4 network. I think we have a router, network,subnet for v4 private networking that you then attach floating IPs to from GATEWAY_NET15:16
clarkbfungi: but give it a go, neutron/nova should error quick if I am right15:16
fungidmsimard: 0.14rc3 will also normalize to
*** esikachev has joined #openstack-infra15:17
fungidmsimard: pbr uses normalized versions in the form of <major>.<minor>.<patch>.<pre> where major, minor and patch can only be digits with leading zeroes omitted if there's more than one digit in the component, and pre starts with 0 followed by a, b or rc plus one or more digits (i believe also with leading zeroes removed)15:18
* fungi goes to find the pbr docs for this15:19
fungidmsimard: is the dissertation15:19
dmsimardWell I'm sure pbr is doing the right thing, it just goes against what I felt was ok :)15:19
*** armax has quit IRC15:19
dmsimardFor me, this is a rc of 0.14.0, so feels out of place15:20
dmsimardBut what do I know, I'm not a software guy15:20
*** cshastri has quit IRC15:20
fungidmsimard: well, it's not it's 0.14.0 release candidate 315:20
*** nicolasbock__ has joined #openstack-infra15:21
fungithe release candidate 3 part is encoded as a prerelease version component "0rc3"15:21
dmsimardRight, so 0.14.0rc3, not ?15:21
fungi0rc3 is not a semver-compliant release version component15:21
fungiso it has to be the fourth component (semver only covers the first three components)15:22
*** esikachev has quit IRC15:22
*** psachin has quit IRC15:22
fungialso that documentation predates a minor change to setuptools pep440 version normalization so says 0.14.0.rc3 when in actuality setuptools turns that into
*** LindaWang has quit IRC15:23
*** electrofelix has quit IRC15:23
fungibecause setuptools interpretation of pep440 is that version components must start with a digit and so normalizes anything starting with a letter by prepending a 015:23
dmsimardI'll try poking around with sdist and figure out if I can find something I'm happy with :D15:24
cloudnullfungi: sorry was in a meeting.15:24
dmsimardThanks for your help15:24
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output
cloudnullyes the node is in error, I can nuke it whenever15:24
*** nicolasbock_ has quit IRC15:24
cloudnulljust let me know when15:24
fungicloudnull: no worries, it finally seems to have deleted on its own15:24
fungiand one of the two volumes detached so i was able to delete it15:25
fungithe other was still showing attached to the nonexistent instance id last time i looked though, so wouldn't allow me to detach or delete15:25
clarkbif you haven't deleted teh node yet you should be able to show it to get the network setup info15:26
clarkbfungi: ^15:26
fungiyeah, volume list says volume 975d5bd9-37f5-4cfe-965a-a917056de961 is in-use attached to instance 8b5b85b7-aaca-45a7-9e1a-5c35903f3079 but server remove volume says instance 8b5b85b7-aaca-45a7-9e1a-5c35903f3079 doesn't exist15:26
fungiclarkb: well, it booted and puppeted similar to when i didn't pass --config-drive but it only has an interface on GATEWAY_NET so far15:27
*** annegentle has quit IRC15:28
fungiclarkb: presumably i just need to work out the openstackclient command necessary to attach GATEWAY_NET_V6 on a second interface?15:28
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for troveclient AttributeError bug 1708190
openstackbug 1708190 in python-troveclient "AttributeError: 'module' object has no attribute '_logger'" [Undecided,Confirmed]
*** armax has joined #openstack-infra15:28
clarkbfungi: ya that is what I am beginning to think we did because osic was up and running ipv4 only for a month or two before we added ipv6. Guessing we just tacked on the interface and didn't boot a new instance15:29
fungido we have a cheat sheet for the syntax? otherwise i'll poke around in context help for a but15:29
clarkbI don't know that we do15:30
clarkbpabelanger: ^ I think you may have added ipv6 to osic, do you recall?15:30
clarkbwe sort of played with it in cityclodu too but that was mostly how to make new network/subnet using subnet pools15:30
cloudnullclarkb: fungi: you could use the IPv6 net and then assign a FIP to the v4 subnet if that makes life easier for folks15:30
*** ralonsoh has joined #openstack-infra15:31
*** martinkopec has quit IRC15:31
clarkbcloudnull: I think fungi was having problems with ssh working initially on that network15:32
fungiclarkb: i think the issue there is our considers whatever you provide via --network as the thing you want attached to eth015:32
fungiand then the instance gets booted with only ipv6 and (something) breaks15:33
morgandid we change the openstack logo on gerrit?15:33
*** esikachev has joined #openstack-infra15:33
morganit looks... funky now15:33
fungimorgan: yes15:33
*** armax has quit IRC15:33
morganftr, i think this version isn't as nice. the O is really fat and the color on "openstack text" is an odd grey. (totally personal opinion)15:34
morganthe wide-ness of the O logo thing is feeling weird. it might be fine once i'm used to it15:35
morgan(it's really... square)15:35
*** Sukhdev has joined #openstack-infra15:35
fungimorgan: if there's a different one available at we should use instead, that can probably be arranged. otherwise you want to talk to the design team at the foundation15:36
morganyeah it's so much just a personal view, i just was voicing my thoughts ;)15:36
morganprrrrooooobably not worth changing15:36
*** annegentle has joined #openstack-infra15:36
morganit's about the only place i see the logo :P15:37
*** dizquierdo has quit IRC15:37
morganso, it surprised me to see a different one15:37
*** ym200 has joined #openstack-infra15:37
pabelangerclarkb: fungi: I cannot remember. I _think_ we created eth1.cfg after the server booted for ipv615:37
clarkbmorgan: I want to say this logo was first used in austin so its been cooking for a while15:39
fungi(the newton summit in austin, not the austin release)15:40
morganclarkb: hm. maybe? i also havent' been to any summits really15:40
*** vhosakot has joined #openstack-infra15:41
fungibut yeah, that version of the logo has been on the www site for around a year i think, and on wiki.o.o for a while, and the updated docs.o.o theme for at least a few months15:41
Sukhdevfungi - can I bother you with one question regarding building Third party CI for a community project?15:42
fungimorgan: and the change to update it in the gerrit webui was proposed back in april, we just didn't get around to approving it until a few hours ago15:42
clarkbpabelanger: that sounds right15:42
morganfungi: heheh15:42
fungiSukhdev: i may not know the answer but doesn't hurt to ask15:42
clarkbfungi: pabelanger so ya we probably want whatever neutron incantation says attach network to existing server. Then configure it quickly15:43
*** gyee has joined #openstack-infra15:43
fungiSukhdev: there's also a #openstack-third-party-ci irc channel which may be more appropriate depending on your question15:43
*** sdague_ has quit IRC15:43
Sukhdevfungi : thanks - I am considering building a CI system and was told that for community related projects I could use nodepool infra to build it -15:43
Sukhdevfungi : was looking for some docs/pointers15:44
mnaserclarkb -- if you have the branch locally still handy and can do a quick git add -- i can show the edits in screenshot so git.o.o cna look nice and pretty as well :>15:44
Sukhdevfungi : let me ping on other channel as well15:44
fungiSukhdev: please provide details; it's unclear to me whether you're talking about building a third-party ci system with a nodepool server, or configuring jobs to run in our upstream ci system15:44
clarkbfungi: pabelanger `openstack server add fixed ip` maybe?15:45
Sukhdevfungi : build the CI system for networking-opencontrail project15:45
clarkbmnaser: thats a good question, let me go look15:46
*** e0ne has joined #openstack-infra15:46
fungiSukhdev: not the detail i was looking for. do you want to build a ci system for the networking-opencontrail project, or configure jobs in our ci system to test changes to the networking-opencontrail project?15:46
*** mtanino_ has joined #openstack-infra15:46
Sukhdevfungi: the former15:46
pabelangerclarkb: don't believe so. IIRC, we rebuilt using GATEWAY_V6, and both ipv4 and ipv6 attached to the server.  But we needed to manually create eth1.cfg on disk15:47
fungiSukhdev: maybe a pointer to where you had the original conversation about "for community related projects I could use nodepool infra to build it" would help me better figure out what you're asking for15:47
pabelangerI don't think I ran any openstack commands for networking15:47
*** ralonsoh has quit IRC15:47
Sukhdevfungi : In order to kick off the tests for the CI system, I need nodes to launch the jobs on - I was told I could use nodepools for that15:47
*** ralonsoh has joined #openstack-infra15:47
clarkbmnaser: I think it got committed in an earlier change15:48
fungiSukhdev: we do maintain software for a suite of services which will allow you to dynamically boot and delete a pool of server instances in an one or more openstack clouds to which you obtain access15:48
Sukhdevfungi : actually somebody mentioned this during a discussion at Boston summit - hence, I started to look for some pointers and did not find anything15:48
clarkbmnaser: so its using the same svg file that we used on the gerrit update15:49
fungiSukhdev: are you asking if you can run that software? if so, the answer is yes, we distribute it under a free and open license so anyone who wants to do so can run a copy of it15:49
fungiSukhdev: and many third-party ci operators are running nodepool to manage their server resources for jobs, as i understand it15:49
Sukhdevfungi : yes - that is exactly what somebody mentioned - where can I get information to read up on it so that I can replicate what these guys are doing15:50
fungiSukhdev: but the fact that you seem to be asking whether or not you're allowed to run free software we publish seems odd, so i have to assume that's not really what you mean to ask15:50
clarkbmnaser: but looks like I have to rebase to address merge conflicts anywayts15:50
mnaserclarkb oooh gotcha15:50
clarkbmnaser: so let me push up a rebase really quickly15:50
mnaseronce you do that i'll do the same for the other patch so reviewers dont have to go through that :p15:50
fungiSukhdev: start reading and see if the documentation there answers your questions15:51
fungiSukhdev: it has a section on nodepool specifically, but that's just one component of the suite of ci infrastructure described in that documentation15:52
clarkbmnaser: oh it doesn't need a rebase it needs to figure out why failed15:52
clarkbfungi: ^ that looks related to ssh key cleanup, is that a known problem?15:52
Sukhdevfungi : thanks - this is exactly I need. This will get me going.15:52
Sukhdevfungi : sorry I confused you a bit the way I asked the question - but, now we on the same page - and I think this documentation will answer my questions15:53
clarkbI've rechecked it on the off chance we have a conflict between what the instance booted with and what is in puppet15:54
*** bauzas has joined #openstack-infra15:55
*** rkukura has joined #openstack-infra15:55
fungiSukhdev: yep, your specific mention of nodepool was what was confusing me i think, since you were looking for documentation/process about setting up a ci system in whole15:55
*** Apoorva has joined #openstack-infra15:56
Sukhdevfungi : thanks a ton15:56
fungiSukhdev: if you're interested in the details of nodepool itself, we have detailed documentation about its configuration and services at but as i said that's just one component of a larger suite of ci software15:57
mnaserclarkb looks like we'll have to do a little patch for the openstack logo15:57
*** ykarel has quit IRC15:57
* mnaser is going to grab something to eat15:58
*** Guest83693 has quit IRC15:58
fungiclarkb: that's a very odd error... the changes to remove and clean up accounts/keys haven't even been approved yet15:58
AJaegermnaser, clarkb : what a nice "small" logo.15:59
*** links has joined #openstack-infra15:59
clarkbfungi: you know I think we've seen this and attributrd it to nova or glean fails16:00
*** lrossetti has quit IRC16:00
*** lrossetti_ has joined #openstack-infra16:00
*** makowals has quit IRC16:00
clarkbfungi: where we write invalif key data to the file (notice lack of rsa prefix)16:00
*** ihrachys has joined #openstack-infra16:00
*** egonzalez has quit IRC16:01
clarkbmnaser: I guss that is downside to an svg?16:01
*** esikachev has quit IRC16:02
*** sdague has quit IRC16:03
*** gyee has quit IRC16:03
jlvillalclarkb: fungi: and anyone else who cares. Is it a big deal if I do push 60 or so patches that all will cause one pep8 job each to run? Seems to me about the same load as me uploading 3 ironic patches.16:03
jlvillalBut I noticed people raising concerns yesterday about it. So wanted to double check.16:03
fungijlvillal: the issue is more with the fact that zuul needs to calculate pre-merge stacked references the entire way up the chain16:04
jlvillalfungi: Oh. I'm guessing that is bad? /me doesn't know how zuul works very well16:05
fungiso ends up being n+(n-1)+(n-2)+... merge operations16:05
fungithe fact that it's 60 interdependent patches is the resource contention. if it were 60 unrelated patches it would handle it without breaking a sweat16:06
fungibut it ended up basically deadlocking zuul for a few hours yesterday when you did it16:06
jlvillalfungi: Ah.16:06
jlvillalOuch :(16:06
clarkbone work around is to just push 5 at a time or so until you get them all in16:06
jlvillalSo, if I push 5, wait till they all finish their tests. Then push next 5, etc..16:07
jlvillalWill that final patch that has 60 merge dependencies cause big issues?16:07
fungialternatively, if you have a git state you need pushed into the repo and don't actually have any need to test them as they go in, then one of our gerrit sysadmins can batch push them straight into the repo and bypass ci (basically to simulate what would have been done at import if you had specified an initial repo state to import)16:07
jlvillalfungi: Let me talk to dtantsur about that.16:08
jlvillalI think we would be willing to do that. Well I am willing to do that :)16:08
*** rcernin has quit IRC16:08
* dtantsur reads16:09
jlvillaldtantsur: This is in regards to ironic-tempest-plugin16:09
*** Rockyg has joined #openstack-infra16:09
fungiwe discourage that because we want people to import initial state and save us the work of doing it manually, but that train has sailed so it may be a viable compromise over monopolizing the ci scheduler queue for hours at a time16:09
dtantsurjlvillal, fungi, the plan sounds good to me16:10
fungithough of course if you can wait a few days to upload and dribble your changes in a few at a time, that's also fine from our perspective16:10
dtantsurI also don't care too much about preserving git history, so I'm fine with one patch imorting everything16:10
jlvillalfungi: dtantsur: Thanks. I will work on getting the patch set ready.16:11
dtantsurjlvillal: please don't forget that we agreed to include ironic-inspector tests in the same repo16:11
fungiwell, we're talking about preserving git history by pushing the commits straight into gerrit16:11
fungibypassing code review and testing16:11
dtantsurjlvillal: this may be another patch chain that we somehow merge with this...16:11
jeblairjlvillal: (to answer the earlier question -- no the single patch with 60 parents at the end isn't bad -- it will only monopolize one of the mergers; we have 8 -- as long as some of them are free, other things can keep moving)16:11
dtantsurfungi: right, yeah, I'm talking about another (less favorable) option16:11
fungiso just get us a url to a git repo somewhere with the desired state into it16:11
fungiif you need us to do that16:12
jlvillalfungi: dtantsur: I will work on it.16:12
*** rkukura has quit IRC16:13
e0nehi team! does depends-on flag work for client and libs dependencies?16:13
openstackgerritClark Boylan proposed openstack-infra/project-config master: Add accessbot to #openstack-ptg
*** pcaruana has quit IRC16:14
clarkbfungi: ttx ^ that should bootstrap the access we need with chanserv so that you can restrict administrator commands to ptgbot16:14
*** slaweq has quit IRC16:14
pabelangere0ne: should work for any project in openstack namespace16:14
fungie0ne: as long as your jobs are written to install those dependencies from source and uses something like zuul-cloner or devstack-gate to obtain the correct refs for them16:14
pabelangeryes, that too16:14
e0nepabelanger: good to know. how can I check it?16:14
*** slaweq has joined #openstack-infra16:14
fungie0ne: generally if your project has a need to test against, say, oslo.config from source you will define a specific job that installs it from source instead of using normal release version dependencies for that16:15
e0nefungi: we've got nova patch that depends on a cinderclient patch16:16
fungie0ne: and then that job will use the proposed changes you depend on for oslo.config when testing your changes16:16
e0nefungi: job's logs
fungie0ne: so nova's devstack-gate jobs will test with the cinderclient depends-on automatically, that should work16:16
mordredinfra-root: gonna send the zuul v3 email now unless anyone has any last-minute comments or concerns16:16
openstackgerritMerged openstack-infra/system-config master: Use https where possible for status site things
ildikovfungi: there's a devstack change too in the depends-on list16:17
fungimordred: can't remember if i said (because i got ratholed on osic mirror breakage when trying to get the related infra manual change to merge) but message looked fine. i left a couple of small edits i think16:17
*** panda is now known as panda|bbl16:17
ildikovfungi: which seems to be needed at this stage to get the system work again16:18
ildikovfungi: does that work out of the box too?16:18
e0nenova patch which we're talking about16:18
jeblairmordred: can i tweak your mention of the zuulv3 infra manual page real quick16:18
*** armax has joined #openstack-infra16:18
fungiildikov: e0ne: multiple depends-on should also work for those devstack-gate jobs16:18
clarkbfungi: e0ne keep in mind on most jobs clients and libs are installed from pypi not from source16:18
mordredjeblair: absolutely!16:19
clarkbfungi: e0ne you need the special src jobs that we run against the clients and libs to test with them installed from source16:19
*** slaweq has quit IRC16:19
ildikovclarkb: the logs look like the devstack change is not applied16:19
fungiclarkb: i thought devstack-gate based jobs installed them from source?16:19
fungiwow that's a lot of depends-on footers16:19
clarkbfungi: no that changed a few years back (it changed in devstack)16:19
mordredfungi: not by default, no - they use released versions16:19
e0neclarkb, fungi: does it mean that cinderclient will be installed from pypi even we've got depends-on flag?16:19
ildikovfungi: my bad, I will clean it up16:20
mordredyah - you have to opt-in to -src jobs16:20
fungiokay, so it's just services which get depends-on handled in normal devstack-gate jobs i guess16:20
clarkbe0ne: yes unless you are running the opt in src job which installs it from source instead16:20
jeblairmordred: okay added a sentence there, look okay?16:20
*** gyee has joined #openstack-infra16:20
mordredjeblair: yup!16:20
jeblairmordred: i'm all set, thanks!16:20
fungie0ne: anyway, the recommended order is to get features you need in clients/libraries merged and released before you depend on them in services. it would be _bad_ to end up with a nova release depending on an unreleased feature in cinderclient16:21
clarkbe0ne: all of the libs and clients themselves should run that job so you can put them at the tail of a chain to get it all tested together from source16:21
*** jistr|mtg is now known as jistr16:22
fungiyep, and as long as we're gating separately on jobs which require testing with the released dependencies too, we avoid accidentally releasing services depending on unreleased library features16:22
ildikovI cleaned up the dependency list by removing what's merged already16:23
*** mpaolino has quit IRC16:23
mordredfungi: yah - exactly. one job that does from source with depends-on and one that does from releases can show that the new features work end-to-end and also don't break with old library releases16:23
openstackgerritAlex Schultz proposed openstack-infra/yaml2ical master: Add adhoc to the README
e0neildikov: looks like we can't test this chain with gates now:(16:24
*** jascott1 has joined #openstack-infra16:24
fungie0ne: you can make a throwaway cinderclient change which depends-on the nova change and look at its install from source devstack-gate jobs16:25
fungilike clarkb suggested16:25
fungior it's possible nova has some experimental deps-from-source jobs16:25
e0nefungi: good idea. thanks for the help16:25
ildikovfungi: thanks!16:25
e0neildikov, fungi:  to be clear: we have to change dependency order. remove depends-on from nova patch and add depends-on to a cinderclient patch16:26
ildikove0ne: ok16:26
ildikove0ne: where to put the devstack dependency?16:27
*** markus_z has quit IRC16:27
e0neildikov: i have no idea. it seems that we can leave it on nova's side16:27
ildikove0ne: ok, I'll remove only yours from the list then16:28
fungie0ne: i was suggesting adding an extra cinderclient change that does something innocuous like adding a blank line in a python file and setting that to depends-on the last change in your current chain (presumably the nova change)16:28
*** lucasagomes is now known as lucas-afk16:28
*** sree has quit IRC16:28
fungithen you abandon that cinderclient change once you're satisfied with the results of the tests from source it ran16:28
e0nefungi: thanks a lot!16:28
fungiobviously you still can't (and shouldn't) merge the nova change until there's a cinderclient release with the things you need to rely on16:29
fungiat which point nova's normal jobs should work to test your new stuff16:29
ildikovhmm, so we still need the client dependency in the Nova patch?16:30
fungiif your nova patches are going to rely on it16:30
fungiildikov: devstack and devstack-gate are always installed from source, so wherever seems logical to put them in the dependency series16:30
ildikovok, I will put the cinderclient dependency back then and leave the devstack one there too16:31
ildikovsorry, I got a little lost by the end :)16:31
e0neI'll create cinderclient patch with nova dependency16:31
*** baoli has quit IRC16:32
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add bits to support stable/pike
*** baoli has joined #openstack-infra16:33
*** EricGonczer_ has joined #openstack-infra16:34
fungiclarkb: pabelanger: sorry, got sidetracked with a dozen other things... did we have a cheat sheet including openstackclient commands for creating a port in a specific neutron network and attaching it to an instance as a new interface?16:35
clarkbfungi: no but I think the command is `openstack server add fixed ip`16:36
clarkbfungi: then we'll have to update /etc/network/interfaces on the host16:36
fungidon't need to create the switchport in neutron separately?16:36
clarkbfungi: I don't think so16:36
fungicool, i think that gets me enough of a starting point to figure the rest out from context help16:36
clarkbfungi: you just give that command a server and a network and it should attach things if I read the help docs properly16:36
clarkbfungi: you can do it the other way I think as well but I don't see a need for that in this case16:37
fungisweet! Internal Server Error (HTTP 500)16:38
fungigotta love vague errors from an api16:38
clarkbfungi: I'm glad you find that entertaining :)16:38
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command
fungii'm going to guess we need to specify some precreated fixed ip here, so just working out how to get a list of them so i can identify it16:39
clarkbfungi: the command help info said network name and server from which it assigns an ip16:40
clarkbbut could be bug in osc16:40
*** sdague has joined #openstack-infra16:40
fungii did `openstack server add fixed ip $FQDN GATEWAY_NET_V6` but maybe i need to use uuids or something16:40
clarkbfungi: you need to provide the server too16:41
fungithat's $FQDN16:42
*** dprince has joined #openstack-infra16:42
fungipassing the network uuid instead of the name seems to have worked16:42
*** sree has joined #openstack-infra16:42
clarkboh I read that as a single var for some reason. Interesti it wouldn't take the name16:42
*** ccamacho has quit IRC16:42
fungidtroyer: is there any consistency expected between which subcommands take resource names and which require uuids?16:42
clarkbosc help says name or id16:42
*** ykarel has joined #openstack-infra16:42
fungiyeah, maybe if i'd retried it would have "just worked" too16:43
*** rossella_s has quit IRC16:45
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output
dtroyerfungi: everything possible that can take a name does (ie, where there is a name to use)16:47
dtroyerany that do not are bugs16:48
fungidtroyer: cool, so it's more likely that the cloud in question had trouble reversing the network name to a uuid, or simply choked on that call for unrelated reasons and the fact that it worked when i retried with the uuid was mere coincidence16:48
fungi"Internal Server Error (HTTP 500)" left a lot of room for interpretation16:49
dtroyeryeah… we need to do a better job to return the body in those cases16:49
fungialso possible we're not using super new openstackclient16:50
dtroyerif it's repeatable, —debug will give you the server's error body text in the traceback16:50
*** ykarel has quit IRC16:50
fungithanks, i'll give that a shot next time16:50
dtroyerI can't say why it would fail one time then succeed another, except that something may not have been in the right state yet on the server16:50
clarkbya could be an error on the cloud side16:51
openstackgerritAttila Darazs proposed openstack-infra/project-config master: Add "check experimental" jobs to mistral
*** LindaWang has joined #openstack-infra16:54
*** LindaWang has quit IRC16:55
*** jascott1 has quit IRC16:55
*** thorst_afk has joined #openstack-infra16:55
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add bits to support stable/pike
*** e0ne has quit IRC16:57
*** e0ne has joined #openstack-infra16:58
*** e0ne has quit IRC16:58
*** esikachev has joined #openstack-infra16:59
*** derekh has quit IRC16:59
*** ralonsoh has quit IRC17:00
clarkbmnaser: I think the two base changes will be merged shortly then you can poke at the last one (
*** dtantsur is now known as dtantsur|afk17:00
*** ramishra has quit IRC17:00
openstackgerritRico Lin proposed openstack-infra/project-config master: Avoid py35 and docs ocata gate for heat-agents
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Add bits to deploy TripleO with stable/pike
*** esikachev has quit IRC17:03
*** ym200 has quit IRC17:05
*** sree has quit IRC17:06
*** tesseract has quit IRC17:07
*** hashar has quit IRC17:08
*** harlowja has joined #openstack-infra17:13
*** EricGonczer_ has quit IRC17:13
*** efoley_ has quit IRC17:15
rm_workfungi: re: colorization, browsing via gerrit log links17:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add bits to support stable/pike
rm_workfungi: the neutron and nova logs are colorized, and ours USED to be17:16
rm_worki am sure there's some setting we have to do or something I just don't know where to start looking17:16
clarkbrm_work: your logs have to be in a known format and whitelisted by os-loganalyze. If the name of the file changed that could break it17:19
clarkbrm_work: os-loganalyze is where I would check first17:19
rm_workthanks I'll look17:20
*** prateek has joined #openstack-infra17:22
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add bits to support stable/pike
pabelangerclarkb: fungi: I think we are ready to bring infracloud-vanilla back online. DNS should be updated by now17:26
*** rkukura has joined #openstack-infra17:26
clarkbandreaf: mtreinish sdague thoughts on giving a go? There is a chance it will result in more reliable access to the mirrors. Possibly also quicker too particularly in europe. I don't expect it to be any less reliable. The big gotcha is going to be cache expiration (max time allowed in cache is 24 hours)17:26
clarkbpabelanger: I missed what happened in infracloud, was it moving the mirror?17:26
pabelangerclarkb: we lost compute015 this morning, which the mirror was running on17:27
pabelangerclarkb: ilo access is down too17:27
clarkbfun fun, I agree that that url is accessbile so should be safe to enable that region again17:27
*** prateek has quit IRC17:28
pabelangerclarkb: I'm manually editing nodepool.o.o since it is still in emergency. We have already landed the revert to project-config17:29
pabelanger#status log infracloud-vanilla back online17:29
openstackstatuspabelanger: finished logging17:29
*** sambetts is now known as sambetts|afk17:30
*** rbrndt has quit IRC17:32
*** esikachev has joined #openstack-infra17:34
*** tosky has quit IRC17:36
*** esikachev has quit IRC17:38
*** baoli has quit IRC17:39
*** baoli has joined #openstack-infra17:40
lin_yangDear team, After the patch got merged, we got new repo and new core list "rsd-core", could anyone help to add me "" into that list as the first core, then I can add others? Thanks in advance.17:42
clarkblin_yang: yes I can get that sorted out17:43
lin_yangclarkb: awesome. Thanks a lot :)17:44
clarkblin_yang: ok all done. If you find you don't have the expected +2 voting ability try a hard refresh of the gerrit page17:44
lin_yangclarkb: got it. Appreciated.17:45
*** esikachev has joined #openstack-infra17:47
*** jamesden_ has joined #openstack-infra17:47
*** jamesdenton has quit IRC17:47
*** dizquierdo has joined #openstack-infra17:47
*** _ari_ is now known as alivigni17:48
*** alivigni is now known as _ari_17:49
mtreinishclarkb: I'm fine with giving it a try17:50
clarkbmtreinish: thanks I think having someone ack it from qa (whcih you did yay) will make others more willign to give it a go17:52
*** krtaylor has quit IRC17:53
*** jamesmcarthur has quit IRC17:53
mtreinishclarkb: I'll trade you for:
clarkbmtreinish: sure17:55
clarkbinfra-root mtreinish is willing to give a try, anyone willing to be a second infra reviewer on that?17:55
*** slaweq has joined #openstack-infra17:56
*** yamahata has joined #openstack-infra17:57
clarkbcmurphy: while I'm taking on stacks of changes to review where was your puppet4 change(s)?17:58
cmurphyclarkb: here
clarkbmtreinish: ok I've approved that stack we'll need to restart the logstash workers once puppet puts everythign in place to pick up the mqtt event publishing18:03
mtreinishclarkb: ok sure, I should be around if any issues pop up18:04
clarkbcmurphy: I've approved yours as well but left a comment if you want to follow up on that (its mostly a consistency in code thing not a correctness issue)18:09
*** jpena is now known as jpena|off18:10
openstackgerritAdam Harwell proposed openstack-infra/os-loganalyze master: Add Octavia logs to filter regex
rm_workfungi: ^^ I assume it's just that, copied from the commit Manila made to add theirs18:11
rm_worki did test just to verify it matches18:11
clarkbinfra-root, if asked for 404 listings from docs.o.o I have been using these two commands on files.o.o. `grep ' 404 ' /var/log/apache2/docs.openstack.org_access.log | sed -n -e 's/.*"GET \(\/.*\) HTTP\/1\.." 404 .*/\1/p' > 404s` and `sort 404s | uniq -c | sort -rn | grep '\(html\|\/$\)' | grep $projectfilter | head -100` They can be stringed together into a single comamdn too18:12
rm_workis there a way to test it on a real gate run? can I do a depends-on and will it use that version? or is this one of the kinda hardcoded things18:12
openstackgerritMatthew Treinish proposed openstack-infra/puppet-openstack_health master: Switch api server to use system python instead of venv
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Replace references to deprecated controllerExtraConfig
cmurphythanks clarkb !18:12
mtreinishjeblair, clarkb, fungi: ^^^ I finally got around to doing that18:12
mtreinishalthough i probably screwed something up18:13
clarkbrm_work: its deployed to the runnign server and runs as a wsgi app so depends on won't help. There test framework for it should allow you to test it pretty well though18:13
clarkbrm_work: just using unittests18:13
fungipabelanger: thanks for getting the vanilla mirror back online. i'm nearly done with the volume creation on the osic mirror replacement now and have it in dns, so we're close to being ready to turn that back up too once i test it18:13
pabelangerfungi: np! and \o/18:13
cmurphycalebb: next step on puppet-4, if you're interested, is getting to install modules to the right place18:13
*** baoli has quit IRC18:13
rm_workhmm yeah i did add one bit to a unit test but removed it before i pushed up, since others didn't seem to be adding stuff specific to them18:13
fungijust got sidelined on the tarball signing zuulv3 discussion for a bit, but finishing up the mirror replacement now18:14
calebbcmurphy: o7 Ill take a look in a bit :)18:15
openstackgerritMerged openstack-infra/system-config master: Use everywhere
openstackgerritMerged openstack-infra/system-config master: Sanitize ansible results in mqtt callback plugin
openstackgerritMerged openstack-infra/system-config master: Add firehose documentation for notifications
openstackgerritMerged openstack-infra/system-config master: Add a golang firehose example
openstackgerritMerged openstack-infra/puppet-log_processor master: Add MQTT support to the gearman worker
mordredclarkb, mtreinish: d-g proxy patch +a'd18:16
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for troveclient AttributeError bug 1708190
openstackbug 1708190 in OpenStack DBaaS (Trove) "AttributeError: 'module' object has no attribute '_logger'" [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)18:16
openstackgerritMerged openstack-infra/project-config master: stop publishing docs when we tag releases
*** kfox1111_ is now known as kfox111118:19
openstackgerritMerged openstack-infra/system-config master: Enable mqtt notifications from logstash gearman worker
openstackgerritMerged openstack-infra/puppet-logstash master: Remove the mqtt options from logstash::indexer
*** rbrndt has joined #openstack-infra18:20
*** baoli has joined #openstack-infra18:20
*** bobh has joined #openstack-infra18:20
openstackgerritMerged openstack-infra/system-config master: Parameterize puppet version in install script
*** rbrndt has quit IRC18:24
*** Rockyg has quit IRC18:25
*** links has quit IRC18:31
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create tox-publisher trusted job
*** esikachev has quit IRC18:32
pabelangerjeblair: mordred: ^First attempt at tox-publisher trusted playbook18:32
fungiinfra-root: is back online, and my lonely held node there in osic is able to install python packages again where it was hanging before. i'm going to abandon 489776 and take nodepool.o.o out of emergency disable again unless there are any objections18:32
*** adisky__ has quit IRC18:32
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create tox-publisher trusted job
*** esikachev has joined #openstack-infra18:33
pabelangerfungi: ++18:34
jeblairpabelanger: cool; should be able to take a look in ~1hr18:34
pabelangerjeblair: np, off for some eats myself18:35
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches
fungidtantsur|afk: i'm going to start digging into your duplicate gerrit accounts situation now... apologies for the delay18:36
*** slaweq has quit IRC18:36
*** slaweq has joined #openstack-infra18:37
*** jaypipesjuryduty is now known as jaypipes18:37
*** esikachev has quit IRC18:39
mnaser added a screenshot to make the review easier and finish moving the logos :)18:39
*** kjackal_ has quit IRC18:40
*** slaweq has quit IRC18:41
*** kjackal_ has joined #openstack-infra18:42
*** ihrachys has quit IRC18:42
*** ihrachys has joined #openstack-infra18:42
dmsimardfungi: so I've poked around sdist and I can't make it and PBR bend to my will so I will have to work with that extra .0 I guess.18:43
clarkbmnaser: tyty18:43
dmsimardfungi: Looking at other projects, say, nova which is also in the format "15.0.0" also carries an extra ".0" for pre-releases (i.e,
dmsimardMy brain still can't make sense out of it but I'll have to comply :)18:44
*** krtaylor has joined #openstack-infra18:44
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches
mnaserclarkb are you going to propose a small patch for the big logo or you want me to do that?18:48
*** jascott1 has joined #openstack-infra18:49
*** rbrndt has joined #openstack-infra18:51
fungidmsimard: to reiterate, semver covers the first three dot-separated components of the version number, so the 16.0.0 part, and has no provisions for pre-release version components so the b1 can't be in one of those first three components. setuptools interpretation of pep440 normalization adds a leading 0 to any version component which starts with a letter, hence the 0 in the 0b1 component18:53
fungii agree it's not terribly aesthetically pleasing, but rather the result of mashing two standards (semver and pep440) together18:54
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo/oooq/cleanup: remove unused jobs in gate
mtreinishpabelanger: the asterisk server doesn't record the audio from the call?18:56
*** annegentle has quit IRC18:56
AJaegerEmilienM: could you quickly check , please?18:57
*** annegentle has joined #openstack-infra18:58
EmilienMAJaeger: good spot. I commented18:58
clarkbmnaser: I'm not quite sure where to start with that. Do we go back to png so the pixels are fixed?18:59
clarkbmnaser: feel free to push something up18:59
AJaegerclarkb: the multi-node/single-node changes in change look wrong to me - do we really want one job for both? Could you review that, please?18:59
mnaserclarkb np, svg just needs a fixed size somewhere in css, ill do it18:59
*** jcoufal has quit IRC19:00
*** apuimedo has quit IRC19:00
*** jamesmcarthur has joined #openstack-infra19:00
*** coolsvap has quit IRC19:01
openstackgerritMerged openstack-infra/project-config master: Transit Heat convergence job to use quickstart
mnaserclarkb i need to edit this file.. but i cant find it in system-config, any ideas?
pabelangermtreinish: it can, but not enabled.  I've thought about enabling it from time to time19:05
clarkbinfra-root sdage what do you think about having mod_rewrite rewrite to a simple listing of the top 1k 404s by count against the last day of docs.o.o apache logs? Then we can have cron update that file daily?19:05
pabelangermtreinish: should be hard to record and publish some place19:05
clarkbinfra-root sdague, that way this doesn't need afs write permissiosn and remains super simple19:05
clarkbmnaser: I think that just comes from the normal gitweb install. We can likely override it somehow let me check some docs19:06
mtreinishpabelanger: yeah I was just a bit surprised considering how we like things to be archived that it wasn't turned on19:06
*** skelso has joined #openstack-infra19:06
mtreinishits not a big deal either way, it's not like I have any interest in listening to it19:06
mnaserclarkb thats the "site" css file, so it contains all the openstack theme related stuff19:06
mtreinishI was just curious19:06
mnaserso it was configured somewhere.. but i cant find where19:06
clarkbmnaser: ah ok let me find it then19:06
mnasernot in system-config anyways19:06
openstackgerritMerged openstack-infra/project-config master: func_py3 experimental check for openstack-ansible-plugins
clarkbmnaser: openstack-infra/system-config/modules/openstack_project/files/gerrit/GerritSite.css ?19:07
pabelangermtreinish: I can write up some patches to support it. Good friday hacking project19:08
clarkbmnaser: looks to be sahred with gerrit too so you'll want to check it against gerrit's rendering of the svg too (though you likely know far more about all that than I do)19:08
pabelangermtreinish: or russellb can do it to bump his stackalytics count19:08
mnaserclarkb i *suspect* someone manually cp'd the GerritSite.css content to the gitweb-site.css file19:09
clarkbmnaser: I think gerrit may do it19:09
clarkbsince gitweb runs as part of gerrit I think?19:09
AJaegerEmilienM: could you review this as well, please?
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo/quickstart: use zuul template for scenarios
EmilienMAJaeger: sure thing19:09
*** rbrndt has quit IRC19:09
clarkbaptitude says gitweb is installed though so maybe not19:10
mnaser vs
*** nicolasbock__ has quit IRC19:10
mnaserfungi mentioned today that the style was automatically loaded in without restarting gerrit19:10
mnasermaybe the restart is what does the copy?19:10
EmilienMAJaeger: ouch, -119:11
mnaseri mean, the css will still need changes but i just want to make sure that editing the GerritSite.css is the right way (and there isn't some other file i have to edit)19:11
clarkbmnaser: ya that is a possibility, looking at the config management for it we just ensure the gitweb package is installed but don't configure it19:11
mnaserthats what i saw in puppet-gerrit too19:11
clarkbI think gerrit must be doing that, but am digging more19:11
AJaegerlbragstad, stevemar , could you review this keystone related project-config change, please?
AJaegerEmilienM: good that I waited ;)19:12
lbragstadAJaeger: sure19:12
EmilienMAJaeger: please ping me everytime CI changes are done for tripleo :D19:12
*** jamesmcarthur has quit IRC19:12
*** EricGonczer_ has joined #openstack-infra19:13
AJaegerEmilienM: I'll try to not +2 before the infra liaisons +1 - and you're listed...19:13
AJaegerthose were older, so I decided to ping...19:13
EmilienMAJaeger: lgtm19:13
fungiclarkb: or we could add a 404s index (in a fixed path or under a new subdomain even) separate from a less noisy 404 page19:14
AJaegerEmilienM: is duplicate of
clarkbmnaser: ya /usr/share/gitweb/static has stuff in it19:15
clarkbmnaser: but I don't see where we generate that19:15
mnaseris it a symlink or is it a static file with the contents of that?19:15
EmilienMbnemec: please abandon
*** rbrndt has joined #openstack-infra19:15
mnasermaybe someone just cp'd it waaaay back and its not managed by puppet19:15
sdagueclarkb: that would be fine for me19:16
sdagueclarkb: I was assuming a cron publish to text file would be just as safe as well19:16
mordredclarkb: works for me19:16
AJaegerEmilienM: so, I would rather have your team ping you instead of me doing it - but will do for odd and old things occasionally19:17
clarkbsdague: mordred fungi oh ya I meant rewrite to a file on disk that isn't in afs19:17
clarkbsdague: mordred fungi that file would update once a day via cron. I just don't want to rely on afs for server specific content as it would get tricky if we added more than one server19:17
clarkbmnaser: maybe? we have an /etc/gitweb.conf too but I don't see what process runs the cgi19:18
mordredclarkb: yup.19:18
mordredmnaser, clarkb: I believe gerrit handles all gitweb related config itself and we don't directly configure it ourselves19:18
fungiclarkb: seems fine as long as we cleanly postprocess out everything except the url requested from the log, and make sure we do it in such a way as to not risk any sort of shellcode injection in whatever script is doing that19:18
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Zun] Disable Swift for dsvm py35 job
EmilienMAJaeger: ok19:19
EmilienMAJaeger: works for me19:19
*** jamesmcarthur has joined #openstack-infra19:19
*** jtomasek has quit IRC19:19
EmilienMAJaeger: is ready for review19:19
*** panda|bbl is now known as panda19:19
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Fuxi] Disable Swift for py35 fullstack job
clarkbmordred: mnaser ya the more I look the more I think gerrit must be injecting its own stuff in there. So yes best guess is restarting gerrit will cause it to update gitweb's configs19:20
mnaserclarkb do you want me to submit a change with the fixes to GerritSite.css and then we can start and hopefully it overrides19:21
mnaserand if it doesnt override it we can figure it out then19:21
clarkbmnaser: ya I think that may be simplest19:21
fungiyay! osic nodes building leave quite a spike in the test nodes graph on the status page ;)19:21
clarkbfungi: well it will run a fixed script from cron so there is no input to sanitize to avoid shellcodei njection19:21
clarkbfungi: and it would run my commands above basically just without a project filter. I'll work on pushuing up a patch with more concrete infos19:22
fungiclarkb: yeah, that command looks safe enough. no actual strings from the file ending up in parameters to anything invoked from the shell19:23
fungiit's all via pipes, so we should be good19:23
*** dizquierdo has quit IRC19:25
AJaegerteam, here's a change to adjust our nodepool ssh setup for project-config that nobody likes to +2: could some expert please review and tell us whether this is wanted or not?19:26
AJaegerpabelanger: do we still need ?19:26
pabelangerAJaeger: no, I think ianw fixed it19:27
pabelangerwill abandon19:27
*** e0ne has joined #openstack-infra19:27
openstackgerritMatthew Treinish proposed openstack-infra/puppet-openstack_health master: Switch api server to use system python instead of venv
AJaegerfungi, clarkb: ages ago you commented on - I think we don't want it, correct?19:30
*** slaweq has joined #openstack-infra19:31
fungidtantsur|afk: it looks like your extra 12773 account from 2015-12-16 wasn't used by anything, so i have deactivated it, leaving only your older 10239 account. please let me know if you experience any issues with gerrit, but i expect you won't notice any difference19:32
AJaegerOops, we maxed out our capacity ;/  I have jobs waiting for an hour for a node to be ready...19:32
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Kuryr] Setup a non-voting py35 job for libnetwork
AJaegerand logstash queue is at 20k events since a good hour - from 0. That looks strange...19:33
openstackgerritMohammed Naser proposed openstack-infra/system-config master: Fix OpenStack header logo in Gitweb
mnaserclarkb ^ there with screenshots19:33
clarkbAJaeger: it often does that if some jobs inject massive log files19:33
pabelangerlarge amount of nodes building in nodepool too19:34
pabelangermust be osic coming back online19:34
fungiyeah, i commented on it in channel earlier19:34
*** e0ne has quit IRC19:34
fungitime to ready there spiked to 15 minutes and we got quite a few error nodes, so likely slammed it hard19:35
AJaegerok, let's give it time to settle ;)19:35
*** skelso has quit IRC19:37
*** EricGonczer_ has quit IRC19:39
*** wolverineav has quit IRC19:39
*** skelso has joined #openstack-infra19:39
*** wolverineav has joined #openstack-infra19:40
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add zuul.executor.work_root to inventory
*** EricGonczer_ has joined #openstack-infra19:40
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
*** EricGonczer_ has quit IRC19:42
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove gate-tempest-dsvm-full from grafana
*** EricGonczer_ has joined #openstack-infra19:42
openstackgerritClark Boylan proposed openstack-infra/system-config master: Generate list of top 100 404s for docs.o.o
clarkbfungi: mordred sdague ^ something like that (note script is untested)19:44
*** wolverineav has quit IRC19:44
clarkbfungi: unless grep, sort, uniq, or sed have a vulnerability and can be influenced on their inputs I think we are ok. I don't think there are shell injections concerns19:44
clarkbAJaeger: I'm reviewing the mulitnode change now then mnasers css fix19:45
AJaegerthanks, clarkb19:46
clarkbAJaeger: what is the concern on the multinode change?19:46
sdagueclarkb: would it be possible to timestamp the files `date +%Y-%m-%d`-docs-404s.txt ?19:46
AJaegerclarkb: normally we have different jobs for multinode and single node - I wonder whether it's ok to use this template this way for both19:46
sdagueI think at least to get started, it would be valuable to keep the old files around and see how they change as we fix things19:47
clarkbsdague: not if we want the apache alias to work and i don't want ot implement rotations. I can add a timestamp prefix to them though19:47
fungiclarkb: i'm trying to work out your requests session params encoding suggestion now. doing a separate change to refactor the duplicate requests logic out of the gerrit query function first and will generalize it enough so it can be used for the new member directory lookups in the later change19:47
clarkbsdague: we could keep the old files around but then I'd have to use proper rewrite19:47
sdagueclarkb: or just dirindex a directory19:47
clarkbsdague: and alias to the dir?19:48
clarkbsdague: the problem I'm trying to work around is that the docroot is in afs and intended to be shared by many apaches potentionally. So I can't put a single apache's log data in there19:49
clarkbwant to keep that local19:49
clarkbI can use the hacky elasticsearch method of file rotation and use find to delete anyhting more than a month old19:49
sdagueclarkb: these are small files, do we really need to worry about delete at this point?19:50
sdaguelike, the entire runtime of this for the history of the project is smaller than the output of a pep8 job19:50
clarkbsdague: yes, beacuse every time we tell ourselves that we inevitably run into problems later19:50
jeblairclarkb: let's stick it in afs.  it's not that hard.19:50
sdagueclarkb: ok, the order of magnitude is so small here though19:50
clarkbjeblair: the problem isn't sticking it in afs. The problem is the data isn't shared data intended to be served by many servers19:50
jeblairclarkb: servername as path component19:51
sdagueclarkb: also, is 100 lines sufficient if it's more than 1 project? I thought the nova 404 dump was 40+19:51
sdagueat least to start we know there are going to be a lot of 404s because of the move19:52
clarkbsdague: thats 100 from right now19:52
clarkbjeblair: or we can just serve off local server and avoid afs because it is unnecessary19:53
clarkbjeblair: this also allows our servers to continue to be anonymous19:53
sdagueclarkb: yeh, so the problem with that is that if a project, like nova, gets on top of it and fixes those top 5, we can't see the rest of our 404s to fix19:53
clarkbthere are ~3k 404'd paths in total19:54
clarkbso we likely could do the whole set19:54
clarkb3107 from today19:54
jeblairclarkb: i don't have a rebuttal to that other than we have a pretty good content publishing system and it seems unecessary to invent another one.  literally no apache configuration is required if you publish it into afs.19:54
*** annegentle has quit IRC19:55
clarkbjeblair: but it does require server configuration changes that change the servers from anonymous read only to needing authenticated write perms19:55
jeblairclarkb: (as for anonymity, the rewrite system requires you know the actual hostnames of every web serving host, so that seems the same either way)19:55
*** sileht has quit IRC19:55
*** annegentle has joined #openstack-infra19:56
jeblairclarkb: yeah, we can make a keytab for the cronjob (we have like 20 already, so that's pretty well established)19:56
jeblairclarkb: (we could also ip acl the server, but keytabs are better)19:56
jeblairbut we don't need to change how apache runs or anything19:57
jeblairi'm happy to pitch in on the afs stuff if you want19:57
kmARChi channel, the upstream-institute project decided to bring the virtual environment source code over to from GitHub and set up Gerrit reviewing workflow, etc. Currently, it's hosted here:
kmARCSince it's not really related to the training slide deck sources, we would create a new repository for it, `upstream-institute-virtual-environment`. Can you guys help me with that?19:57
clarkbjeblair: and is that preferable to maintaining ro access? I quite like read only access as being a thing that is possible for functioning servers19:58
clarkbmakes them far simpler to deploy and reduces the security concern of having lots of things with lots of access19:58
pabelangerkmARC: sure, we have the process documented at
pabelangerkmARC: be sure to note the section about 'upstream' field19:58
jeblairclarkb: the keytab would grant write access to only one subdirectory, and would only be used by the cronjob.19:59
jeblairclarkb: a server deployed without the keytab for whatever reason would still function as a web head, but its cronjob to build 404 lists would fail19:59
kmARC@pabelanger thanks you, will read it now /sorry for the disruption/ ;-)19:59
clarkbjeblair: which we weigh against 4 linse of apache config change?20:00
fungi#status log OSIC environment is active in Nodepool and running jobs normally once more20:00
openstackstatusfungi: finished logging20:00
pabelangerkmARC: np, were here to help20:00
clarkbI dunno to me updating apache is far simpler here20:00
pabelangerfungi: thanks!20:00
pabelangerfungi: will be a sad day when we lose it :(20:00
*** e0ne has joined #openstack-infra20:01
*** lrossetti_ has quit IRC20:01
fungipabelanger: we'll get by, but it's been nice while we have it!20:01
*** lrossetti has joined #openstack-infra20:01
clarkbkmARC: pabelanger also you'll want to "curate" the repo branches and tags before hand as we import them all as is20:02
*** skelso has quit IRC20:03
*** skelso has joined #openstack-infra20:03
sdagueclarkb: are those really all today? because some of those 404s we already have rules for20:03
clarkbsdague: its the last ~day. I'm actually looking at last two log files20:03
clarkband they get rotated daily20:04
sdagueso fixes we landed yesterday will not be reflected here yet20:04
sdaguethat's fine20:04
jeblairclarkb: 4 lines of apache doesn't include creating and managing the directories, etc.  honestly either solution is about the same complexity.  4 lines of apache vs 4 lines of puppet.  i'm just suggesting that using afs the way we're already using it seems very straightforward.20:04
clarkbsdague: correct20:04
*** skelso has quit IRC20:08
*** skelso has joined #openstack-infra20:08
jeblairclarkb: to be clear, i'm happy to have either.20:08
*** krtaylor has quit IRC20:08
clarkbjeblair: I'm just concerned that we are taking a compeltely local resource and by hosting it remotely can only make it more complicated20:08
jeblairclarkb: yep.  i'm just trying to convince you that having our afs-based webservers serve data from afs isn't complicated.  if i've failed, it's okay, i'll move on.20:10
jeblairit's not that important and it's easy to change20:10
jeblairit's more important for sdague to get the list somehow.  :)20:11
sdagueyes, I do not care about implementation at all. Anything that gets me a list is more information than I have now. :)20:11
sdagueI would prefer to have a historical record, so we can track progress, as that's been important in getting people to contribute20:12
AJaegerclarkb: thanks for the list, I addressed the top 4 of the 404s - and a few more further down...20:13
AJaegerclarkb: for progress, it would be nice to give the total list of pages in the list as summary.20:14
*** e0ne has quit IRC20:15
*** slaweq has quit IRC20:18
*** slaweq has joined #openstack-infra20:19
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Kuryr] Setup a non-voting py35 job for libnetwork
openstackgerritClark Boylan proposed openstack-infra/system-config master: Generate list of 404s for docs.o.o
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
clarkbsdague: jeblair ^ that addresses most of the comments I've seen on irc. And should be largely applicable whether we host locally or in afs20:21
*** Julien-zte has quit IRC20:21
clarkbgives you a month worth of data, doesn't trim it to top 100, and serves it out of a dir with file per day20:21
jeblairclarkb: is the require ssl/private intentional?20:22
clarkbno thats like copy pasta fail20:23
*** krtaylor has joined #openstack-infra20:23
*** Julien-zte has joined #openstack-infra20:23
clarkbya copy pasta fail20:23
*** slaweq has quit IRC20:23
*** Douhet has quit IRC20:23
jeblairclarkb: otherwise lgtm20:24
*** slaweq has joined #openstack-infra20:25
openstackgerritClark Boylan proposed openstack-infra/system-config master: Generate list of 404s for docs.o.o
clarkbthat removes the unnecessary require. Now to properly test that script20:25
*** https_GK1wmSU has joined #openstack-infra20:26
*** bobh has quit IRC20:27
*** e0ne has joined #openstack-infra20:27
*** https_GK1wmSU has left #openstack-infra20:27
*** Douhet has joined #openstack-infra20:29
openstackgerritMerged openstack-infra/zuul-jobs master: Add README for tox role
openstackgerritMerged openstack-infra/zuul-jobs master: Update tox-tarball to use tox role
clarkboh there is slight bug20:33
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use undefined values instead of 40 zeroes
openstackgerritClark Boylan proposed openstack-infra/system-config master: Generate list of 404s for docs.o.o
openstackgerritMatthew Treinish proposed openstack-infra/puppet-openstack_health master: Switch api server to use system python instead of venv
clarkbI've run that script against test input and output looks good now. But there was bug in last patchset which this one fixes20:37
*** Douhet has quit IRC20:39
*** Douhet has joined #openstack-infra20:40
*** hashar has joined #openstack-infra20:40
*** lrossetti has quit IRC20:42
*** jascott1 has quit IRC20:42
*** efoley has joined #openstack-infra20:42
*** lrossetti has joined #openstack-infra20:42
rm_workfungi / clarkb: so would require testing to merge it? It seems like the testing is only really in place for core projects20:44
clarkbsdague: ok responded to the list with details and an example output of running that script against my test inputs20:44
sdagueclarkb: thank you20:45
*** gyee has quit IRC20:48
*** krtaylor has quit IRC20:50
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap
*** e0ne has quit IRC20:52
*** jrist has quit IRC20:54
*** jamesmcarthur has quit IRC20:54
openstackgerritMark Korondi proposed openstack-infra/project-config master: Bringing upstream training virtual environment over here
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create publish-tox-openstack-tarballs trusted job
*** Goneri has quit IRC20:57
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels
*** vhosakot has quit IRC20:58
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets
*** vhosakot has joined #openstack-infra21:00
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test
pabelangerEmilienM: it is likely the variable isn't on disk just yet21:05
fungiclarkb: i have some annoying news... apparently gerrit's rest api is befuddled by %-encoded query strings... before i dig too deeply into requests documentation, do you happen to know of simple ways to figure out which encoded things the remote system chokes on and disable encoding just those?21:05
EmilienMpabelanger: I know but I'm ready :)21:06
*** thorst_afk has quit IRC21:06
EmilienMpabelanger: doing the same for tripleo now21:06
*** jamesden_ has quit IRC21:06
fungiclarkb: aha, after much trial and error, it's that gerrit dislikes + being rewritten to %2B21:07
pabelangerEmilienM: I can kick off a image build for centos-7 now to pick it up21:08
*** esberglu has quit IRC21:08
*** ijw has joined #openstack-infra21:08
pabelangerEmilienM: or wait until tonight21:08
*** yamamoto_ has joined #openstack-infra21:08
jeblairfungi: is what gertty uses in everything it passes to gerrit as a url21:09
*** eharney has quit IRC21:10
fungijeblair: i'll give that a shot. other solutions i'm seeing suggested involve subclassing to override requests.Session.send() with a custom urllib.quote list21:10
jeblairthat could have been more clear:  that's what gertty uses when constructing all of the gerrit rest api urls it uses21:10
*** thorst_afk has joined #openstack-infra21:11
*** slaweq has quit IRC21:13
*** annegentle has quit IRC21:13
EmilienMpabelanger: I don't mind waiting21:14
*** annegentle has joined #openstack-infra21:14
*** thorst_afk has quit IRC21:15
*** yamamoto_ has quit IRC21:15
*** hashar has quit IRC21:15
*** ldnunes has quit IRC21:15
clarkbfungi: sorry was getting csa box, I'm not sure of a way to easily determine that but sounds like you got it sorted21:16
jesusaurthe gerrit change-ids in commit messages become links in a git hook, right? where does that live in config?21:16
*** yamamoto_ has joined #openstack-infra21:17
clarkbjesusaur: they become links via the comment links config can be found at system-config/modules/openstack_projects/manifests/review.pp iirc21:17
fungijeblair: clarkb: actually, i just found requests.utils.requote_uri(), but turns out that requests seems to use urllib.parse.quote_plus() by default so just having " " in place of "+" in my parameters seems to do the trick!21:17
openstackgerritMark Korondi proposed openstack-infra/project-config master: Bringing upstream training virtual environment over here
clarkbfungi: ah its the whitespace that needs to become + instead of + becoming escaped21:18
fungijeblair: clarkb: i added some debugging to splat out the url object from the request and indeed " " was transformed to "+" right before my very eyes21:18
openstackgerritMark Korondi proposed openstack-infra/project-config master: Bringing upstream training virtual environment over here
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Make timer tests less racy
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync
jesusaurclarkb: ah, thanks21:20
jeblairfungi: magic!21:21
*** jascott1 has joined #openstack-infra21:21
*** priteau has quit IRC21:21
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Refactor requests usage in owners script
fungiclarkb: jeblair: i'll wip that ^ until i get a comparison run before and after to confirm they produce the same data21:22
fungiand then i'll rebuild the rest of the other series on top of that reusing the new functions21:23
*** dprince has quit IRC21:23
fungiclarkb: hopefully that will address your parameter encoding concerns with the member directory api calls in 483974?21:23
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add /etc/lsb-release to zuul bwrap environment
*** jrist has joined #openstack-infra21:26
openstackgerritMerged openstack-infra/system-config master: Use trusted_ro_paths
*** baoli has quit IRC21:28
*** Douhet has quit IRC21:33
*** Douhet has joined #openstack-infra21:33
openstackgerritMark Korondi proposed openstack-infra/project-config master: Bringing upstream training virtual environment over here
*** jamesmcarthur has joined #openstack-infra21:35
*** esberglu has joined #openstack-infra21:37
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow use of the uri module from localhost
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules
*** jamesmca_ has joined #openstack-infra21:39
*** jamesmcarthur has quit IRC21:39
*** slaweq has joined #openstack-infra21:43
*** thorst_afk has joined #openstack-infra21:46
*** vhosakot has quit IRC21:47
openstackgerritMark Korondi proposed openstack-infra/project-config master: Bringing upstream training virtual environment over here
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add zuul.executor.work_root to inventory
*** thorst_afk has quit IRC21:50
*** slaweq has quit IRC21:52
*** slaweq has joined #openstack-infra21:52
*** kgiusti has left #openstack-infra21:53
*** slaweq has quit IRC21:56
*** slaweq has joined #openstack-infra21:56
*** ijw has quit IRC21:57
*** jamesmca_ has quit IRC21:57
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create publish-openstack-python-tarball trusted job
*** jamesmcarthur has joined #openstack-infra21:58
clarkbfungi: yup +2 lgtm21:59
*** efoley has quit IRC21:59
*** jamesmcarthur has quit IRC21:59
*** jamesmcarthur has joined #openstack-infra21:59
*** xyang1 has quit IRC21:59
*** annegentle has quit IRC22:00
*** jkilpatr has quit IRC22:01
*** zer0c00l_ has joined #openstack-infra22:01
zer0c00l_Any one running into issues with devstack stable/ocata branch on RHEL 7?22:01
*** armax has quit IRC22:01
zer0c00l_i am having issues with libvirt-python package22:01
openstackLaunchpad bug 1636567 in devstack "devstack mitaka installation fails with error "Running bdist_wheel for libvirt-python: finished with status 'error'" in Ubuntu 16.10" [Undecided,Opinion]22:01
zer0c00l_Not sure if this is an appropriate channel for devstack questions22:02
pabelangerzer0c00l_: we only have access to centos-7 upstream22:02
*** panda has quit IRC22:02
pabelangerzer0c00l_: ianw does a good job keeping devstack running on centos-722:02
clarkbzer0c00l_: pabelanger lookng at it though looks like new rhel release has libvirt 3.2 but we use libvirt-python 2.522:03
*** marst has quit IRC22:03
clarkbyou may need to update to libvirt-python 3.something?22:03
zer0c00l_clarkb: yes22:03
clarkbzer0c00l_: that is what I would try. Update constraints to use whatever is latest libvirt-python and see if that fixes22:03
zer0c00l_clarkb: so, how do i change the upper constraints in the middle of devstack installation?22:04
zer0c00l_stable/ocata seems to be tied to libvirt-python 2.5.022:04
zer0c00l_master has libvirt-python 3.5.x i think22:04
*** skelso has quit IRC22:05
clarkbzer0c00l_: one way is to make the commit in the requirements repo then set the error on clone flag in devstack to prevent it from updatign the repo22:05
clarkbI think you might also be able to set the ref you want22:05
zer0c00l_Ah i see22:05
zer0c00l_You are suggesting i summit a review to stable/ocata branch and see the REPO BRANCH  to use the review?22:06
zer0c00l_Am i understanding it right?22:06
clarkbyou don't have to submit a review to test it locally but that is one way to do it22:06
clarkbyou can just edit it locally22:07
zer0c00l_I see, i am using chef to do my devstack deployments.22:07
zer0c00l_There is no manualy editing there :(22:07
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some lookup plugin tests
clarkbfungi: before I forget and before you disappear for the evening. Is september 18 still a go for upgrading gerrit?22:09
clarkbI haven't heard any one say its a bad day, but you had the known conflict22:09
*** ijw has joined #openstack-infra22:09
clarkb(want to make sure I get the email out soon)22:09
*** https_GK1wmSU has joined #openstack-infra22:10
*** felipemonteiro__ has quit IRC22:10
*** https_GK1wmSU has left #openstack-infra22:11
fungiclarkb: yeah, christine has given me a list of ways to make it up to her ;)22:12
fungiannounce at will22:12
*** slaweq has quit IRC22:13
*** slaweq has joined #openstack-infra22:13
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in project definition
clarkbmtreinish: mqtt things are bropken in logstash woprker22:15
clarkbmtreinish: File "/usr/local/bin/", line 366 IndentationError: unexpected indent22:15
*** gouthamr has quit IRC22:16
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets
openstackgerritMerged openstack-infra/project-config master: Create publish-openstack-python-tarball trusted job
*** slaweq has quit IRC22:17
clarkbmtreinish: and then AttributeError: 'Server' object has no attribute 'configget'22:18
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor master: Fix syntax errors
clarkbmtreinish: fungi ^ review on that much appreciated22:19
clarkbwe are mostly not running logstash workers right now because of that22:19
*** thingee has quit IRC22:19
fungiclarkb: when did that break? wondering what bug we landed22:22
mtreinishclarkb: sigh, I was really off by 1 space on that indent22:23
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in secrets
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor master: Fix syntax errors
mtreinishfungi: clarkb approved the rest of the more-firehose patches earlier today, which included the support for the mqtt worker stuff22:24
clarkbya so it broke earlier today when puppet applied it.22:24
clarkbnew patchset fixes more things22:24
clarkbmtreinish: also tls is unworking22:24
clarkbmtreinish: ValueError: Python 2.7.9 and 3.2 are the minimum supported versions for TLS.22:24
fungicool, so at least we weren't broken terribly long :/22:24
*** ijw has quit IRC22:24
mtreinishclarkb: oh that's the latest version of paho thing22:25
mtreinishwe hit that on another trusty server22:25
clarkbmtreinish: I think it may be breaking publishing to elasticsaerch too because it bails out the entire run loop22:25
mtreinishwe can downgrade to <1.3.0 iirc22:25
clarkbbut maybe not because there is a time delta between pushing logs to logsatsh and getting that message22:26
clarkbmay just force us to redo run loop for every file22:26
fungiyeah, on that note review of 485275 would be appreciated too22:26
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Use undefined values instead of 40 zeroes
*** gordc has quit IRC22:28
clarkbmtreinish: can you push a port of for the logstash workers? or should I go ahead and do that?22:28
*** yamamoto_ has quit IRC22:29
*** EricGonczer_ has quit IRC22:29
mtreinishclarkb: I can, unless you want to just add it onto 49023322:30
clarkbmtreinish: nah 490233 is already approved and neatly fixes the python bugs22:30
mtreinishit gets installed in init.pp of puppet-log_processor22:30
clarkbmtreinish: so second change would be good22:30
mtreinishah ok, I'll push that up right now, one sec22:30
*** thingee has joined #openstack-infra22:32
*** yamamoto has joined #openstack-infra22:32
*** jamesmcarthur has quit IRC22:33
openstackgerritMatthew Treinish proposed openstack-infra/puppet-log_processor master: Ensure we use paho 1.2.3
mtreinishclarkb: ^^^22:34
fungiclarkb: catching up on e-mail, should we exclude the 404s page in robots.txt? might cause some mighty strange search results on things appearing in those url strings22:35
fungihappy to propose the patch for that22:35
clarkbfungi: oh probably a good idea, you could tack it onto my change or a followup22:36
*** yamamoto has quit IRC22:36
openstackgerritMerged openstack-infra/system-config master: Pin paho-mqtt to 1.2.3 on the puppetmaster
*** jkilpatr has joined #openstack-infra22:38
*** Sukhdev has quit IRC22:40
*** yamamoto has joined #openstack-infra22:41
clarkbinfra-root how does that look for gerrit upgrade notice22:42
mordredclarkb: reading22:42
*** yamamoto has quit IRC22:44
openstackgerritMerged openstack-infra/puppet-log_processor master: Fix syntax errors
fungiclarkb: i just reread the change and realized you're serving that as files.o.o/404s not docs.o.o/404s, so no need to adjust robots.txt assuming that's the url you meant for it to have?22:46
clarkbfungi: oh I meant docs.o.o. Did I not have the right vhost?22:46
fungiseems like you did not22:47
fungiyou put it in modules/openstack_project/templates/files.vhost.erb rather than modules/openstack_project/templates/docs.vhost.erb22:47
clarkblooking at it though files.o.o is where we put the unrendered stuff so maybe that is a better home for this data?22:48
clarkbthen I can rename it to docs-404s22:48
fungii was initially thrown because of your pointing 404s to docs-404s and assumed it was on the docs site until i tried to figure out why the robots.txt entry was entirely missing from the docs vhost definition22:48
fungiso yes, just keep the url the same as the filename22:49
fungiand then we're all good22:49
clarkbya let me push that patch up22:49
clarkbI do think files.o.o is a better home for this looking at what it is serving22:49
mordredclarkb: etherpad looks great22:49
clarkband then I don't even need my alias22:49
fungionly downside is that'll shadow any top-level docs-404s file/directory from our afs tree, but not really that critical22:49
mordredclarkb: do you think it's worth mentioning that we expect this to be the last time we have to do an offline secondary index update?22:49
clarkboh is that all afs? then ya still need alias22:50
clarkbmordred: I think if we get better at upgrading release to release yes22:50
*** jkilpatr has quit IRC22:50
clarkbsince they seem to be trying to make it so that n-1 to n can do an online reindex22:50
*** jkilpatr has joined #openstack-infra22:50
*** sdague has quit IRC22:50
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in secrets
openstackgerritClark Boylan proposed openstack-infra/system-config master: Generate list of 404s for docs.o.o
clarkbfungi: ^ how is that?22:52
openstackgerritMerged openstack-infra/puppet-log_processor master: Ensure we use paho 1.2.3
clarkbmordred: I'm hoping that with 2.13 behind us going to 2.14 while complicated will be easier simply because we've got a recent one under our belts22:54
clarkbmordred: then as long as we upgrade release to release we should be good.22:54
fungiclarkb: that seems saner, though i haven't finished reading through the script itself yet22:55
*** pbourke has quit IRC22:55
*** jrist has quit IRC22:55
fungiclarkb: also, maintenance announcement lgtm--thanks!22:55
clarkbok sending out maintenance announcement now22:55
*** yamamoto has joined #openstack-infra22:56
*** pbourke has joined #openstack-infra22:57
mordredclarkb: ++22:57
mordredclarkb: I'd love to get on a more frequent cadence with that - especially if we can do it with minimal downtime and whatnot22:58
fungipart of the delay for this one is that gerrit released a new version while we were evaluating the old one and then we mostly started over again22:58
clarkbgerrit upgrades have honestly also gotten a lot more scary22:59
fungiand when they released a new version the second time, we decided to stick with what we'd already tested rather than keep spinning our wheels forever22:59
clarkbthe track record of not having world breaking bugs is not great recently22:59
clarkb(I think 2.13 is in good shape though as its at the end of its life and has had 9 point releases and wmf runs it)22:59
clarkbbut ya22:59
fungii loved the forum reply where they asked you why we were upgrading to 2.13 and not 2.1423:00
*** jaypipes has quit IRC23:00
*** xarses has quit IRC23:00
clarkbI do think that step 0 of 2.14 is replacing the database23:01
clarkbwe should get on mysql 5.6 or 5.7, have 4 byte utf8.23:02
fungii wouldn't mind seeing us do that soon after the 2.13 upgrade even23:02
fungijust to have it out of the way23:02
fungithat would be a relatively fast maintenance23:02
*** rbrndt has quit IRC23:03
clarkbas a heads up I have removed logstash-worker01 from the ansible emergency file. I think it was put in there the first time we did mqtt things23:03
*** gtrxcb has joined #openstack-infra23:06
fungioh, hah. fim23:07
*** jascott1 has quit IRC23:07
*** jascott1 has joined #openstack-infra23:08
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules
clarkbas soon as paho version updates I am going to start all the daemons23:09
clarkb*logstash daemons23:09
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow use of the uri module from localhost
*** jascott1 has quit IRC23:10
*** yamamoto has quit IRC23:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add zuul:var directive and role
clarkbthe puppet run starting now should update paho-mqtt for us23:14
*** yamamoto has joined #openstack-infra23:14
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref
*** hongbin has quit IRC23:21
*** dingyichen has joined #openstack-infra23:22
*** aeng has joined #openstack-infra23:23
*** http_GK1wmSU has joined #openstack-infra23:23
*** http_GK1wmSU has left #openstack-infra23:24
*** gildub has joined #openstack-infra23:28
*** gouthamr has joined #openstack-infra23:32
*** aviau has quit IRC23:36
*** aviau has joined #openstack-infra23:36
clarkbmtreinish: fungi fyi23:36
clarkbI think the user there wants to be logproccessor23:37
* clarkb writes a patch23:38
*** mtanino_ is now known as mtanino23:39
ianwnew logo day huh23:39
openstackgerritClark Boylan proposed openstack-infra/system-config master: Don't use subunit user on logstash workers
clarkbianw: indeed!23:40
clarkbmtreinish: fungi ianw that should fix the subunit user error in the paste up there23:40
*** jascott1 has joined #openstack-infra23:42
*** krtaylor has joined #openstack-infra23:43
mtreinishclarkb: :/ lol, I really made a bunch of mistakes on this roll out23:44
clarkbansible looks like it may have gone out to lunch at 23:22 too23:44
clarkbpabelanger: what does it mean when all of the ansible playbook processes are defunct?23:45
pabelangerclarkb: usually problems SSHing to remote node23:45
pabelangerclarkb: if on puppetmaster, they will get killed by timeout fuction23:46
pabelangerI wonder if it is related to new mirrors23:46
pabelangerI didn't ssh into it from puppetmaster this morning23:46
pabelangeror delete old infracloud mirror in openstack client23:46
clarkbah ok, I'm impatietn so I went ahead and downgarded paho and restarted things23:50
clarkbbut when puppet runs it should be happy23:51
fungipabelanger: also if you replace a server you need to blow away the inventory cache23:51
fungiotherwise it gets mightily confused from what i've seen23:52
clarkbmtreinish: fungi can you check that mqtt is happening as expected now?23:54
clarkball but logstash-worker01 should be configured to mqtt23:55
*** thorst_afk has joined #openstack-infra23:55
fungithe example from is at least showing gerrit events...23:56
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Use py35 tox envlist for documentation
mtreinishclarkb: I haven't seen any logstash event come by yet23:57
*** http_GK1wmSU has joined #openstack-infra23:57
funginor ansible23:57
mtreinishjust launchpad, gerrit, and gearman-subunit so far23:58
mtreinishfungi: it might be in an ansible lull23:58
mtreinishthe storm is coming :)23:58
fungi{"status": "success", "build_uuid": "a672db40bc664541aab995f926724aeb"}23:58
fungithat's gearman-subunit?23:58
*** http_GK1wmSU has left #openstack-infra23:58
mtreinishI'm also printing the topics in mine and I haven't seen a logstash one yet23:59
fungiif the ansible storm comes, it'll start in about 45 seconds23:59
mtreinishI also have another window subscribed to gearman-logstash/#23:59
mtreinishwhich hasn't printed anything yet23:59
fungihow do you print topics?23:59
fungiwith mosquitto_sub anyway?23:59

Generated by 2.15.3 by Marius Gedminas - find it at!