Tuesday, 2016-02-02

*** sabeen2 has quit IRC00:01
fungimadorn: the upstream gerrit issue is tracked as https://code.google.com/p/gerrit/issues/detail?id=383100:03
fungitook me a few minutes to hunt that down00:03
madornfungi: great, yeah i was lookin too, thanks00:03
*** davideagnello has quit IRC00:04
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate devstack plugins list  https://review.openstack.org/26992100:05
*** krotscheck1 has joined #openstack-infra00:06
*** salv-orl_ has quit IRC00:08
*** tiswanso has joined #openstack-infra00:08
*** krotscheck1 has quit IRC00:08
*** dims has quit IRC00:10
*** Swami has joined #openstack-infra00:12
*** annegentle has quit IRC00:12
*** tiswanso has quit IRC00:13
clarkbhttps://jenkins01.openstack.org/job/gate-tempest-dsvm-multinode-full/2526/console is running on bluebox00:14
*** vmbrasseur_ has joined #openstack-infra00:14
*** notnownikki has quit IRC00:20
*** annegentle has joined #openstack-infra00:21
*** jpr has joined #openstack-infra00:21
*** mtanino_ has joined #openstack-infra00:22
*** mtanino has quit IRC00:24
*** annegentle has quit IRC00:25
*** vmbrasseur has quit IRC00:29
*** vmbrasseur_ is now known as vmbrasseur00:29
*** MarkAtwood has quit IRC00:32
*** ZZelle_ has quit IRC00:32
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Remove hpcloud from clouds.yaml listings  https://review.openstack.org/27495000:36
fungiclarkb: ^ any idea if there was a pending change for that already?00:36
fungior i guess i could check gerrit's conflicting changes list00:36
fungianyway, that's a new development which has caused ansible to cease puppeting us entirely, from what i can tell in the logs00:36
clarkbfungi: no I don't recall a pending change for that already00:36
*** gildub has quit IRC00:37
clarkbwe can cleanup all our secret data files too00:37
fungiless urgent, but sure00:37
*** eil397 has quit IRC00:41
clarkbfungi: I guess that means they truly shut the cloud down00:45
clarkbone thing I have noticed about neutron and our overlay networks is they circumvent some things00:45
clarkbwhich is :|00:45
fungirolled down the security door and turned off the fluorescents00:46
ianwfungi: when you have time, re centos dib images, maybe you could take a look at https://review.openstack.org/#/c/272857/ (fix for yum.conf)00:46
clarkbI set IPs in the first /24 of as our hypervisor IPs and looks like neutron goes through and sets them out of the second /24 because thta is where we floating ip00:46
clarkbso everything still works, its just not what I wanted00:47
clarkboh that is because its not dvr so the second hypervisor doesn't have a thing00:47
*** devananda has quit IRC00:47
clarkbok maybe I care less then00:47
*** stevebaker has quit IRC00:52
*** stevebaker has joined #openstack-infra00:52
*** unicell has joined #openstack-infra00:52
clarkbthe bluebox with vxlan multinode test failed but ran to completion00:55
*** jamielennox|away is now known as jamielennox00:55
clarkbdoes not appear to have failed on networking but rather timeouts00:55
openstackgerritMerged openstack-infra/system-config: Don't exclude python-setuptools  https://review.openstack.org/27285700:55
*** annegentle has joined #openstack-infra00:56
openstackgerritMerged openstack/diskimage-builder: Skip centos functional testing  https://review.openstack.org/27493700:56
*** SumitNaiksatam has quit IRC01:00
*** MarkAtwood has joined #openstack-infra01:00
*** ashleighfarnham has quit IRC01:01
*** unicell1 has joined #openstack-infra01:02
openstackgerritMerged openstack-infra/system-config: Remove hpcloud from clouds.yaml listings  https://review.openstack.org/27495001:04
*** unicell has quit IRC01:04
*** [1]Thelo has joined #openstack-infra01:04
*** thorst has quit IRC01:05
*** krotscheck1 has joined #openstack-infra01:05
*** thorst has joined #openstack-infra01:06
openstackgerritMerged openstack-infra/system-config: Add $YUM to pickup dnf on Fedora>=22  https://review.openstack.org/25222101:06
*** Thelo has quit IRC01:06
*** [1]Thelo is now known as Thelo01:06
*** shashank_hegde has quit IRC01:07
*** doug-fish has quit IRC01:10
*** bhunter71 has quit IRC01:10
*** doug-fish has joined #openstack-infra01:10
*** tiswanso has joined #openstack-infra01:10
*** EricGonczer_ has quit IRC01:12
*** doug-fis_ has joined #openstack-infra01:12
*** piet has quit IRC01:12
*** krotscheck1 has quit IRC01:13
*** thorst has quit IRC01:14
*** doug-fi__ has joined #openstack-infra01:14
*** doug-fish has quit IRC01:14
nibalizerfungi: I haven't been reading all the scrollback01:15
nibalizerdid we resolve the issue with ansible01:15
nibalizerI get errors when running  ansible all --list-hosts01:15
*** markvoelker has joined #openstack-infra01:15
*** doug-fi__ has quit IRC01:15
clarkbnibalizer: https://review.openstack.org/274950 should resolve it01:16
*** piet has joined #openstack-infra01:16
fungiexcept if we're maybe in a catch-2201:16
*** doug-fish has joined #openstack-infra01:16
*** MarkAtwood has quit IRC01:16
clarkbsince puppet won't apply that change01:16
fungiwill that get applied to the puppetmaster if the run_all script is breaking?01:16
*** doug-fish has quit IRC01:16
*** doug-fis_ has quit IRC01:16
clarkbbut if we get it merged we can do it by hand01:16
clarkbfungi: it won't01:16
nibalizerfungi: we can puppet apply by hand01:16
fungiwell, it merged 10 minutes ago01:16
clarkbfungi: we could fix that by changing run_all to localhost and not ansible01:16
clarkbfungi: maybe do that first01:16
nibalizeroh is that what we broke01:17
clarkber still ansible but just use local connection without inventory01:17
fungiwell, it's the second thing which broke01:17
fungifirst we had 2+ hour timeouts trying to reach the pypi mirror in hpcloud to puppet it, then we started breaking on being able to authenticate to hpcloud01:17
nibalizerwe could run puppet apply at the top of run_all01:17
funginibalizer: since you're on there running stuff already, wanna give it a go?01:18
fungibut yeah, any way to future-proof the existing process would also be great01:18
*** krotscheck1 has joined #openstack-infra01:19
*** markvoelker has quit IRC01:20
clarkbso I think vxlan does work around the bluebox problems01:20
clarkbwe can switch to vxlan and/or work with bluebox to make gre work more reliably01:20
clarkbhowever I think there may be performance issues on bluebox running those tests as they don't seem to be very reliable01:21
clarkbbluebox has a much higher fail rate in general01:21
*** tiswanso has quit IRC01:22
clarkbbut ost of that appears related to vxlan/gre trouble01:22
*** jamielennox is now known as jamielennox|away01:23
*** mtanino_ has quit IRC01:27
clarkball that to say I think we should consider the switch at least temporarily while we work through the rest01:28
nibalizerclouds.yaml updated01:33
fungithanks, nibalizer!01:34
nibalizernote that you have to git pull manually to update the system-config checkout01:34
nibalizerand if the problem was in a module, you'd have to run install_modules.sh01:34
clarkbnibalizer: be careful that you don't implicit merge01:34
*** Daisy has joined #openstack-infra01:35
nibalizerit look like I didn't01:35
nibalizerin the future I should run git fetch, get checkout master or so01:35
clarkbI don't think I ever use git pull01:35
fungii tend to: git remote update ; git checkout master ; git reset --hard origin/master01:38
fungijust to be clear that i'm on an actual commit from the origin's copy of that branch and not some local hork-up01:38
*** angdraug has quit IRC01:39
*** marekd has quit IRC01:40
*** marekd has joined #openstack-infra01:40
*** krotscheck1 has quit IRC01:40
*** Sukhdev has quit IRC01:41
*** annegentle has quit IRC01:46
*** sflanigan has quit IRC01:46
*** annegentle has joined #openstack-infra01:46
*** julim has joined #openstack-infra01:49
*** annegentle has quit IRC01:49
*** tiswanso has joined #openstack-infra01:49
*** tiswanso has quit IRC01:49
*** jgriffith is now known as jgriffith_away01:50
*** tiswanso has joined #openstack-infra01:50
*** Apoorva has quit IRC01:52
*** jgriffith_away is now known as jgriffith01:53
*** tphummel has quit IRC01:53
*** bpokorny has quit IRC01:55
*** Swami has quit IRC01:56
*** jpr has quit IRC01:59
*** krotscheck1 has joined #openstack-infra01:59
*** pvaneck has quit IRC01:59
*** krotscheck1 has quit IRC02:01
*** Jeffrey4l has joined #openstack-infra02:03
*** Sukhdev has joined #openstack-infra02:04
*** jgriffith is now known as jgriffith_away02:04
*** Sukhdev has quit IRC02:05
*** piet has quit IRC02:05
*** krotscheck1 has joined #openstack-infra02:07
*** thorst has joined #openstack-infra02:09
*** dims has joined #openstack-infra02:11
*** thorst has quit IRC02:12
*** thorst has joined #openstack-infra02:13
*** markvoelker has joined #openstack-infra02:16
*** markvoelker has quit IRC02:20
*** thorst has quit IRC02:21
*** EricGonczer_ has joined #openstack-infra02:24
*** krotscheck1 has quit IRC02:26
*** coolsvap|away is now known as coolsvap02:26
*** yamahata has quit IRC02:28
*** krotscheck1 has joined #openstack-infra02:28
*** krotscheck1 has quit IRC02:31
*** krotscheck1 has joined #openstack-infra02:33
prometheanfireianw: I think I know why that revert happened, it's in another branch that was 'fixed'02:38
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add new cloud-init element  https://review.openstack.org/27376402:41
*** jamielennox|away is now known as jamielennox02:44
openstackgerritBen Swartzlander proposed openstack/requirements: Raise manilaclient upper to 1.6.1  https://review.openstack.org/27498302:44
openstackgerritMatthew Thode proposed openstack/diskimage-builder: fix gentoo hardened support  https://review.openstack.org/27379502:44
*** zz_dimtruck is now known as dimtruck02:44
*** EricGonczer_ has quit IRC02:47
*** EricGonczer_ has joined #openstack-infra02:48
*** annegentle has joined #openstack-infra02:48
*** gildub has joined #openstack-infra02:49
*** MarkAtwood has joined #openstack-infra02:49
ianwprometheanfire: i don't want to send your reviews around again, but could you please fix up your subjects to have a captial letter, and reflow them (emacs ctl-q if you use that) so they're indented/wrapped properly02:49
prometheanfireianw: do you mean the commit message?02:50
prometheanfireianw: any one specifically?02:50
ianwprometheanfire: the outstanding ones.  basically, make them look like all the other commit messages02:51
prometheanfirethat doesn't tell me what's wrong with them, also I use vim, so don't have that reflow02:51
prometheanfireianw: any one without workflow?02:52
*** EricGonczer_ has quit IRC02:53
prometheanfireok, what is wrong with them? max line length is 79 for https://review.openstack.org/#/c/273795/02:54
*** sbalukoff has quit IRC02:55
prometheanfirethe other one just needs more info?02:55
ianwprometheanfire: i don't know why but everything after the first line is indented02:56
prometheanfireI did that on purpose for readability :|02:56
ianwThe subject should start with a captial letter if it's a regular word02:56
prometheanfirethose two things it?02:57
prometheanfireor are there other minor nits?02:57
*** annegentle has quit IRC02:58
prometheanfireok, commiting that one then02:59
ianwpromethanfire: i think that's it02:59
*** coolsvap is now known as coolsvap|away02:59
prometheanfireand I'll caps the https://review.openstack.org/#/c/273764/02:59
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Fix Gentoo hardened support  https://review.openstack.org/27379502:59
ianwi think the hardened one is ok, presumably the default won't switch to hardened?  because it seems that where that check is would be too late if that were the case02:59
prometheanfiredefault isn't hardened03:00
prometheanfirethough I kinda would like that03:00
* prometheanfire is part of the gentoo hardened team03:00
clarkbfungi: jeblair we only need room for the mirror host on the openstackci account right?03:01
*** annegentle has joined #openstack-infra03:01
* bkero is curious what kind of stuff grsec is going to be grumpy about WRT Ironic and friends.03:02
clarkbfungi: jeblair trying to make sure we split up our quota properly on osic and don't sacrifice too much of it to unnecessarily oversized mirrors03:02
*** wei-wang has joined #openstack-infra03:02
clarkbI will go ahead with that then we can rebalance later if we need more for mirrors03:03
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add new cloud-init element  https://review.openstack.org/27376403:04
*** Daisy_ has joined #openstack-infra03:05
*** krotscheck1 has quit IRC03:06
clarkbnibalizer: did you make progress running tests on infra cloud? were we able to get the test run times down to something similar to the gate?03:06
*** wei-wang has quit IRC03:06
nibalizerbest I got was 78 minutes03:07
nibalizerjeblair aksmed to poke around in the test directory to find specifics on timing03:07
nibalizerbut I wasn't able to glean any information out of that (mostly because I don't really know what I am looking for)03:08
*** piet has joined #openstack-infra03:09
*** yamamoto has joined #openstack-infra03:09
*** Daisy_ has quit IRC03:09
*** krotscheck1 has joined #openstack-infra03:10
*** amotoki has quit IRC03:10
*** sridhar_ram has quit IRC03:10
ianwnibalizer: devstack at the end prints out some timing, could be quickly compared to see what's out-of-line with others03:11
clarkbnibalizer: I would look at the devstack log and the tempest log to see where the majority of time is spent03:11
clarkbya then compare that to rax/ovh/internap/bluebox et al03:11
openstackgerritMerged openstack/diskimage-builder: Add support for Gentoo to source-repositories  https://review.openstack.org/27377603:15
openstackgerritMerged openstack/diskimage-builder: Add Gentoo support to growroot  https://review.openstack.org/27376903:15
*** links has joined #openstack-infra03:15
openstackgerritMerged openstack/diskimage-builder: Remove outdated translation files  https://review.openstack.org/27445703:16
openstackgerritMerged openstack/diskimage-builder: Fix package-installs for python3  https://review.openstack.org/27378203:16
*** krotscheck1 has quit IRC03:18
*** thorst has joined #openstack-infra03:19
prometheanfireif someone else can review this that'd be nice as it's the thing the other reviews generally need (and one specifically) https://review.openstack.org/#/c/273795/03:19
*** yamamoto has quit IRC03:20
prometheanfireianw: you don't happen to be a core for glean do you?03:21
*** krotscheck1 has joined #openstack-infra03:21
prometheanfireare you, not do you03:21
*** yuanying_ has joined #openstack-infra03:21
ianwprometheanfire: i am not core for glean, yolanda / greghaynes / modred probably people to ping03:22
prometheanfireya, need one more +2 there, but I'll bug them tomorrow03:22
prometheanfireya, guess I need more other dib cores at this point03:24
*** yuanying has quit IRC03:24
clarkbnibalizer: and that was on an image built with cached stuff? 414 seconds for apt get seems slow03:26
clarkbbut I would need to compare03:26
*** thorst has quit IRC03:26
nibalizerclarkb: http://paste.openstack.org/show/485691/03:26
nibalizerI don't think the image had caching no03:26
*** krotscheck1 has quit IRC03:26
*** jgriffith_away is now known as jgriffith03:26
nibalizerjust an ubuntu-minimal vm03:26
ianwgreghaynes: dib released, right ...03:27
openstackgerritSachi King proposed openstack-dev/pbr: Don't attempt to test with 0.6c11 with Py3  https://review.openstack.org/27335803:27
greghaynesianw: nope :( will tomorrow am03:27
greghaynesI spaced on that03:28
prometheanfiregreghaynes: ohai :P03:28
ianwgreghaynes: ok ... yeah f23 needs it for growroot support so it stops running out of disk03:28
greghaynesprometheanfire: hey, I'm EOD so not likely to review anything until tomorrow03:29
*** zhurong has joined #openstack-infra03:29
greghaynesianw: got it03:29
*** krotscheck1 has joined #openstack-infra03:30
*** annegentle has quit IRC03:30
prometheanfiregreghaynes: np, what tz are you btw, don't want to bother when off03:31
*** unicell1 has quit IRC03:31
*** krotscheck1 has quit IRC03:32
prometheanfirek, cst here03:32
*** krotscheck1 has joined #openstack-infra03:33
clarkbnibalizer:  you should compare with one of our images (either buold it there or cp from nodepool.o.o)03:33
*** Daisy has quit IRC03:34
clarkbso ya thats ~5 minutes we can save right there03:34
clarkbandaybe more with git caches03:34
*** sdake has joined #openstack-infra03:35
prometheanfirewill be nice when the osic stuff comes online, perf wise03:35
*** amotoki has joined #openstack-infra03:35
*** annegentle has joined #openstack-infra03:36
*** tphummel has joined #openstack-infra03:36
*** annegentle has quit IRC03:41
*** yuanying has joined #openstack-infra03:41
*** krotscheck1 has quit IRC03:43
*** achanda has joined #openstack-infra03:43
*** yuanying_ has quit IRC03:44
*** SumitNaiksatam has joined #openstack-infra03:48
*** samyewell_ has joined #openstack-infra03:50
*** samyewell has quit IRC03:51
*** samyewell_ is now known as samyewell03:51
*** dims has quit IRC03:53
*** MarkAtwood has quit IRC03:58
*** amotoki has quit IRC04:01
*** armax has quit IRC04:02
*** flwang1 has quit IRC04:03
*** achanda has quit IRC04:04
*** armax has joined #openstack-infra04:05
*** yuanying has quit IRC04:05
*** yuanying has joined #openstack-infra04:06
*** coolsvap|away is now known as coolsvap04:07
*** yuanying_ has joined #openstack-infra04:07
*** yuanying has quit IRC04:07
*** yamamoto_ has joined #openstack-infra04:08
*** diana_clarke has quit IRC04:08
*** jamielennox is now known as jamielennox|away04:10
*** mrmartin has joined #openstack-infra04:11
*** achanda has joined #openstack-infra04:11
fungiclarkb: yeah, just the mirror server for now (and actually very little for that? just enough to hold a warm afs cache probably...)04:14
*** shashank_hegde has joined #openstack-infra04:14
*** mriedem has left #openstack-infra04:15
*** woodster_ has quit IRC04:16
*** markvoelker has joined #openstack-infra04:17
*** sbalukoff has joined #openstack-infra04:18
*** markvoelker has quit IRC04:21
*** isaacb has joined #openstack-infra04:23
*** thorst has joined #openstack-infra04:24
*** thorst has quit IRC04:31
*** tiswanso has quit IRC04:33
openstackgerritIan Wienand proposed openstack-infra/puppet-nodepool: Add option to install diskimage-builder from git  https://review.openstack.org/27500404:37
*** rguillebert has quit IRC04:37
*** jamielennox|away is now known as jamielennox04:39
*** isaacb has quit IRC04:49
openstackgerritIan Wienand proposed openstack-infra/system-config: Use diskimage-builder from git for nodepool  https://review.openstack.org/27500604:55
*** aukhan has joined #openstack-infra04:55
aukhanHi Folks, could you kindly share how to install "Zuul"  (i.e. without using puppet) ?04:57
prometheanfirefigure out what puppet's doing, and do that :P04:58
openstackgerritJake Yip proposed openstack/os-testr: correct typo  https://review.openstack.org/27500804:59
aukhanThanks @prometheanfire, I was looking for a quick way to install it :P04:59
aukhanI think i am just being too lazy04:59
*** gongysh has joined #openstack-infra05:00
jrollpuppet is the easy way :P05:00
aukhanI thought there was a way to like do a "apt-get install zuul" and get done :P05:00
aukhanI agree with you both @jroll/@prometheanfire05:00
jrollso, pip install zuul should just work, but there's a handful of services it looks like05:01
prometheanfireya, there's a setup.py05:01
*** harlowja_at_home has joined #openstack-infra05:01
aukhanI did try isntallin using the setup.py05:01
aukhanbut had to resolve a lot of dependencies manually05:01
nibalizeraukhan: i think you'd do best to use this http://docs.openstack.org/infra/openstackci/05:02
prometheanfirepip should do deps for you05:02
aukhanpip did not i think there was some issue with my pip version or something05:03
*** fedexo has joined #openstack-infra05:03
aukhanhello @nibalizer, thanks will check that out05:04
aukhanthanks a ton guys for the quick response05:04
*** aopincar has joined #openstack-infra05:04
*** aopincar has quit IRC05:04
*** amotoki has joined #openstack-infra05:05
*** sdake has quit IRC05:05
*** unicell has joined #openstack-infra05:07
nibalizeraukhan: also assuming you work at hpe, asselin is a great resource to work with05:07
*** jamielennox is now known as jamielennox|away05:13
*** shashank_hegde has quit IRC05:17
*** Sukhdev has joined #openstack-infra05:17
*** thorst has joined #openstack-infra05:30
jeblairnibalizer: ah, yeah, if you're working without a cache on infra cloud, then you'll want to only directly compare the tempest runtimes as printed out e.g. here http://logs.openstack.org/12/272412/2/check/gate-tempest-dsvm-full/f20896d//console.html#_2016-02-02_04_25_41_45005:31
jeblairnibalizer: the first number there -- 1144 in that example is the most important for comparison05:32
*** gyee has quit IRC05:32
jeblairclarkb: yeah, i've been setting aside 100-200G for afs caches but only configured it to use 50G so far (and it's using way less than that).05:35
jeblairclarkb: see http://docs-draft.openstack.org/73/273673/1/check/gate-infra-specs-docs/f9e8572//doc/build/html/specs/unified_mirrors.html#servers05:35
jeblairclarkb: and i've been doing 2G servers05:36
*** thorst has quit IRC05:36
*** Daisy has joined #openstack-infra05:36
*** exploreshaifali has joined #openstack-infra05:39
*** AzherKhan has joined #openstack-infra05:44
*** piet has quit IRC05:44
*** aukhan has quit IRC05:44
*** sileht has quit IRC05:44
*** piet has joined #openstack-infra05:45
*** Swami has joined #openstack-infra05:49
*** bradm_ is now known as bradm05:54
*** changbl has quit IRC05:56
*** kushal has quit IRC05:57
*** vivekd has joined #openstack-infra05:57
*** dimtruck is now known as zz_dimtruck06:00
*** hdd has quit IRC06:02
*** gongysh has quit IRC06:04
*** oomichi has joined #openstack-infra06:04
*** abregman has joined #openstack-infra06:06
*** abregman has quit IRC06:06
*** vgridnev has joined #openstack-infra06:06
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee  https://review.openstack.org/26942706:07
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee  https://review.openstack.org/26942706:09
*** piet has quit IRC06:11
*** abregman has joined #openstack-infra06:11
*** harlowja_at_home has quit IRC06:11
*** kushal has joined #openstack-infra06:11
*** fawadkhaliq has joined #openstack-infra06:11
*** gildub has quit IRC06:11
*** fawadkhaliq has quit IRC06:11
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee  https://review.openstack.org/26942706:14
*** jtomasek has quit IRC06:15
*** jyuso1 has joined #openstack-infra06:16
*** markvoelker has joined #openstack-infra06:18
*** diana_clarke has joined #openstack-infra06:18
*** fedexo has quit IRC06:18
*** aopincar has joined #openstack-infra06:19
*** rcernin has joined #openstack-infra06:22
*** markvoelker has quit IRC06:22
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/27502706:22
*** boris-42 has quit IRC06:23
openstackgerritdaisy-ycguo proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite Approved specification: translation_check_site.rst  https://review.openstack.org/27470106:24
*** shashank_hegde has joined #openstack-infra06:26
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/27390706:26
*** Swami has quit IRC06:28
openstackgerritZhenyu Zheng proposed openstack-infra/project-config: Add py27 and docs job to terracotta.  https://review.openstack.org/27503106:35
*** thorst has joined #openstack-infra06:35
openstackgerritZhenyu Zheng proposed openstack-infra/project-config: Add py27 and docs job to terracotta.  https://review.openstack.org/27503106:39
*** vgridnev has quit IRC06:40
*** vgridnev has joined #openstack-infra06:41
*** thorst has quit IRC06:41
*** tphummel has quit IRC06:45
*** rcernin has quit IRC06:46
*** shashank_hegde has quit IRC06:48
*** dalgaaf has joined #openstack-infra06:51
*** amotoki_ has joined #openstack-infra06:51
*** exploreshaifali has quit IRC07:00
*** followinfrared has joined #openstack-infra07:02
*** armax has quit IRC07:03
*** amotoki has quit IRC07:03
*** keedya has joined #openstack-infra07:04
AJaegerSam-I-Am: I can delete files on docs.openstack.org - as can annegentle. Send us an email...07:04
*** eliqiao has quit IRC07:10
*** eliqiao has joined #openstack-infra07:12
openstackgerritdaisy-ycguo proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite Approved specification: translation_check_site.rst  https://review.openstack.org/27470107:13
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/27502707:13
*** _nadya_ has joined #openstack-infra07:16
*** coolsvap is now known as coolsvap|away07:16
*** shashank_hegde has joined #openstack-infra07:18
*** _nadya_ has quit IRC07:20
openstackgerritAkihiro Motoki proposed openstack-infra/irc-meetings: Drop I18N Translation test meeting  https://review.openstack.org/27504507:21
*** zaro has joined #openstack-infra07:23
openstackgerritZhenyu Zheng proposed openstack-infra/project-config: Add py27, docs and publishing to pypi jobs to terracotta.  https://review.openstack.org/27503107:24
*** keedya has quit IRC07:26
AJaegeramotoki_: do you agree to push https://review.openstack.org/#/c/274421/ forward? Today's imports look fine - https://review.openstack.org/#/q/status:open+topic:zanata/translations,n,z07:28
* AJaeger just double checks jenkins log files...07:28
amotoki_AJaeger: looking07:29
*** zul has joined #openstack-infra07:29
*** sileht_ has joined #openstack-infra07:29
*** yamahata has joined #openstack-infra07:29
*** ianw has quit IRC07:29
*** ianw has joined #openstack-infra07:30
nibalizerjeblair: http://paste.openstack.org/show/485697/07:31
*** scheuran has joined #openstack-infra07:33
*** sileht_ is now known as sileht07:35
*** abregman has quit IRC07:35
*** Sukhdev has quit IRC07:38
*** esikachev has joined #openstack-infra07:39
*** thorst has joined #openstack-infra07:39
*** achanda has quit IRC07:40
openstackgerritIan Wienand proposed openstack-infra/system-config: Use diskimage-builder from git for nodepool  https://review.openstack.org/27500607:41
*** zul has quit IRC07:41
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use new translation setup for all python projects  https://review.openstack.org/27442107:41
AJaegeramotoki_: here's an update ^. Thanks for reviewing07:41
AJaegernibalizer, ianw: Could you review 274421 as next step for translation setup, please? We're starting to clean up now ...07:42
*** achanda has joined #openstack-infra07:43
nibalizerAJaeger: sorry but I don't really understand that well enough to review quickly07:45
nibalizerafter my talk today I'll try to cirlce around07:45
AJaegernibalizer: understood. What are you talking about?07:46
*** thorst has quit IRC07:46
*** fhubik has joined #openstack-infra07:49
*** sileht has quit IRC07:50
*** sileht has joined #openstack-infra07:50
nibalizerAJaeger: ansible+puppet07:51
scheuranGood morning, is there a known problem with the gate-tempest-dsvm-layer4 job? My tempest patch always fails with:  Details: No "_member_" role found. I don't think that this is related to my change...07:54
*** flepied has quit IRC07:55
*** achanda has quit IRC07:55
AJaegerscheuran: better ask on #openstack-qa - I'm not aware of anything08:00
scheuranAJaeger, ok, thanks!08:00
*** diana_clarke has quit IRC08:02
*** amotoki_ is now known as amotoki08:03
*** fhubik has quit IRC08:05
*** sdake has joined #openstack-infra08:09
esikachevhi all! i need requirements cores. take a look around:)08:10
*** jistr has joined #openstack-infra08:11
*** oomichi has quit IRC08:12
*** ildikov has quit IRC08:13
*** jtomasek_ has joined #openstack-infra08:13
*** oomichi has joined #openstack-infra08:13
*** samuelBartel has joined #openstack-infra08:14
*** matrohon has joined #openstack-infra08:14
*** vivekd has quit IRC08:16
*** boris-42 has joined #openstack-infra08:17
*** markvoelker has joined #openstack-infra08:18
*** nikil89_ has joined #openstack-infra08:19
*** nikil89_ is now known as naggappan08:19
*** mrmartin has quit IRC08:19
*** shashank_hegde has quit IRC08:19
*** fhubik has joined #openstack-infra08:20
*** mikelk has joined #openstack-infra08:22
*** vivekd has joined #openstack-infra08:22
*** markvoelker has quit IRC08:22
*** zeih has joined #openstack-infra08:23
*** zeih has quit IRC08:23
*** tlbr_ has quit IRC08:23
*** zeih has joined #openstack-infra08:24
*** k4n0 has joined #openstack-infra08:26
*** mrmartin has joined #openstack-infra08:26
*** k4n0 has quit IRC08:26
*** cznewt has joined #openstack-infra08:27
*** dingyichen has quit IRC08:29
*** flepied has joined #openstack-infra08:32
*** mrmartin has quit IRC08:34
*** fhubik_real has joined #openstack-infra08:37
yolandaAJaeger, things are a bit quieter today, i can start looking at pending things08:37
*** hashar has joined #openstack-infra08:38
*** e0ne has joined #openstack-infra08:38
AJaegermorning, yolanda. 274421 is my only one right now ;) Thanks!08:38
*** mrmartin has joined #openstack-infra08:39
*** fhubik has quit IRC08:40
*** fhubik_real has quit IRC08:42
AJaegeryolanda: There's the usual round of open reviews, nothing critical to point to... Thanks!08:42
yolandaok going to do that round08:42
*** tlbr has joined #openstack-infra08:43
*** vivekd has quit IRC08:43
yolandaAJaeger about https://review.openstack.org/#/c/275031 ... this says it adds docs jobs, but i'm not seeing those on the change08:44
*** thorst has joined #openstack-infra08:45
*** rubasov has joined #openstack-infra08:45
*** yaume has joined #openstack-infra08:47
AJaegerthey are part of the template added08:47
*** kzaitsev_mb has joined #openstack-infra08:47
AJaegerThe python-jobs templates add docs jobs to check and gate08:48
yolandaah ok08:48
*** vivekd has joined #openstack-infra08:48
openstackgerritMerged openstack-infra/project-config: Add growroot element to Fedora 23  https://review.openstack.org/27190708:49
*** marekd has quit IRC08:50
*** abregman has joined #openstack-infra08:50
*** thorst has quit IRC08:51
*** marekd has joined #openstack-infra08:52
*** yamahata has quit IRC08:52
*** exploreshaifali has joined #openstack-infra08:53
*** abregman is now known as abregman|intervi08:54
openstackgerritMerged openstack-infra/project-config: Add py27, docs and publishing to pypi jobs to terracotta.  https://review.openstack.org/27503108:55
*** zul has joined #openstack-infra08:56
ianwAJaeger: "In addition, enable new setup for vmware-nsx and ironic-inspector" ... is that in there?08:56
*** aopincar has quit IRC08:56
*** rossella_s has joined #openstack-infra08:56
openstackgerritEvgeny Antyshev proposed openstack-infra/ciwatch: Ability to select specific CI results  https://review.openstack.org/27441208:56
*** jlanoux has joined #openstack-infra08:57
*** abregman has joined #openstack-infra08:58
*** ifarkas has joined #openstack-infra08:58
*** abregman|intervi has quit IRC08:58
*** marekd has quit IRC09:01
*** dprince has joined #openstack-infra09:03
openstackgerritMerged openstack-infra/project-config: zuul/layout: optimize puppet jobs  https://review.openstack.org/27413709:05
*** Kevin_Zheng has joined #openstack-infra09:06
AJaegerianw: let me double check...09:06
*** jyuso1 has quit IRC09:07
AJaegerianw: https://review.openstack.org/#/c/274421/3/jenkins/scripts/propose_translation_update.sh line 239 in old has:09:07
AJaegerbut new line 188 only09:08
openstackgerritMerged openstack-infra/glean: enable glean in gentoo's init on install  https://review.openstack.org/27450509:08
ianwAJaeger: ok, cool09:09
AJaegerthanks, ianw09:10
*** vgridnev has joined #openstack-infra09:10
ianwAJaeger: lgtm, but i didn't +w as i'm not super confident in the area, feel free to self approve09:10
AJaegerianw: will do - and monitor as usual ;)09:11
*** abregman is now known as abregman|intervi09:11
*** jordanP has joined #openstack-infra09:15
openstackgerritMerged openstack-infra/project-config: Added three new repos for fuel-plugins  https://review.openstack.org/26900709:16
*** mrmartin has quit IRC09:18
*** dizquierdo has joined #openstack-infra09:19
*** mrmartin has joined #openstack-infra09:20
*** ildikov has joined #openstack-infra09:20
openstackgerritMerged openstack-infra/project-config: Use new translation setup for all python projects  https://review.openstack.org/27442109:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-Enable django_openstack_auth/designate-dashboard translations  https://review.openstack.org/27483209:25
*** HeOS_ has quit IRC09:25
*** achanda has joined #openstack-infra09:25
*** jcoufal has joined #openstack-infra09:28
*** dalgaaf has quit IRC09:28
*** zhurong has quit IRC09:30
*** achanda has quit IRC09:30
prometheanfireyolanda: thanks for the review :D09:32
*** vivekd has quit IRC09:33
*** vivekd_ has joined #openstack-infra09:33
*** vivekd_ is now known as vivekd09:33
*** andymaier has joined #openstack-infra09:34
*** mrmartin has quit IRC09:35
*** wei-wang has joined #openstack-infra09:36
*** krotscheck1 has joined #openstack-infra09:37
*** zeih has quit IRC09:37
*** wei-wang has quit IRC09:37
*** zeih has joined #openstack-infra09:38
*** _nadya_ has joined #openstack-infra09:39
*** followin_ has joined #openstack-infra09:39
*** ihrachys has joined #openstack-infra09:40
*** amotoki has quit IRC09:40
nibalizerjeblair: so the difference is 1883.0000 - 1144 = 73909:41
nibalizerwhich is about 12 minutes09:41
*** notnownikki has joined #openstack-infra09:41
*** followinfrared has quit IRC09:42
*** zeih has quit IRC09:43
yolandahi nibalizer, AJaeger, have you ever used curl with gerrit REST API?09:43
*** _nadya_ has quit IRC09:43
yolandai'm trying to fix the documentation of third party, but i tested authenticating with --digest and fails for me09:43
*** kzaitsev_mb has quit IRC09:43
yolandai always did it just being authenticated on a browser, so i may be doing something wrong09:43
nibalizerdid you have /a/ in the url?09:44
*** hashar_ has joined #openstack-infra09:44
nibalizeryolanda: i used python with https with gerrit09:45
yolandacurl --digest --user yolanda.robla:<<pass>> -X GET https://review.openstack.org/a/accounts/self09:45
yolandai get 40109:45
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Ignore empty django files  https://review.openstack.org/27509009:45
nibalizeryolanda: https://review.openstack.org/#/c/216877/1/jeepyb/cmd/branch_mgmt.py is the script Iwas using09:46
*** hashar has quit IRC09:47
yolandanibalizer, i use digest as well, and i send my gerrit username09:49
*** ihrachys has quit IRC09:49
yolandagoing to try with projects endpoint instead of accounts...09:49
*** thorst has joined #openstack-infra09:49
yolandasame thing09:50
yolandaand i have to write instructions for curl09:50
nibalizeri wonder if that used to work for 2.8 and something changed in 2.11?09:50
yolandathen we have another problem apart from writing the documentation :(09:51
*** zeih has joined #openstack-infra09:51
yolandai cannot see anything related on gerrit logs09:51
yolandamm, maybe pass needs to be hashed09:52
*** rhallisey has joined #openstack-infra09:53
*** mrmartin has joined #openstack-infra09:54
*** amotoki has joined #openstack-infra09:55
*** gongysh has joined #openstack-infra09:56
*** thorst has quit IRC09:57
yolandano luck :(09:57
*** abregman|intervi is now known as abregman09:58
*** mrmartin has quit IRC09:58
*** vivekd has quit IRC10:03
*** abregman has quit IRC10:03
*** alivigni has joined #openstack-infra10:04
*** rguillebert has joined #openstack-infra10:08
*** amotoki has quit IRC10:10
*** dprince has quit IRC10:12
*** mkwiek has joined #openstack-infra10:13
mkwiekhi guys! is there anything else I need to do to get this patch merged to master in fuel-web https://review.openstack.org/#/c/274034/ ? It was in merge-check for several minutes, but now I can't see it anywhere on zuul, nor I can see it merged to master.10:14
*** abregman has joined #openstack-infra10:15
*** openstackgerrit has quit IRC10:17
*** openstackgerrit has joined #openstack-infra10:17
openstackgerritMoshe Levi proposed openstack/diskimage-builder: Adding InfiniBand Support  https://review.openstack.org/27478110:19
*** markvoelker has joined #openstack-infra10:19
*** esikachev has quit IRC10:21
*** esikachev has joined #openstack-infra10:22
*** markvoelker has quit IRC10:23
AJaegeryolanda: no, never10:23
*** amotoki has joined #openstack-infra10:24
yolandalet's see if any other infra-root knows10:24
AJaegeramotoki: translation import works fine for python projects. Just noticed one issue on django: https://review.openstack.org/27509010:25
*** gongysh has quit IRC10:25
AJaegermkwiek: merge-check has nothing to do with merge - changes are in to check that they can merge, this is triggered by another change merging.10:28
AJaegermkwiek: you need to check gate queue10:28
AJaegermkwiek: your change has a depends-on on another change - and your change was approved before the other merged.10:29
mkwiekoh, so if +1 workflow is removed and added again it should merge?10:29
AJaegermkwiek: since the jobs do not share a queue, there'S no notification of this and so somebody else needs to approve your change again.10:29
mkwiekcool, thank you AJaeger10:29
AJaegermkwiek: Exactly, another +W or toggling the existing +W will start it10:29
*** amotoki_ has joined #openstack-infra10:30
*** _nadya_ has joined #openstack-infra10:31
*** rossella_s has quit IRC10:31
*** amotoki has quit IRC10:31
*** rossella_s has joined #openstack-infra10:32
*** vivekd has joined #openstack-infra10:33
*** yamamoto_ has quit IRC10:37
openstackgerritVincent Françoise proposed openstack-infra/project-config: Enable multinode testing in Watcher  https://review.openstack.org/27406810:38
*** jcoufal has quit IRC10:39
*** AzherKhan has quit IRC10:42
*** amotoki_ has quit IRC10:42
*** followinfrared has joined #openstack-infra10:45
*** followinfrared has quit IRC10:45
*** amotoki has joined #openstack-infra10:45
*** followinfrared has joined #openstack-infra10:45
*** kzaitsev_mb has joined #openstack-infra10:47
*** rfolco has joined #openstack-infra10:47
*** followin_ has quit IRC10:48
*** followinfrared has quit IRC10:50
*** ildikov has quit IRC10:51
*** electrofelix has joined #openstack-infra10:53
*** dtantsur|afk is now known as dtantsur10:54
*** followinfrared has joined #openstack-infra10:54
*** thorst has joined #openstack-infra10:56
*** ociuhandu has quit IRC10:58
*** thorst has quit IRC11:01
*** dims_ has joined #openstack-infra11:02
*** _nadya_ has quit IRC11:03
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Provide fallback mirrors for centos pin  https://review.openstack.org/27428711:03
*** aysyd has joined #openstack-infra11:03
*** [1]Thelo has joined #openstack-infra11:04
*** _amrith_ is now known as amrith11:05
*** Thelo has quit IRC11:07
*** [1]Thelo is now known as Thelo11:07
*** exploreshaifali has quit IRC11:09
*** samuelBartel has quit IRC11:15
openstackgerritMerged openstack-infra/release-tools: don't require launchpad milestone for stable branch  https://review.openstack.org/27313311:17
*** ildikov has joined #openstack-infra11:19
openstackgerritZara proposed openstack-infra/storyboard-webclient: Use 'Tasks' Icon for Tasks  https://review.openstack.org/27513011:21
*** jistr has quit IRC11:22
*** jlanoux_ has joined #openstack-infra11:27
*** jlanoux has quit IRC11:28
*** achanda has joined #openstack-infra11:29
*** amotoki has quit IRC11:30
*** ociuhandu has joined #openstack-infra11:34
*** achanda has quit IRC11:34
*** MarkAtwood has joined #openstack-infra11:38
*** zeih has quit IRC11:39
*** esp has joined #openstack-infra11:46
*** tcammann has joined #openstack-infra11:46
*** dizquierdo is now known as dizquierdo_afk11:47
*** gus_ is now known as gus11:47
*** NobodyCa1 has joined #openstack-infra11:48
*** tcammann has quit IRC11:51
*** tzn has joined #openstack-infra11:52
*** NobodyCa1 has quit IRC11:53
*** esp has quit IRC11:53
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-enable python and networking repos for translation  https://review.openstack.org/27514011:54
*** jlanoux_ has quit IRC11:55
*** exploreshaifali has joined #openstack-infra11:56
*** jlanoux has joined #openstack-infra11:58
*** kzaitsev_mb has quit IRC11:58
AJaegeryolanda: The change above and https://review.openstack.org/275090 are ready for review, please. These are again translation related.11:59
*** thorst has joined #openstack-infra12:00
*** thorst has quit IRC12:07
*** dalgaaf has joined #openstack-infra12:08
*** maishsk has joined #openstack-infra12:10
*** vikram has joined #openstack-infra12:10
*** Daisy has quit IRC12:10
*** rhallisey has quit IRC12:13
*** kzaitsev_mb has joined #openstack-infra12:14
*** andymaier has quit IRC12:15
*** yamamoto has joined #openstack-infra12:15
*** _nadya_ has joined #openstack-infra12:16
*** vikram has quit IRC12:17
amaretskiy AJaeger: hi12:18
AJaegerhi amaretskiy12:18
*** zeih has joined #openstack-infra12:18
amaretskiy AJaeger: have you looked at http://amaretskiy.droppages.com/lost-seconds-while-setting-up-dsvm-workspace.html ?12:18
AJaegeryes, seen it - and the comments by fungi and jeblair yesterday. Thanks.12:18
amaretskiyAJaeger: I prepared some stat regarding the problem with stuck requests12:18
amaretskiyAJaeger: i missed comments (being offline) :(12:19
AJaegeramaretskiy: use eavesdrop, let me give you a link...12:19
*** vikram_ has joined #openstack-infra12:19
AJaegeramaretskiy: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-01.log.html#t2016-02-01T17:12:5512:19
*** markvoelker has joined #openstack-infra12:20
AJaegerQuestion is how to move forward with this and figure out the root cause. fungi and jeblair gave some suggestions, perhaps you want to followup with them? I won't have time to do this myself.12:20
*** amotoki has joined #openstack-infra12:20
amaretskiyI found comment sin history, great12:20
amaretskiyAJaeger: I could dig this deeper but maybe a bit later, too busy now with the current task12:22
*** yamamoto has quit IRC12:22
amaretskiyAJaeger: I keep this in mind, maybe will be back to this soon12:23
*** yamamoto has joined #openstack-infra12:23
*** weshay_xchat has joined #openstack-infra12:24
*** chaitu has joined #openstack-infra12:24
*** markvoelker has quit IRC12:24
*** thiagop_ has joined #openstack-infra12:25
chaituI am trying to setup fresh ci deployment when i start nodepool service, I dont see any image bulding in nodepool12:25
openstackgerritTimur Sufiev proposed openstack/requirements: Exclude xvfbwrapper 0.2.8 from global requirements  https://review.openstack.org/27515512:26
chaituMy nodepool image.log is also empty12:26
*** bnemec has joined #openstack-infra12:26
*** thiagop has quit IRC12:26
chaitucan someone please tell me how could i make sure that my image is building12:27
*** yamamoto has quit IRC12:27
*** thiagop_ is now known as thiagop12:30
*** vikram_ has quit IRC12:31
*** dizquierdo_afk is now known as dizquierdo12:33
*** julim_ has joined #openstack-infra12:34
*** j_king_ has joined #openstack-infra12:35
*** vikram_ has joined #openstack-infra12:35
*** gordc has joined #openstack-infra12:35
*** jlanoux_ has joined #openstack-infra12:36
*** j_king has quit IRC12:36
*** julim has quit IRC12:36
*** DuncanT has quit IRC12:36
*** links has quit IRC12:36
*** anthonyper has quit IRC12:36
*** ptoohill has quit IRC12:36
*** AJaeger has quit IRC12:36
*** AJaeger has joined #openstack-infra12:36
*** jlanoux has quit IRC12:36
*** bswartz has quit IRC12:36
*** DuncanT has joined #openstack-infra12:37
*** ptoohill has joined #openstack-infra12:38
*** anthonyper has joined #openstack-infra12:38
*** links has joined #openstack-infra12:39
*** lucasagomes is now known as lucas-hungry12:39
openstackgerritVincent Françoise proposed openstack-infra/project-config: Watcher Dashboard project creation  https://review.openstack.org/27516712:41
*** thorst has joined #openstack-infra12:42
*** boris-42 has quit IRC12:43
*** yamamoto has joined #openstack-infra12:46
openstackgerritVincent Françoise proposed openstack-infra/project-config: Watcher Dashboard project creation  https://review.openstack.org/27516712:47
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added three new repos for fuel-plugins  https://review.openstack.org/26956712:49
*** deva_ has joined #openstack-infra12:49
*** tcammann has joined #openstack-infra12:49
*** esp has joined #openstack-infra12:50
*** baoli has joined #openstack-infra12:53
*** andymaier has joined #openstack-infra12:53
*** yamamoto has quit IRC12:56
*** tcammann has quit IRC12:56
*** erlon has joined #openstack-infra12:57
*** esp has quit IRC12:57
*** trown|outttypeww is now known as trown12:57
*** dtardivel has joined #openstack-infra12:57
*** deva_ has quit IRC12:58
*** NobodyCa1 has joined #openstack-infra12:59
*** anteaya has joined #openstack-infra12:59
*** lxsli has joined #openstack-infra12:59
*** amotoki has quit IRC13:00
*** tcammann_ has joined #openstack-infra13:01
openstackgerritOndřej Nový proposed openstack/requirements: Bump PyECLib version to 1.0.8  https://review.openstack.org/27518113:02
AJaegerfungi, anteaya: Did you ever have see a followup from angdraug to the fuel plugin discussion from 2015-12-29? I commented now on 269567 as well...13:02
*** flepied has quit IRC13:05
*** oomichi has quit IRC13:05
*** ayoung has quit IRC13:05
*** tcammann_ has quit IRC13:05
*** anteaya has quit IRC13:06
*** exploreshaifali has quit IRC13:06
*** lxsli has quit IRC13:06
*** NobodyCa1 has quit IRC13:06
*** doug-fish has joined #openstack-infra13:07
openstackgerritTristan Cacqueray proposed openstack-infra/zuul: Add poll.poll() timeout  https://review.openstack.org/27444513:08
*** rlandy has joined #openstack-infra13:09
*** dizquierdo has quit IRC13:09
openstackgerritVictor Ryzhenkin proposed openstack-infra/project-config: Make murano tempest job voring and add it to gate pipeline  https://review.openstack.org/27519113:11
*** maishsk_ has joined #openstack-infra13:12
*** maishsk has quit IRC13:12
*** maishsk_ is now known as maishsk13:12
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: WIP/PoC: Live-updating kanban boards  https://review.openstack.org/27519313:13
*** hashar_ is now known as hashar13:13
*** amotoki has joined #openstack-infra13:16
*** ihrachys has joined #openstack-infra13:16
*** MarkAtwood has quit IRC13:17
*** thiagop has quit IRC13:18
*** samuelBartel has joined #openstack-infra13:20
clarkbyolanda make sure that you arent mixing up accounts and that the password from the web ui belongs to that user13:20
*** markvoelker has joined #openstack-infra13:21
*** davidlenwell has quit IRC13:24
*** thiagop has joined #openstack-infra13:24
*** xyang1 has joined #openstack-infra13:25
*** gongysh has joined #openstack-infra13:25
*** markvoelker has quit IRC13:26
*** markvoelker_ has joined #openstack-infra13:26
*** yamamoto has joined #openstack-infra13:27
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Check branch-override in ofagent job-template  https://review.openstack.org/27493013:27
*** yamamoto has quit IRC13:27
*** yamamoto has joined #openstack-infra13:28
*** davidlenwell has joined #openstack-infra13:29
*** FallenPegasus has joined #openstack-infra13:30
*** vivekd has quit IRC13:31
*** mikelk has quit IRC13:31
*** achanda has joined #openstack-infra13:33
*** zeih has quit IRC13:33
*** vivekd has joined #openstack-infra13:33
krotscheck1clarkb: One of the things I've been asked to do for wheel mirrors is to generate a short lsb_release slug that's reasonably unique. I've got the code, but given that it's going to be used in both builders and in configure_mirror.sh, I'm wondering if there's a place where I can just have that slug written into the env. So I can reuse it elsewhere. Any suggestions?13:34
krotscheck1clarkb: For reference: http://paste.openstack.org/show/485734/13:36
*** pradk has joined #openstack-infra13:36
*** pradk has quit IRC13:36
krotscheck1clarkb: Sorry, quick syntax revision : http://paste.openstack.org/show/485735/13:37
AJaegerkrotscheck1: can't we use /etc/os-release nowadays? Aren't all our supported distros new enough to support that instead?13:37
*** edmondsw has joined #openstack-infra13:37
*** tlbr has quit IRC13:37
amotokiAJaeger: neutron-fwaas translation patch just has been put into the gate13:37
AJaegeramotoki: yippie! Great!13:37
*** achanda has quit IRC13:38
AJaegerkrotscheck1: os-release has variables that are ease to parse...13:38
*** thiagop has quit IRC13:39
*** keedya has joined #openstack-infra13:40
jesusaurclarkb: when you're around can you take a look at https://review.openstack.org/254992 ?13:40
krotscheck1AJaeger: Lemme check.13:40
*** annegentle has joined #openstack-infra13:40
krotscheck1AJaeger: fedora23 has it. CHecking centos and trusty13:41
*** thiagop has joined #openstack-infra13:41
openstackgerritFlavio Percoco proposed openstack-infra/irc-meetings: Remove glance-drivers meeting  https://review.openstack.org/27520913:41
openstackgerritFlavio Percoco proposed openstack-infra/irc-meetings: Change chair for Glance's meeting  https://review.openstack.org/27521013:41
*** andymaier has quit IRC13:42
*** burgerk has joined #openstack-infra13:43
*** pradk has joined #openstack-infra13:43
*** pradk has quit IRC13:43
*** tzn has quit IRC13:44
AJaegerkrotscheck1: not sure whether there's a best practice for usage in OpenStack infra.13:44
AJaegerkrotscheck1: centos7 should have it.13:45
*** regXboi has joined #openstack-infra13:45
*** zul has quit IRC13:45
krotscheck1AJaeger: They all do. The only catch is that ubuntu doesn't include 'trusty' in any of the vars, so I have to get that using lsb_release13:46
*** zeih has joined #openstack-infra13:46
krotscheck1AJaeger: I have to generate a slug that's less than 13 characters long.13:47
*** hichihara has joined #openstack-infra13:47
*** tzn has joined #openstack-infra13:47
krotscheck1AJaeger: Oh, doh. I need to include the architecture as well.13:48
krotscheck1So, trusty64, centos7-x6413:48
clarkbkrotscheck1: you can put it in project-cpnfig/jenkins/scripts then any script can source it13:48
*** zeih has quit IRC13:48
krotscheck1clarkb: Gotcha13:49
clarkbalso we have been using a - between distro and release elsewhere and you need to encode the cpu arch too iirc13:49
AJaegerclarkb: good idea, write a simple script and place it there...13:49
*** zeih has joined #openstack-infra13:49
clarkbthere is possibility we may have arm at some point13:49
AJaegermorning, Sam-I-Am13:49
*** BobBall_ has joined #openstack-infra13:50
*** BobBall_ is now known as Guest9815513:50
*** lucas-hungry is now known as lucasagomes13:50
*** Guest98155 is now known as Bobba13:51
AJaegerproject-config cores, translation unification is working fine, we're switching everything over to new setup and cleanup. Could I get reviews for https://review.openstack.org/#/c/274832 , please?13:51
*** Bobba is now known as BobBall113:51
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix on username generation  https://review.openstack.org/27521513:51
*** krtaylor has quit IRC13:51
*** amrith is now known as _amrith_13:51
AJaeger275090 and 274359 are also related but have less priority13:51
*** keedya has quit IRC13:51
*** amitgandhinz has joined #openstack-infra13:52
esikachevttx: hi!13:52
*** BobBall has quit IRC13:53
esikachevttx: patch from your comment updated https://review.openstack.org/#/c/270676/13:53
*** anteaya has joined #openstack-infra13:53
*** jtomasek_ has quit IRC13:55
anteayamorning pabelanger13:56
openstackgerritAlexey Shtokolov proposed openstack-infra/project-config: Add fuel-plugin-etckeeper repository for Fuel  https://review.openstack.org/26020813:57
*** jgriffith is now known as jgriffith_away13:58
AJaegerhi anteaya, feeling well again?13:58
*** zul has joined #openstack-infra13:59
anteayaAJaeger: not a 100% but better than yesterday, thanks for asking13:59
anteayamy weechat server went away :(13:59
*** BobBall1 is now known as BobBall13:59
*** jcoufal has joined #openstack-infra14:00
anteayaI'm trying to acertain whether there is or is not a neutron meeting today14:00
anteayaor if it was yesterday and I missed it14:00
*** baoli has quit IRC14:00
AJaegerseems there is - was just mentioned on #openstack-neutron14:00
*** yaume_ has joined #openstack-infra14:00
*** dkranz has joined #openstack-infra14:00
*** woodster_ has joined #openstack-infra14:01
*** baoli has joined #openstack-infra14:01
*** salv-orlando has joined #openstack-infra14:01
*** Guest37814 has quit IRC14:01
*** yaume has quit IRC14:02
*** BobBall has quit IRC14:04
*** krtaylor has joined #openstack-infra14:05
*** msuriar has left #openstack-infra14:06
*** tzn has quit IRC14:07
*** changbl has joined #openstack-infra14:07
*** tzn has joined #openstack-infra14:07
*** FallenPegasus has quit IRC14:08
*** jtomasek__ has joined #openstack-infra14:09
krotscheck1Can anyone here help me assemble a list of possible output for uname -m? I've got 'x86_64', I need output from a 32bit system, as well as an ARM processor.14:10
* krotscheck1 can't believe he just asked that.14:10
*** tlbr has joined #openstack-infra14:11
*** sdake has quit IRC14:11
*** thomasem-away is now known as thomasem14:13
*** rhallisey has joined #openstack-infra14:13
*** dmsimard has quit IRC14:15
*** piet has joined #openstack-infra14:16
*** Hal has joined #openstack-infra14:16
*** Hal is now known as Guest1280614:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-enable repos for translation  https://review.openstack.org/27514014:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Cleanup translation scripts  https://review.openstack.org/27522514:17
AJaegerkrotscheck1: for a 32-bit x86 system, just run "linux32 uname -a" ;)14:18
AJaeger"Linux byrd 4.4.0-6.g9b438ee-default #1 SMP PREEMPT Mon Jan 25 17:01:54 UTC 2016 (9b438ee) i686 i686 i386 GNU/Linux"14:18
*** achanda has joined #openstack-infra14:18
*** zz_dimtruck is now known as dimtruck14:18
openstackgerritVictor Ryzhenkin proposed openstack-infra/project-config: Make murano tempest job voting and add it to gate pipeline  https://review.openstack.org/27519114:18
krotscheck1AJaeger: i386, thanks14:19
*** annegentle has quit IRC14:20
*** sgordon` is now known as sgordon14:21
pabelangeryolanda: do you mind helping approve 274900? Adds functional tests into the gate14:22
*** tiswanso has joined #openstack-infra14:22
*** _skolekonov is now known as skolekonov14:22
*** jsavak has joined #openstack-infra14:23
*** Guest12806 has quit IRC14:25
openstackgerritMerged openstack-infra/openstackid: Fix on username generation  https://review.openstack.org/27521514:25
*** peristeri has joined #openstack-infra14:26
*** mikelk has joined #openstack-infra14:26
SotKI'm looking into what is causing all the rogue storyboard-worker-daemon processes on storyboard.openstack.org, and I was wondering if anyone can perhaps please explain how `puppet apply` gets invoked to update storyboard?14:27
*** achanda has quit IRC14:27
* SotK finds http://docs.openstack.org/infra/system-config/puppet.html immediately after asking that14:28
*** piet has quit IRC14:29
*** dizquierdo has joined #openstack-infra14:30
*** dizquierdo_ has joined #openstack-infra14:30
*** dizquierdo_ has quit IRC14:30
*** dizquierdo has quit IRC14:30
*** jistr has joined #openstack-infra14:31
*** rossella_s has quit IRC14:31
*** dizquierdo has joined #openstack-infra14:31
*** rossella_s has joined #openstack-infra14:32
*** zul has quit IRC14:33
*** jsavak has quit IRC14:33
*** tiswanso has quit IRC14:33
*** tiswanso has joined #openstack-infra14:34
*** piet has joined #openstack-infra14:35
*** gnuoy_ has joined #openstack-infra14:36
*** gnuoy_ has quit IRC14:36
yolandapabelanger, done14:41
*** dimtruck is now known as zz_dimtruck14:41
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila / cinder no valid host bug 1540947  https://review.openstack.org/27523614:42
openstackbug 1540947 in Manila "gate-manila-tempest-dsvm-neutron fails randomly due to no valid host from cinder (insufficient free space)" [Undecided,Confirmed] https://launchpad.net/bugs/154094714:42
*** zz_dimtruck is now known as dimtruck14:42
*** mriedem has joined #openstack-infra14:43
*** pradk has joined #openstack-infra14:43
openstackgerritMerged openstack-infra/elastic-recheck: Add query for magnum API test timeout  https://review.openstack.org/27474414:45
*** keedya has joined #openstack-infra14:47
*** Daisy has joined #openstack-infra14:48
openstackgerritMerged openstack-infra/project-config: Add functional tests for windmill  https://review.openstack.org/27490014:48
pabelangeryolanda: danke14:48
*** sabeen1 has joined #openstack-infra14:50
*** annegentle has joined #openstack-infra14:50
*** jsavak has joined #openstack-infra14:51
*** _amrith_ is now known as amrith14:54
*** sabeen1 has quit IRC14:54
*** rbrndt has joined #openstack-infra14:55
*** dimtruck is now known as zz_dimtruck14:55
openstackgerrityolanda.robla proposed openstack-infra/puppet-lodgeit: fix puppet-lodgeit to support latest mysql  https://review.openstack.org/27524414:56
*** zul has joined #openstack-infra14:57
openstackgerrityolanda.robla proposed openstack-infra/puppet-etherpad_lite: fix module to work with latest mysql version  https://review.openstack.org/27524714:57
*** followinfrared is now known as infrared14:58
pabelangeryolanda: could I also trouble you for 274694 too14:58
*** infrared is now known as infra-red14:58
openstackgerritMerged openstack-infra/elastic-recheck: Add query for manila / cinder no valid host bug 1540947  https://review.openstack.org/27523614:59
openstackbug 1540947 in Manila "gate-manila-tempest-dsvm-neutron fails randomly due to no valid host from cinder (insufficient free space)" [Undecided,Confirmed] https://launchpad.net/bugs/154094714:59
yolandaah, for accessbot we need to be sure that there are no meetings going on in a wide period of time14:59
*** angdraug has joined #openstack-infra14:59
yolandathere are things going on now14:59
*** naggappan has quit IRC15:00
*** esker has joined #openstack-infra15:00
pabelangeryolanda: ah, good to know. Didn't know that. But make sense15:01
*** annegentle has quit IRC15:02
*** dmsimard has joined #openstack-infra15:02
*** bryan_att has joined #openstack-infra15:03
*** krtaylor has quit IRC15:03
*** vikram_ has quit IRC15:03
*** annegentle has joined #openstack-infra15:04
yolandai gave a +2 without approval15:05
AJaegeryolanda: you can do project-config changes AFAIK for accessbot - the problem is the system-config update for meetbot to enable eavesdropping logging support15:05
AJaegeryolanda: so 274694 is fine to merge15:06
yolandaah i mixed concepts there15:06
yolandaok approved then15:07
*** acruz has quit IRC15:08
*** piet has quit IRC15:08
*** Hal has joined #openstack-infra15:08
fungiyolanda: meetbot, not accessbot15:08
fungioh, AJaeger just said that (nevermind!)15:08
*** Hal is now known as Guest6738615:08
* fungi goes back to getting things done15:08
eantyshevclarkb, yolanda, anteaya: Hello! Can I get some attention to my fix for periodic pipelines in Zuul: https://review.openstack.org/#/c/272123/15:09
*** zul has quit IRC15:10
yolandafungi, need your knowledge on using curl with gerrit rest api. I am writing the doc for third party, and jeblair suggested to add full curl commands there. But i tried that, using auth digest, and i always get a 40115:10
*** sigmavirus24_awa is now known as sigmavirus2415:10
yolandahave you ever used that? all times that i used api , i have done it from browser, with some plugins to send requests15:10
fungiyolanda: using the "a/" prefix on your queries?15:10
fungithose are necessary for authenticated api calls15:11
yolandafungi i tried that15:11
*** rajinir has joined #openstack-infra15:11
*** links has quit IRC15:11
*** zul has joined #openstack-infra15:11
yolanda curl -v --digest --user --user yolanda.robla:<<my_pass>> https://review.openstack.org/a/projects for example15:12
yolandaand i get 40115:12
yolandai double checked pass and username, and is correct15:12
yolandaone --user flag, not two...15:12
fungicurl --digest --user yolanda:somepassword15:13
fungii think that's what i've done in the past15:13
yolandafungi, no , doesn't work15:13
*** hichihara has quit IRC15:13
*** rbrndt has quit IRC15:14
yolandapass should be sso pass right?15:14
*** ihrachys has quit IRC15:14
*** jed56 has joined #openstack-infra15:14
fungino, http api password15:14
yolandaah , so i shall configure it?15:14
yolandaok, i see now15:14
fungigenerate one at https://review.openstack.org/#/settings/http-password if you don't have one already15:14
yolandaah, i didn't know about htat15:15
*** krtaylor has joined #openstack-infra15:15
*** eharney has joined #openstack-infra15:15
yolandafungi, works now!15:15
yolandafirst time i used that http pass...15:15
fungialso be aware that gerrit prefixes the json with ~4 bytes of garbage you'll need to strip15:16
fungiif you don't trim it off, you won't be able to parse the response as valid json15:16
yolandait's just for writing documentation about how to disable third party ci accounts, and didn't want to write something without ensuring it will work15:16
fungioh, yep! good call15:16
fungithough you can also disable them through the ssh api15:16
fungi(gerrit set-account --inactive xxxxx)15:17
yolandawell yes, but the section i'm writing talks about both15:17
fungivery cool!15:17
yolandafungi, can you get account active details by ssh? haven't seen the command for it15:17
*** baoli has quit IRC15:17
*** ajmiller has joined #openstack-infra15:18
*** zz_dimtruck is now known as dimtruck15:18
*** zul has quit IRC15:18
*** ayoung has joined #openstack-infra15:20
*** dmsimard has quit IRC15:21
fungiyolanda: not that i'm aware, no. at least i don't see it listed at https://review.openstack.org/Documentation/cmd-index.html#_server so the rest api does provide a useful mechanism to validate your work15:22
yolandafungi yes, i was looking at the ssh equivalent for rest api command and i was surprised to don't find it. I thought both apis should be pretty equivalent15:23
*** jaosorior has joined #openstack-infra15:23
fungithe ssh api is missing at least a few informational methods15:24
*** amotoki has quit IRC15:24
fungibut i agree that's the one which surprises me the most too15:24
*** esikachev has quit IRC15:25
*** mtanino has joined #openstack-infra15:27
*** placko_ has joined #openstack-infra15:27
*** zul has joined #openstack-infra15:30
*** jpr has joined #openstack-infra15:36
*** zul has quit IRC15:36
krotscheck1fungi: Can I have an underscore in a node name?15:36
AJaegeryolanda: great!15:37
*** baoli has joined #openstack-infra15:38
*** jpr has quit IRC15:39
*** tiswanso has quit IRC15:39
*** jpr has joined #openstack-infra15:39
yolandafungi, also in the article jeblair points that i should show some command to retrieve the account numeric id, given the email or username. I haven't found any way in REST API to get that (i can only think about mysql) . Do you know if that's possible?15:39
*** piet has joined #openstack-infra15:39
anteayakrotscheck1: looks like hyphens or periods are preferred in node names: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=18115:39
AJaegerfungi, anteaya: Did you ever have see a followup from angdraug to the fuel plugin discussion from 2015-12-29? I commented on 269567 as well and wonder how to continue with that one...15:40
anteayayolanda: yes: curl -X GET 'https://review.openstack.org/accounts/pci-ci@intel.com'15:40
AJaegerfungi, anteaya : as well with 26020815:40
fungikrotscheck1: depends on what you mean by "node name" but if it's going into dns then no, you really shouldn't (it violates rfc 952/1123, and is clarified in 1912 particularly)15:40
krotscheck1fungi: Thanks15:41
anteayaAJaeger: I have not seen a followup, mind you I have been away for a week at nova mid-cycle and have lost my weechat server, so the possibility exists15:41
fungikrotscheck1: i'm guessing you mean "hostname"15:41
*** infra-red has quit IRC15:41
AJaegeranteaya: I haven't seen anything either, just wanted to double check15:41
*** tiswanso has joined #openstack-infra15:41
anteayaAJaeger: yup, good idea15:41
krotscheck1fungi: I'm trying to figure out if it's possible to have the hostname contain the same string output from uname -m,15:41
krotscheck1It's not critical.15:42
AJaegeranteaya: regarding voting jobs only in the check queue: Something to discuss in the team meeting today? Here's an example: https://review.openstack.org/#/c/274182/15:42
fungiyolanda: you can query accounts by e-mail address, username, full name, et cetera and one of the details returned is the account id15:42
fungi(in the rest api that is)15:42
angdraugAJaeger: still working on that followup15:42
yolandafungi, anteaya, thx15:42
AJaegerangdraug: then I'll wait with further reviews...15:43
anteayaAJaeger: that only leaves one neutron gate job15:43
fungikrotscheck1: probably needs to be some transformation of that (replace any characters not in the hostname regex with a hyphen?)15:43
* yolanda was trying incorrectly, with /a/accounts/email/detail15:43
anteayaAJaeger: I would like to hear what tempest reviewers have to say on that patch15:44
AJaegeranteaya: there're several templates with gate jobs...15:44
krotscheck1fungi: Gimme 20 minutes, I'll update my patches for you to doublecheck.15:44
anteayaAJaeger: I already voiced my opinion about communication drift in neutron in today's meeting so I'm not popular with neutron at the moment15:44
AJaegeranteaya: do you want to add them?15:44
*** tcammann_ has joined #openstack-infra15:44
*** pradk_ has joined #openstack-infra15:44
*** pradk has quit IRC15:44
*** pradk_ is now known as pradk15:44
anteayaAJaeger: I can comment on the patch that I would like to see tempest reviewers opinion on the patch15:45
anteayabut I'm not feeling obliged to track them down15:45
fungikrotscheck1: _technically_ the domain part can contain other characters, they're really only forbidden as part of the last (hostname) component as far as i know, but convention is to avoid them throughout the entire fqdn for records which reference hosts by name (there are other record types for which underscores are okay)15:45
*** sfinucan has joined #openstack-infra15:45
*** NobodyCa1 has joined #openstack-infra15:45
anteayaAJaeger: thanks for bringing that patch to my attention15:45
*** esp has joined #openstack-infra15:46
anteayaI now have to do battle with networks to see if I can get a server up, wish me luck15:46
AJaegeranteaya: as discussed last week ;)15:46
anteayaAJaeger: yup, thank you :)15:46
* AJaeger wishes anteaya luck to master the network daemons ;)15:46
anteayaI don't know about master, if I can get one server up I'll be happy15:47
*** baoli has quit IRC15:47
AJaegeranteaya: One step at a time to mastery ;)15:47
*** baoli has joined #openstack-infra15:48
anteayaha ha ha15:48
tristanCyolanda: thank you for the zuul poll timeout review!15:49
*** bhunter71 has joined #openstack-infra15:49
*** tcammann_ has quit IRC15:50
*** esp has quit IRC15:50
*** ilyashakhat has joined #openstack-infra15:50
*** NobodyCa1 has quit IRC15:51
krotscheck1When configure_mirror.sh is executed (nodepool), does it have access to jenkins/slave_scripts?15:51
* krotscheck1 is guessing not15:51
*** ihrachys has joined #openstack-infra15:51
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account  https://review.openstack.org/26535915:53
*** placko_ has quit IRC15:53
*** zeih has quit IRC15:53
*** _nadya_ has quit IRC15:54
*** dims has joined #openstack-infra15:54
clarkbkrotscheck1: it will beacuse we install all of the slave_scripts on all of our images15:54
*** dims_ has quit IRC15:54
clarkbkrotscheck1: however the path will be the installed path and not in the local dir15:55
*** jlanoux_ has quit IRC15:55
krotscheck1clarkb: Gotcha15:55
krotscheck1clarkb: Thanks15:55
clarkb/usr/local/jenkins/scripts or something, can double check with puppet to find the rooted path15:56
*** ihrachys has quit IRC15:58
*** maishsk has quit IRC15:58
*** devananda has joined #openstack-infra16:00
openstackgerritAl Miller proposed openstack/requirements: Add neutron-lbaas-dashboard to projects.txt  https://review.openstack.org/27529116:01
*** david-lyle has joined #openstack-infra16:02
*** tphummel has joined #openstack-infra16:02
*** samuelBartel has quit IRC16:02
*** jsavak has quit IRC16:02
*** jsavak has joined #openstack-infra16:03
*** yamamoto has quit IRC16:05
AJaegeranteaya, yolanda, clarkb: Could either of you review the stack starting with https://review.openstack.org/#/c/274832 on translation changes, please?16:06
*** rbrndt has joined #openstack-infra16:07
AJaegerwhenever it's convenient for you - anteaya, get your network server up first ;)16:07
*** yamamoto has joined #openstack-infra16:07
*** kzaitsev_mb has quit IRC16:08
*** yamamoto has quit IRC16:08
yolandaAJaeger i cannot now but i'll take a look as soon as i can16:09
anteayaAJaeger: thanks16:09
anteayabtw Sam-I-Am is magic16:09
anteayajust pure magic16:09
*** sfinucan has quit IRC16:09
AJaegerSam-I-Am: what did he do? Yeah, he is...16:10
*** sfinucan has joined #openstack-infra16:10
*** pigmej has joined #openstack-infra16:10
pigmejyolanda: hey, could you please look at https://review.openstack.org/#/c/267453/ ? It just needs one more +16:11
AJaegeryolanda: no worries, thanks16:11
jeblairyolanda: just GET /a/accounts/emc.isilon.ci@emc.com16:12
jeblairyolanda: returns {"_account_id":15942,"name":"EMC Isilon CI","email":"emc.isilon.ci@emc.com","username":"e"}16:12
*** esikachev has joined #openstack-infra16:12
* AJaeger will be back later...16:12
yolandajeblair yes, i know now... i was misunderstandng the documentation and was trying with /a/accounts/email/detail16:13
yolandaand was failing16:13
*** gokrokve has joined #openstack-infra16:13
*** annegentle has quit IRC16:13
*** vgridnev has quit IRC16:14
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711716:15
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354916:15
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492716:15
*** mattt has left #openstack-infra16:17
*** Daisy has quit IRC16:17
*** gongysh has quit IRC16:18
openstackgerritJames E. Blair proposed openstack-infra/system-config: Temporarily stop releasing after bandersnatch  https://review.openstack.org/27529916:18
openstackgerritJames E. Blair proposed openstack-infra/system-config: Revert "Temporarily stop releasing after bandersnatch"  https://review.openstack.org/27530016:18
openstackgerritJames E. Blair proposed openstack-infra/system-config: Revert "Temporarily switch to AFS RW mirrors"  https://review.openstack.org/27530116:18
jeblairfungi: i'd like to go aheand and land  https://review.openstack.org/275299 for safety16:19
*** Daisy has joined #openstack-infra16:19
fungijeblair: lgtm16:19
fungii agree that's a good idea for now16:19
*** samuelBartel has joined #openstack-infra16:20
fungialso didn't get far on the mirror reviews last night but trying to squeeze some time in between my current meeting and the team meeting to get through them16:20
jeblairlast night i cleaned up the mess i made yesterday and started a new vos release in k5start+screen. i'm expecting that to finish wed night16:20
krotscheck1Ok, I _think_ that last project_config patch chain will work as advertised.16:20
krotscheck1Also, added bonus, generic afs-release builder.16:20
*** sdake has joined #openstack-infra16:21
*** esikachev has quit IRC16:22
krotscheck1jeblair, fungi: Should I WIP the wheel mirror patches until AFS things settle down?16:22
jeblairkrotscheck1: nope16:22
jeblairkrotscheck1: none of the volume replication stuff we're working through on the pypi mirrors should affect the wheel mirror work16:22
*** abregman has quit IRC16:23
krotscheck1jeblair: Different volume?16:23
openstackgerritice4o@hotmail.com proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support.  https://review.openstack.org/27530416:24
*** bhunter71 has quit IRC16:24
jeblairi don't understand the difference between the "packages" and "user_packages" variables in modules/openstack_project/manifests/users_install.pp16:24
krotscheck1Alright then, I'll pester y'all when all the builds pass.16:24
*** ilyashakhat has quit IRC16:24
jeblairwe only install user_packages if we install users16:25
jeblairthis is strange, but oh well.16:25
*** ilyashakhat has joined #openstack-infra16:26
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add iftop to all Ubuntu servers  https://review.openstack.org/27530816:26
*** infra-red has joined #openstack-infra16:26
*** woodster_ has quit IRC16:26
jeblairhow have we lived without that ^ ?16:26
*** ilyashakhat has quit IRC16:27
jeblairiftop -f "not port domain" -P16:27
jeblairis my preference16:27
*** jlanoux has joined #openstack-infra16:29
*** fesp has joined #openstack-infra16:29
*** mdenny has quit IRC16:29
*** ddecapit has joined #openstack-infra16:30
*** ddecapit is now known as DuaneDeC716:30
*** jpr has quit IRC16:31
*** jpr has joined #openstack-infra16:31
fungiit's like openbsd's pftop16:32
*** amaretskiy has quit IRC16:32
fungiwhich i'd regularly found myself desiring an equivalent of for linux16:32
clarkbjeblair: greghaynes we need a release of gear for that double disconnect thing right?16:33
fungithough i guess it's somewhere between pftop and ntop feature wise16:33
jeblairi just ran it on the afs fileservers, and noticed that one of them the bar graph goes to 954Mb, but the other only goes to 95.4Mb.  i wonder how it decides how long it should be...16:33
* clarkb is trying to make a list of nodepool imge builder related items that should be done in the near future16:33
jeblairclarkb: yeah, but that's just a test code cleanup basically16:34
EmilienMwhy http://git.openstack.org/cgit/openstack-infra/ansible-puppet/ does not use http://docs.ansible.com/ansible/puppet_module.html ? Is there any plan to migrate to the ansible module?16:34
jeblairclarkb: i'll go ahead and release it16:34
jeblairEmilienM: they're the same code16:34
clarkbEmilienM: I think because ansible-puppet is the source for that puppet module and we just haven't switched over yet16:35
clarkbjeblair: thank you16:35
jeblairEmilienM: we'll probably move to it eventually, but we still keep our own copy so it's easy for us to CD it without CDing ansible16:35
EmilienMok no problem, I was just asking :-)16:35
jeblairnp, good question16:35
EmilienMI was actually for that kind of tool and I found it :)16:36
*** ilyashakhat has joined #openstack-infra16:36
*** DuaneDeC7 has quit IRC16:37
*** ilyashakhat has quit IRC16:38
*** ilyashakhat has joined #openstack-infra16:38
clarkbour last image builds were from saturday, I think we must still be having errors which cause the nodepoold process to think builds are in progress when the builder has nothing queued16:38
*** infra-red has quit IRC16:38
clarkbtime for me to go review the changes that should address disconnects16:38
*** infra-red has joined #openstack-infra16:39
clarkbinfra-root https://review.openstack.org/#/c/274821/1  should hopefully make multinode more reliable for us16:39
*** gabor_antal_ has joined #openstack-infra16:39
jeblairclarkb: yes, the disconnect stuff is still a problem16:39
freerunnerHi jeblair fungi clarkb AJaeger and project-config folks! Could you please review it https://review.openstack.org/#/c/275191/2 ?16:40
fungiclarkb: last pass on 274821 failed the multinode-full job?16:41
*** mgkwill_training is now known as mgkwill16:41
jeblairfreerunner: bare review requests like that aren't helpful16:41
clarkbjeblair: it is worth noting that on internap we now have a decent number of multinode passes. I think updating the security groups there may have been the ticket16:41
*** infra-re_ has joined #openstack-infra16:41
clarkbfungi: it did, but not due to networking from what I can tell. Just general performance issues16:41
clarkbfungi: it ran on bluebox too, I think the broken networking may be masking another issue there16:42
*** esker has quit IRC16:42
jeblairclarkb: woot!  mgagne ^16:42
*** fesp has quit IRC16:42
mgagneclarkb :D16:42
gabor_antal_Hello there, I'd like to commit into gerrit. Can someone help me with some docs, or anything like that?16:43
jeblairclarkb: should we ask bluebox folks why our gre packets might be getting dropped?  maybe they can check on whether conntrack_gre is present or needed?16:43
mgagnejeblair you can get a smiley face too :D16:43
freerunnerjeblair Oh. So, this patch will replace our (murano) not-tempest gate job with a tempest one.16:43
*** bpokorny has joined #openstack-infra16:43
clarkbjeblair: that would be great16:43
*** ianw has quit IRC16:43
clarkbjeblair: but I think vxlan is a viable workaround if we want to punt16:44
mgagnejeblair any instance UUID I can check?16:44
fungigabor_antal_: you mean specifically for the openstack community's gerrit code review instance? or something else?16:44
mgagnejeblair want to check host for conntrack helper presence of not16:44
mgagneor not*16:44
*** infra-red has quit IRC16:44
clarkbmgagne: I think internap may be working now. I will keep an eye on it though16:44
clarkbmgagne: just bluebox seems to have trouble now16:45
mgagneany instance you know of I can check?16:45
clarkbya let me dig up the uuid for this one I have a fail log on16:45
mgagneclarkb we haven't officially loaded the gre conntrack helper on our hosts16:45
*** bpokorny has quit IRC16:45
fungigabor_antal_: presumably you're looking for http://docs.openstack.org/infra/manual/developers.html16:46
mgagnefail at internap or bluebox? I don't control bluebox host =)16:46
*** sc68cal has joined #openstack-infra16:46
clarkbmgagne: internap16:46
*** bpokorny has joined #openstack-infra16:46
mgagneok, some are failing and others not?16:46
openstackgerritMerged openstack-infra/tripleo-ci: Deal with a blank ZUUL_CHANGES later  https://review.openstack.org/27155916:46
*** gibi_ is now known as gibi16:47
clarkbmgagne: 614a9590-9efe-44de-b398-a99bf6d6d4f6 and 2a631ff9-8e50-41d4-9059-fa598f3cef55 were the internap pair. Yes we now have a 50% pass rate over the last 10 days but those two ran a job recently and it failed to ssh over the overlay network16:47
*** dims_ has joined #openstack-infra16:47
mgagneI would like to know a pair that worked too to compare16:47
clarkbmgagne: the fail rate was higher before I opened up the security group with unspecified protocol types16:47
clarkbmgagne: I can dig that up too if I have more time16:48
gabor_antal_fungi: I also got that documents, any of the infra module's are the same as other OpenStack modules?16:48
mgagnecould fail be unrelated to provider? like race or whatever? trying to see if I need to dig more in hosts or not.16:48
jeblairgreghaynes, clarkb: gear 0.5.9 pushed16:48
*** jaosorior has quit IRC16:49
*** Swami has joined #openstack-infra16:49
*** dims has quit IRC16:49
fungigabor_antal_: i don't understand the context of your question. i thought you were asking if someone could help you find documentation on submitting changes to our code review system. can you be more specific about what you're looking for help with?16:49
*** DuaneDeC7 has joined #openstack-infra16:50
clarkbhrm not so easy to find the stuff in the logs that the health dashboard claims succeeded16:51
clarkbI wonder if the health dashboard is buggy, I see that the same job is recordred twice, once as success and once as failure16:51
clarkbwhcih would explain the roughtly 1:1 ratio16:51
clarkbmgagne: looking at http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj01?groupKey=node_provider&resolutionKey=hour fwiw16:51
mgagneI had difficulty finding actual jobs associated to a provider16:51
clarkbso maybe I need to do more digging there16:51
mgagneok, what I know for sure is that gre conntrack helper isn't loaded on the hosts the jobs ran16:52
*** armax has joined #openstack-infra16:52
*** infra-re_ has quit IRC16:52
clarkbmgagne: ok, from your perspective would it be better if we used vxlan?16:52
clarkbbecause that is the other alternative I have been working16:53
mgagneclarkb does vxlan require a conntrack helper?16:53
openstackgerritVincent Françoise proposed openstack-infra/project-config: Watcher Dashboard project creation  https://review.openstack.org/27516716:53
*** bhunter71 has joined #openstack-infra16:53
*** bnemec has quit IRC16:53
mgagneclarkb just trying to see if gre/vxlan requires collaboration from providers or not16:53
*** jistr has quit IRC16:53
*** rhallisey has quit IRC16:53
mgagnehaven't tried any of them, well, tried vxlan on grizzly install 1-2 years ago16:53
clarkbmgagne: I don't think so, it is udp and the endpoints should manage their own state16:54
mgagneour netadmin was flabbergasted that I didn't need his help to setup16:54
*** reed_ has joined #openstack-infra16:55
clarkbmgagne: we aren't using neutron fwiw. Its just an ovs bridge connected to another ovs brdige via gre/vxlan16:55
mgagneI don't have a strong opinion. whatever works without involving providers is fine tbh16:55
*** e0ne has quit IRC16:55
clarkbthe big downside to vxlan is that it will use multicast for broadcast among members but almost nothing supports multicast16:55
clarkbwhich makes arp less fun aiui but it should mostly work in these small setups16:56
*** annegentle has joined #openstack-infra16:56
*** achanda has joined #openstack-infra16:57
*** matrohon has quit IRC16:57
fungiyeah, i'm honestly surprised that multicast groups are working where something as simple as a less common ip protocol is being filtered16:57
mgagneI have yet to fully understand the underlying details =)16:57
*** scheuran has quit IRC16:57
clarkbfungi: well mutlicast doesn't work, it fails back to unicast iirc16:57
fungiclarkb: or are you saying that multicast doesn't come into play because we're using it for a point-to-point tunnel anyway?16:57
fungithat, got it16:57
gabor_antal_fungi: I'd like to ask that is there any special things with the infra modules? Or is it the same as anywhere else16:58
sc68calclarkb: it can use multicast for broadcast, otherwise it'll just flood all the nodes16:58
sc68calsadly I don't know how well it is tested, that implies hardware that can do IGMP16:58
fungigabor_antal_: code contribution process for all official openstack projects is consistent, including the infra projects16:58
*** infra-red has joined #openstack-infra16:58
jeblairkrotscheck1: would you mind trying to avoid rebasing so frequently?16:58
*** mikelk has quit IRC16:58
jeblairkrotscheck1: it's basically impossible for me to tell whether anything was changed between PS 26 and PS 27 of https://review.openstack.org/16492716:59
gabor_antal_fungi: okay, thanks16:59
fungigabor_antal_: also you had mentioned in a private irc message you were wanting to assist with code quality in openstack. there is actually a quality assurance team, if you haven't talked to them already. they tend to hang out in the #openstack-qa channel16:59
*** baoli has quit IRC16:59
*** Guest67386 has quit IRC16:59
openstackgerritIgor Belikov proposed openstack-infra/system-config: Get debian kernel headers based on architecture  https://review.openstack.org/27532316:59
gabor_antal_fungi: thanks, we've already talked with them17:00
*** kzaitsev_mb has joined #openstack-infra17:00
*** vivekd has quit IRC17:00
fungivery cool. welcome to the community!17:00
jeblairkrotscheck1: basically it's most helpful to only rebase if you have to because there is a conflict.17:00
*** baoli has joined #openstack-infra17:00
*** kzaitsev_mb has quit IRC17:01
*** kzaitsev_mb has joined #openstack-infra17:01
*** kzaitsev_mb has quit IRC17:01
*** kzaitsev_mb has joined #openstack-infra17:01
*** samuelBartel has quit IRC17:01
*** ashleighfarnham has joined #openstack-infra17:01
*** kzaitsev_mb has quit IRC17:02
*** kzaitsev_mb has joined #openstack-infra17:02
*** vivekd has joined #openstack-infra17:02
krotscheck1jeblair: I'll try, but I may be too ocd to feel comfortable with that.17:02
fungikrotscheck1: if it helps, you can be *even more( ocd about trying to rebase onto conflicting changes only17:03
clarkbjeblair: greghaynes https://review.openstack.org/#/c/272384/ depends on the new gear release. Is that a strict dependency and we need to restart the geard before updating nodepool?17:03
fungii get pretty ocd about introducing the smallest divergence possible in updating my changes17:03
*** jsavak has quit IRC17:04
*** jsavak has joined #openstack-infra17:04
jeblairkrotscheck1: what fungi said -- and seriously -- as a reviewer, when i work through changes it's important to see how they evolve over time.  when there is a rebase + change in a single update, the entire flow is lost and there is no narrative connection with the previous patchsets17:04
openstackgerritMerged openstack-infra/git-restack: Initial commit of git-restack  https://review.openstack.org/25964717:04
*** ilyashakhat has quit IRC17:04
jeblairkrotscheck1: so the most ideal thing is updates only without rebasing; then if something has a conflict, a rebase only (ideally with a comment saying so), and then if an update is needed, a separate patchset to do that17:05
jeblairkrotscheck1: i'm working on making the git-restack tool to make that a bit easier for long-running patch series17:05
fungiwhen the rebase is ugly enough, i've even gone so far as to do back-to-back patchsets to first perform the rebase and then make the new modifications i needed17:05
fungi(review comments pointing out the rebase patchset are also very helpful for that)17:05
*** achanda has quit IRC17:06
*** ashleighfarnham has quit IRC17:06
greghaynesclarkb: it depends on the client only17:07
fungizaro: not sure if you saw my discussion with madorn yesterday, but curious if you had recommendations for how best to clean up the faux "registered users" groups file entries in our gerrit once manage-projects is no longer creating them17:07
greghaynesclarkb: that test fail17:07
greghaynesclarkb: is wierd17:07
clarkbgreghaynes: cool so that will get updated when we update nodepool itself17:07
jeblairkrotscheck1: PS27 in 164927 is in fact a rebase + a significant change17:07
jeblairkrotscheck1: so there's no good way for me to review that17:07
jeblairkrotscheck1: would you like to restack those without the rebases, or shall i?17:07
fungizaro: especially with regards to whether there is a way to do it live without taking gerrit offline17:07
clarkbjeblair: git review has a thing that works when there aren't conflicts17:07
*** ashleighfarnham has joined #openstack-infra17:08
zarofungi: just got myself back in channel today after the HP cloud going offline17:08
clarkb-m ?17:08
jeblairclarkb: how do i leave comments with git review?17:08
fungizaro: or if it's going to be easier to do that cleanup in a maintenance window17:08
openstackgerritVincent Françoise proposed openstack-infra/project-config: Watcher Dashboard project creation  https://review.openstack.org/27516717:08
* zaro looks at yesterday's backscroll17:08
clarkbjeblair: you don't, but it gives you the ability to review the delta in an easier to understand diff17:08
anteayazaro: Sam-I-Am just helped me get a server up, I'm working on instructions to share17:09
clarkband the line numbers should be static for the "new" side17:09
fungizaro: discussion started around 23:47 utc17:09
krotscheck1jeblair: restack on... what, the previous commit?17:09
dhellmannfungi : do you think you'll have time this week to look at setting up releases.openstack.org? the patch series is https://review.openstack.org/#/q/topic:releases-openstack-org but I'm sure there are some steps to handle outside of that (DNS?)17:09
clarkbkrotscheck1: my trick is to git rebase -i HEAD~X17:09
jeblairkrotscheck1: yeah, PS26 was based on 8ef6e559e802f9e8969be9f3d7d4ff5261b2bcb6; so we need to cherry-pick cd5fe79762a7251420a33e64b1aecf3c37ff6ec9 (PS27) on top of that17:10
fungidhellmann: i think adding a couple cname entries in dns is all we need if it's being hosted from static.openstack.org like i remember17:10
fungidhellmann: if so, i can add those now--there's no need to wait and it only takes a moment17:10
dhellmannfungi : is that something I can do with a patch, or is that a manual step?17:11
dhellmannfungi : ah, cool17:11
krotscheck1Doesn't git review do rewinds anyway?17:11
jeblairkrotscheck1: nope17:11
* krotscheck1 can try17:11
fungidhellmann: manual process since rackspace doesn't use openstack to host dns17:11
*** fesp has joined #openstack-infra17:11
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711717:11
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354917:11
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492717:11
dhellmannfungi : ack17:11
*** fesp has quit IRC17:12
openstackgerritIgor Belikov proposed openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1'  https://review.openstack.org/27532917:12
fungidhellmann: though if we also need https then i'll have to get a cert issued, which means i may need to step up priority for looking into switching the cert authority i've been using17:12
*** sridhar_ram has joined #openstack-infra17:12
jeblairkrotscheck1: perfect, thanks!17:12
dhellmannfungi : I hadn't thought about https explicitly, but it makes sense. ttx, thoughts?17:12
fungidhellmann: we can always add it a little later if it wasn't an up-front requirement, but i think it should be added soonish17:13
fungii don't want to hold up the deployment for that though17:13
dhellmannfungi : I agree we should do it, and I'm OK with not blocking on it if adding it later isn't a big deal17:13
zaroanteaya: ugh! that hp cloud down was a big headache for me17:13
*** diana_clarke has joined #openstack-infra17:13
*** dtantsur is now known as dtantsur|afk17:13
anteayazaro: yup17:14
fungidhellmann: yeah, i already set up https for other important sources of information (governance, tarballs, security) and i think releases falls into a similar category17:14
jeblairgit-restack 1.0.0 pushed17:14
dhellmannfungi : ++17:14
*** salv-orlando has quit IRC17:14
*** ifarkas has quit IRC17:14
fungidhellmann: so anyway, if the change series is close to being landable i can add whatever else we need to my pile for this week17:15
*** sdake has quit IRC17:15
*** tphummel has quit IRC17:16
*** baoli has quit IRC17:16
dhellmannfungi : the system-config and project-config patches have one +2 each; the doc change to add the redirect is blocked waiting for those to land (makes sense); the work in the releases repo to reorganize the pages is done17:16
*** Daisy has quit IRC17:16
dhellmannfungi : so I think we're ready to go when you have time to do it17:16
clarkbgreghaynes: looking at that failure it looks like dib was just slow17:16
greghaynesclarkb: yep, I rechecked and left a comment17:17
pabelangeryolanda: fungi: I incorrectly rechecked 274694, it had a +1 workflow on it, but I think my recheck stopped the merge.  Do either of you mind checking it out?17:17
fungidhellmann: awesome! it may be in your best interests to prod me about it again tomorrow since today is meetingsmeetingsmeetings17:17
greghaynesclarkb: those gearman logs in console.log are odd17:17
greghaynesclarkb: but I think it is just a red herring17:17
dhellmannfungi : I'll schedule a reminder ;-)17:17
fungipabelanger: recheck shouldn't stop anything already being tested17:17
clarkbthough the image log cuts off before the job finishes17:18
pabelangerclarkb: greghaynes: looking forward to afs mirrors for yum and apt. That should help speed up somethings17:18
clarkbbut it was going slowly before that so could be IO issues17:18
*** dalgaaf has quit IRC17:18
greghaynesclarkb: Yep, I see it died while dib was downloading packages17:18
pabelangerfungi: Ya, I thought that too, but seems stuck on +1 workflow17:18
*** zul has joined #openstack-infra17:18
fungipabelanger: zuul never left a jenkins comment about enqueuing it17:18
fungipabelanger: we may have something else going on. i'll check the zuul logs17:19
pabelangerfungi: That is right, forgot about that17:19
chaituIam trying setup fresh ci setup i could not shh to the instance created in openstack env and here is mynodepool log file http://paste.openstack.org/show/485758/17:19
chaitucan some one help me out with it17:19
*** gabor_antal_ has quit IRC17:20
pabelangerchaitu: are you using the right ssh public / private keys?17:20
zarofungi, madorn : i don't think removing a group can be done live.  it will require downtime because the only way to remove groups is to remove it directly from db.  there's no gerrit feature to do that.  here the script to remove groups: https://review.openstack.org/#/c/270537/17:20
pabelangerfungi: just did now17:20
pabelangerfungi: zuul left the message17:21
fungizaro: oh, it's also in the db tables? not just an entry in the groups files in the meta/config ref of each repo?17:21
pabelangeryolanda: redid her workflow17:21
yolandapabelanger, yes, i redid before fungi told that17:21
fungizaro: i mean, i know in the normal case there is a database entry for created groups, just wondering about in this special case of it having the same name as a system group, whether the database entries actually got created17:21
jeblairkrotscheck1: i think https://review.openstack.org/164927 looks good but i left comments on two things where i think the documentation is wrong -- my guess is they are harmless and left over from a previous iteration where you were generating the short name automatically rather than us generating it manually by convention.17:22
ttxdhellmann: https is better, but we should not block on it, since we don't have it currently17:22
krotscheck1jeblair: Cool, time for a rebase! ;)17:22
*** baoli has joined #openstack-infra17:22
jeblairkrotscheck1: let me know if that's correct.  i think, on balance, it's probably worth one more ps to correct them.17:22
dhellmannttx: that's what fungi and I agreed, so we'll add that in the near future17:22
jeblairkrotscheck1: :|17:22
chaitupabelanger: yes iam using the right keys17:23
jeblairgit-restack is on pypi17:23
*** reed_ has quit IRC17:24
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711717:24
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354917:24
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492717:24
yolandapabelanger, i can see gate jobs starting now17:25
zarofungi: it's actually created as a normal group.17:25
*** reed_ has joined #openstack-infra17:25
pabelangeryolanda: fungi: odd how zuul never started before17:25
*** yamahata has joined #openstack-infra17:25
zarofungi: they call it "internal group"17:25
pabelangerchaitu: not sure, nodepool cannot connent. So either your keys are wrong, username is wrong, or user does not exist17:25
fungipabelanger: yolanda: i don't see any mention of a comment-added event for 274694,2 at 15:07 utc when the original workflow +1 was recorded17:25
pabelangerfungi: yolanda: network blip with gerrit?17:26
*** salv-orlando has joined #openstack-infra17:26
fungitrying to figure that out now17:26
*** annegentle has quit IRC17:26
yolandalooks as it didn't pick the event17:26
*** gabor_antal has joined #openstack-infra17:26
*** mdenny has joined #openstack-infra17:26
*** Daisy has joined #openstack-infra17:26
*** maishsk has joined #openstack-infra17:27
openstackgerritMerged openstack-infra/project-config: Add IRC notification for windmill  https://review.openstack.org/27469417:28
fungithere is a traceback at 15:07:12 about a dependency loop, but it's between 202459 and 20244017:28
fungiso unless as a side effect it dropped the rest of its buffer or something, i don't think that was related17:28
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool  https://review.openstack.org/26472617:28
*** alivigni has quit IRC17:28
*** infra-re_ has joined #openstack-infra17:29
*** infra-r__ has joined #openstack-infra17:29
fungiyeah, i'm not finding anything to explain the missing event17:30
fungii guess it's possible gerrit never emitted one--i'll see if there's an obvious needle in its log haystack17:31
*** infra-red has quit IRC17:31
*** Daisy has quit IRC17:31
*** reed_ has quit IRC17:31
*** openstackgerrit has quit IRC17:32
*** openstackgerrit has joined #openstack-infra17:32
*** infra-re_ has quit IRC17:33
*** Jeffrey4l has quit IRC17:33
jeblairkrotscheck1: so i should create mirror.wheel.trustyx64 and mount it at /afs/openstack.org/mirror/wheel/ubuntu-14.04-x86_64  correct?17:33
*** maishsk has quit IRC17:34
*** dalgaaf has joined #openstack-infra17:34
*** Apoorva has joined #openstack-infra17:35
*** infra-r__ has quit IRC17:37
clarkbif you scroll through a diff in new gerrit ui it expands the context automagically which is :(17:37
*** Daisy has joined #openstack-infra17:37
clarkb* scroll through with keyboard17:37
fungipabelanger: yolanda: nothing useful around that time in gerrit's error log either17:38
*** tphummel has joined #openstack-infra17:38
pabelangerfungi: yolanda: I wonder of the +2 approve followed up with the +1 workflow on the same user might have something to do with it.17:39
krotscheck1jeblair: Yes17:39
pabelangereither way, I cannot think of a reason for the event to be missing17:39
pabelangernever seen that before17:39
igorbelikovpabelanger: hi, regarding https://review.openstack.org/#/c/264726/ - debian-minimal is still busted in latest diskimage-builder release, so Is it ok to use non-minimal image for now? My patch to fix it was merged to master17:40
*** annegentle has joined #openstack-infra17:40
fungipabelanger: that pattern has worked fine in the past17:41
*** Daisy has quit IRC17:41
*** unicell has quit IRC17:41
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool  https://review.openstack.org/26472617:42
clarkbgreghaynes: commented on https://review.openstack.org/#/c/272384/7 but I think it can be addressed in afollow up17:42
clarkbgreghaynes: and the existing child fails pep8 and tests17:42
greghaynesclarkb: Yep, I have a tox going right now...17:43
jeblairclarkb, fungi: can you take a sec to look at this release announcement for git-restack?  https://etherpad.openstack.org/p/iriJVixL9z17:43
pabelangerigorbelikov: Ya, I think we want -minimal for all images, mordred will know more.  Keep in mind, it took fedora 23 a few months to land because of a series of issue around -minimal. So, you'll likely need to get that working correctly before jessie can land.  We already do ubuntu-minimal, so it shouldn't be to much work for debian-minimal17:45
*** SumitNaiksatam has quit IRC17:46
pabelangerigorbelikov: if you fixed it in diskimage-builder, you can add a Depend-on header to the commit. We run from git I believe with diskimage-builder17:46
clarkbjeblair: seems fine17:47
clarkbexplains what it is used for and how i tworks17:47
igorbelikovpabelanger: ok, I'll update it with minimal in a couple of minutes, thanks17:47
*** esikachev has joined #openstack-infra17:47
*** sdake has joined #openstack-infra17:48
pabelangerigorbelikov: nice. Like I said, do you mind posting the output online some place.  Won't mind looking it over see what is different between fedora and debian.17:48
zarofungi, madorn : also the upstream gerrit fix has not been merged yet,  i just asked DavidP to take a another look.  https://gerrit-review.googlesource.com/#/c/74312/17:48
igorbelikovpabelanger: yeah, will do. I'll start building minimal one now, will update request with link to the logs17:49
fungijeblair: lgtm17:49
*** smarcet has joined #openstack-infra17:49
pabelangerigorbelikov: thanks17:50
smarcetfungi: afternoon17:50
fungijeblair: a great explanation of why people should want to use it17:50
smarcetfungi: there is any way to fix an already commited tag message on infra github ?17:50
fungismarcet: hey there. jimmy mentioned earlier that you may need help with something time-sensitive today17:50
*** sdake has quit IRC17:50
*** salv-orl_ has joined #openstack-infra17:50
smarcetfungi: i tried but it reject it17:50
fungismarcet: the message is part of the tag. for a new message you need a new tag17:50
smarcetoh ok , well not big deal17:51
smarcetyes we have a new version of openstackid17:51
*** krotscheck1 has quit IRC17:51
smarcetready to go to production (Openid connect)17:51
*** hdd has joined #openstack-infra17:51
clarkbmordred: /etc/openstack/clouds.yaml is 0600 root, what do you think about opening that up to a common infra-root group?17:52
fungibasically git combines the tag name, tag message and the object reference being tagged, then creates a hash of that data and the result is what you cryptographically sign17:52
clarkbmgagne: sorry that is on puppetmaster17:52
smarcetfungi: ok got it, thx for info :)17:52
clarkbgah mordred ^ mgagne I apologize and you may ignore17:52
smarcetfungi: basically to deploy openstackid to production, from what i understand, i need to update tag vesion here modules/openstack_project/manifests/openstackid_prod.pp17:53
smarcetthat would be enough to fire the deployment process17:53
*** rhallisey has joined #openstack-infra17:53
*** infra-red has joined #openstack-infra17:53
*** salv-orlando has quit IRC17:54
fungismarcet: yes, i believe you change it to $release = '1.0.7',17:54
fungistray newline there17:54
*** sfinucan has quit IRC17:54
smarcetfungi : cool17:54
fungichange it to $release = 'your_new_tag',17:54
jeblair11 new messages arrived on openstack-dev in the time it took me to write that one17:55
clarkbmordred: also `venv/bin/openstack --help` fails if it can't read the clouds.yaml17:55
clarkbI will file a bug for that17:55
smarcetfungi: roger that for info :)17:55
*** zul has quit IRC17:56
*** infra-red has quit IRC17:58
clarkbhttps://bugs.launchpad.net/python-openstackclient/+bug/1541047 has been filed17:58
openstackLaunchpad bug 1541047 in python-openstackclient "openstack --help fails if clouds.yaml cannot be read" [Undecided,New]17:58
clarkbgreghaynes: you making a dib release today?18:00
greghaynesclarkb: yes! ty for reminding me18:00
greghaynesclarkb: after this phone call...18:00
*** yamahata has quit IRC18:00
jeblairkrotscheck1's irc bouncer situtation is not optimal18:02
*** rossella_s has quit IRC18:02
jeblairfungi, greghaynes: any idea how much space a single-arch wheel mirror built from our requirements ought to be?18:03
*** sigmavirus24 is now known as sigmavirus24_awa18:03
*** sridhar_ram1 has joined #openstack-infra18:03
*** sridhar_ram has quit IRC18:03
*** simondodsley has joined #openstack-infra18:03
*** rossella_s has joined #openstack-infra18:04
*** diana_clarke has quit IRC18:04
*** annegentle has quit IRC18:05
*** rhallisey has quit IRC18:05
fungii really don't recall what it used to look like size wise. i wonder if we can recheck a requirements change, hold the node and then look at the resulting wheel cache18:06
*** tzn has quit IRC18:06
jeblairmeh, i'll just set it to 20G for now and we can watch the wheel build job18:06
clarkbfungi: that would be very close yes (remember that we leave old wheels in place so it will grow over time)18:06
clarkb20GB should be plenty to start18:06
fungigood point. we definitely need room to grow18:06
*** rfolco has quit IRC18:07
*** yaume_ has quit IRC18:07
*** davideagnello has joined #openstack-infra18:09
*** jsavak has quit IRC18:09
*** yamahata has joined #openstack-infra18:09
*** jtomasek__ has quit IRC18:10
*** jsavak has joined #openstack-infra18:10
*** rbrndt has quit IRC18:11
*** anteaya has quit IRC18:11
*** ybathia has joined #openstack-infra18:11
*** anteaya has joined #openstack-infra18:12
*** jordanP has quit IRC18:12
jeblair/afs/openstack.org/mirror/wheel/ubuntu-14.04-x86_64 exists18:13
jeblairand is replicated on both fileservers :)18:13
*** Swami has quit IRC18:14
* fungi cheers18:14
*** annegentle has joined #openstack-infra18:14
*** gibi has quit IRC18:14
*** kzaitsev_mb has quit IRC18:15
*** annegentle has quit IRC18:15
*** rfolco has joined #openstack-infra18:15
*** ianw has joined #openstack-infra18:15
*** gibi has joined #openstack-infra18:15
*** pvaneck has joined #openstack-infra18:16
*** jcoufal has quit IRC18:16
*** unicell has joined #openstack-infra18:16
*** SumitNaiksatam has joined #openstack-infra18:17
openstackgerritShamail Tahir proposed openstack-infra/system-config: Adding meetbot and statusbot support for openstack-product.  https://review.openstack.org/27536418:18
openstackgerritMerged openstack-infra/system-config: Temporarily stop releasing after bandersnatch  https://review.openstack.org/27529918:18
openstackgerritShamail Tahir proposed openstack-infra/project-config: Adding gerritbot and accessbot support for openstack-product.  https://review.openstack.org/27536518:19
*** achanda has joined #openstack-infra18:19
anteayathe meeting topic in -alt needs a refresh if anyone has a moment18:19
*** alivigni has joined #openstack-infra18:20
*** annegentle has joined #openstack-infra18:20
anteayanow that I have the ability to work again, I have some backscroll to read18:20
*** dimtruck is now known as zz_dimtruck18:21
*** yamahata has quit IRC18:22
fungianteaya: thanks--done!18:22
anteayathank you18:22
zaroanteaya: trying to setup now18:24
*** jsavak has quit IRC18:25
prometheanfiregreghaynes: hi :P18:25
*** baoli has quit IRC18:25
openstackgerritAndreas Jaeger proposed openstack-infra/git-restack: Remove argparse from requirements  https://review.openstack.org/27041118:26
pabelangerjeblair: fungi: is there any reason not to move pep8 jobs from bare-trusty to ubuntu-trusty images?18:26
pabelangeror are we waiting until bare-precise is removed18:27
fungipabelanger: mostly waiting until dib uploads to rackspace are stable18:27
anteayazaro: thanks, share your feedback in a pm if you would18:27
anteayaas this is internal18:27
*** salv-orl_ has quit IRC18:28
*** salv-orlando has joined #openstack-infra18:28
fungipabelanger: but also, the jobs need to use a different job template to call the install-distro-packages builder first, which also implies reordering the other builders since we need to checkout the change, then install system packages, then revoke sudo, then invoke tox18:29
*** vivekd has quit IRC18:29
clarkbfungi: we can in theory do that before the switch18:29
*** kzaitsev_mb has joined #openstack-infra18:29
clarkbnot sure if the extra effort is worth it18:29
fungii have an example in the experimental-jobs.yaml of doing that for nova unit tests18:29
anteayaclarkb: as you your comment from last night, yes they did shut down the cloud, as you surmised18:29
*** gokrokve has quit IRC18:29
*** annegentle has quit IRC18:30
fungier, experimental-workers.yaml18:31
*** angdraug has quit IRC18:31
*** salv-orlando has quit IRC18:31
*** esikachev has quit IRC18:31
kzaitsev_mbfolks, who should I ask to remove stable/juno from murano repositories? =)18:32
*** salv-orlando has joined #openstack-infra18:32
pabelangerfungi: okay, let me take a look into that18:32
pabelangerfungi: curious what is all needed to do the switch18:32
*** ilyashakhat has joined #openstack-infra18:32
kzaitsev_mbI wrote a letter to openstack-infra last week, but got now response =)18:32
anteayakzaitsev_mb: is is timing18:32
*** Swami has joined #openstack-infra18:33
anteayakzaitsev_mb: it needs to be an infra-root and last week there weren't many around18:33
fungipabelanger: also we should switch the bindep-fallback-ubuntu-trusty job to voting and add similar jobs for any other worker types where we may want to use the same job templates18:33
fungipabelanger: that will make sure we don't break the world by merging a broken fallback package list18:34
anteayakzaitsev_mb: this week there are more but they are catching up from what they missed last week, thanks for your patience18:34
*** DuaneDeC7 has quit IRC18:34
*** amitgandhinz has quit IRC18:34
pabelangerfungi: Ya, I need to see what that job does. Haven't been following bindep much18:34
kzaitsev_mbanteaya: thanks for update =) there's obviously no rush from our side of things =)18:34
*** ilyashakhat has quit IRC18:34
fungikzaitsev_mb: i saw your e-mail and have put it into my tentative to-do pile, in case nobody else gets to it before i am freed up18:34
kzaitsev_mbfungi: thanks = )18:35
*** esikachev has joined #openstack-infra18:35
*** ilyashakhat has joined #openstack-infra18:35
fungipabelanger: it basically exercises the same script which jobs would use to invoke bindep, but overrides it to test proposed changes to our fallback list or proposed changes to bindep accordingly18:35
*** amitgandhinz has joined #openstack-infra18:35
fungipabelanger: that way we get it verified with the tool which would consume it18:36
kzaitsev_mbjust wanted to be sure, that I, say wrote to the correct ML and didn't mix things up )18:36
*** annegentle has joined #openstack-infra18:36
kzaitsev_mbthanks again for info18:36
*** gyee has joined #openstack-infra18:37
*** DuaneDeC7 has joined #openstack-infra18:37
fungipabelanger: and covers corner cases such that we can't merge a fallback list which the installed bindep on our workers is unable to use, nor change bindep such that it can't work with the current state of the fallback list on our workers18:37
fungibelt meet suspenders18:37
timrczaro, Khai long time not speak.  Are you around for a quick chat about jjb?18:37
*** exploreshaifali has joined #openstack-infra18:38
*** trown is now known as trown|lunch18:38
*** zz_dimtruck is now known as dimtruck18:38
fungipabelanger: no doubt there are other corner cases we'll discover as we start relying on this more and more, but i at least tried to plan for the obvious ones18:38
*** shashank_hegde has joined #openstack-infra18:38
fungipabelanger: i've really not had the available time i've wanted to spend on it, so welcome any assistance you end up being able to render18:40
prometheanfirethink I scared him off18:40
fungiseems from his earlier messages that he's on a call18:40
*** infra-red has joined #openstack-infra18:40
*** ilyashakhat has quit IRC18:41
*** jsavak has joined #openstack-infra18:41
*** ilyashakhat has joined #openstack-infra18:42
pabelangerfungi: Sure, don't mind diving into it.  Let me setup some time to look into it and see what is needed to move it forward18:42
*** salv-orlando has quit IRC18:43
fungipabelanger: my hero!18:44
*** esikachev has quit IRC18:44
*** kzaitsev_mb has quit IRC18:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch ansible-role-diskimage-builder jobs voting  https://review.openstack.org/27537318:45
*** zul has joined #openstack-infra18:45
*** salv-orlando has joined #openstack-infra18:45
*** jlanoux has quit IRC18:46
*** tzn has joined #openstack-infra18:47
AJaegerdhellmann: could you review https://review.openstack.org/#/c/243295/, please? that's the solum releasenotes addition...18:47
dhellmannAJaeger : looking18:48
*** tzn has quit IRC18:48
*** annegentle has quit IRC18:49
*** angdraug has joined #openstack-infra18:50
*** annegentle has joined #openstack-infra18:53
pabelangerfungi: Ya, so I'd be curios which job would be a good place to gate on bindep. We have an experimental job on project-config, maybe start by moving that to non-voting18:53
pabelangerfungi: I should also see what is needed to run this on centos / fedora18:54
*** changbl has quit IRC18:54
AJaegerjohnthetubaguy, sdague : could you review https://review.openstack.org/#/c/273326 , please? This adds a voting ceph gate job for nova.18:55
fungipabelanger: plan was to adjust that bindep-fallback project-template so that it was running in check and gate instead of experimental. i've already tested it on multiple changes for both project-config and bindep so it should be stable18:55
*** Sukhdev has joined #openstack-infra18:55
pabelangerfungi: okay, I can start on that18:55
*** changbl has joined #openstack-infra18:55
fungipabelanger: so i guess move the ubuntu-trusty version of the job to check and gate, add centos-7 and fedora-23 (or whatever the dib node labels are for those) to experimental18:55
pabelangerfungi: right18:56
anteayaAJaeger: did you want to address the comment on https://review.openstack.org/#/c/27435918:56
fungipabelanger: and we can switch those to check and gate too once they're proven on a few changes in experimental18:56
AJaegeranteaya: I'll do some more cleanups on that file soon and can do it at that time...18:57
AJaegeranteaya: thanks for asking18:57
fungipabelanger: though maybe only if we're actually consistently building working images for those labels--don't want to block development on project-config and bindep if we have trouble satisfying demand for check and gate jobs with those platforms18:57
zarotimrc: i can try to answer your question.  but tbh, i've been out of touch lately.18:57
AJaegeranteaya: if that's fine with you18:57
fungipabelanger: since that can leave us in a catch-22 where we need to intervene manually18:57
zaroanteaya: i sent you an email with feeback.18:57
anteayaAJaeger: that's fine with me18:58
anteayazaro: thank you18:58
pabelangerfungi: okay18:58
SpamapSjeblair: ^5 for restack!18:58
timrczaro: Who's in touch these days? I'm interested in the future of jjb as openstack moves away.  Has anyone expressed interested in maintaining it?18:58
*** kdas_ has joined #openstack-infra18:59
jeblairSpamapS: v5 down low!18:59
timrczaro: It seems quite a few shops now use it.  Blue Box being one fo them.  We're not going to move away from Jenkins at the same time as Openstack, so I'd like to know that patches are still going to get reviewed / merged18:59
jeblairdid that work?  i've never tried that on irc18:59
*** tiswanso has quit IRC18:59
openstackgerritMerged openstack-infra/project-config: Update App Catalog dead link checker  https://review.openstack.org/26993518:59
prometheanfireSpamapS: hi :P18:59
fungimeeting time!19:00
fungi-> #openstack-meeting19:00
*** electrofelix has quit IRC19:00
*** tiswanso has joined #openstack-infra19:01
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add centos7 and fedora-23 to bindep-fallback  https://review.openstack.org/27537619:01
*** DuaneDeC7 has quit IRC19:01
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711719:02
openstackgerritJames E. Blair proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354919:02
openstackgerritJames E. Blair proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492719:02
greghaynesclarkb: ianw dib released19:02
*** jsavak has quit IRC19:02
openstackgerritMerged openstack-infra/project-config: Ignore empty django files  https://review.openstack.org/27509019:02
*** jsavak has joined #openstack-infra19:03
*** rbrndt has joined #openstack-infra19:03
openstackgerritShamail Tahir proposed openstack-infra/system-config: Adding meetbot and statusbot support for openstack-product. The openstack-product IRC channel will be used by the OpenStack Product Working Group for additional conversations when the meetings take longer than our allocated time on openstack-meeting-alt.   https://review.openstack.org/27536419:03
*** weshay_xchat is now known as weshay19:03
zarotimrc: get in touch with waynr, zxiiro and electrofelix.19:03
*** DuaneDeC7 has joined #openstack-infra19:04
*** matrohon has joined #openstack-infra19:04
clarkbtimrc: zaro I don't think we had any intention of killing the project. It has users that maintain it and so on19:04
zarotimrc: i know waynr is doing a lot of work with new features for jjb.19:04
*** sridhar_ram1 has quit IRC19:04
*** Swami_ has joined #openstack-infra19:04
zxiirotimrc: My organization will depend on JJB for a long time and we hope to keep it maintained even if OpenStack stops using it.19:05
*** esker has joined #openstack-infra19:05
*** crinkle_ has joined #openstack-infra19:05
*** asselin_ has joined #openstack-infra19:05
*** xwizard__ has joined #openstack-infra19:06
*** crinkle has quit IRC19:06
*** crinkle_ is now known as crinkle19:07
*** YorikSar_ has joined #openstack-infra19:07
*** jamespd_ has joined #openstack-infra19:07
*** mrmartin has joined #openstack-infra19:07
*** sridhar_ram has joined #openstack-infra19:07
timrcSo let's just take https://review.openstack.org/#/c/266970/ for an example...19:07
timrcWhere are the cores at? How do we get their attention?19:07
mgagnewhen your coworker compares cloud-init to django...19:08
openstackgerritShamail Tahir proposed openstack-infra/system-config: Adding meetbot and statusbot support for openstack-product  https://review.openstack.org/27536419:09
zarotimrc: other projects have consumed my attention these days :(19:10
*** flepied has joined #openstack-infra19:10
zxiirotimrc: I feel you, folks are busy and reviews aren't quick. It's something I've complained on several occasions myself. Now that I'm a core I try to keep up with reviews but the day job unfortunately takes priority so I can only review a few patches each week.19:10
timrcSo... I'll put something on next Tuesday's meeting.19:10
*** luv_ has joined #openstack-infra19:10
*** jpr has quit IRC19:10
timrcWe need to find people that can actively review and merge stuff to jjb.19:10
*** sbadia_ has joined #openstack-infra19:10
*** f1ller- has joined #openstack-infra19:10
*** jpr has joined #openstack-infra19:10
*** itsuugo_ has joined #openstack-infra19:10
*** lbragstad_ has joined #openstack-infra19:11
timrcNow, I must BBQ in... Seattle19:11
*** bknudson_ has joined #openstack-infra19:11
*** lucasagomes_ has joined #openstack-infra19:11
*** mari0jv has joined #openstack-infra19:11
openstackgerritShamail Tahir proposed openstack-infra/project-config: Adding gerritbot and accessbot support for openstack-product  https://review.openstack.org/27536519:11
*** htruta` has joined #openstack-infra19:11
*** pabelanger_ has joined #openstack-infra19:11
*** coreycb` has joined #openstack-infra19:11
zxiirotimrc: for what it's worth I'm mentoring a student this upcoming summer to work on JJB. Hoping he can help with some of the reviews.19:11
jeblairtimrc: thank you for volunteering19:11
*** charz has joined #openstack-infra19:11
*** mrodden_ has joined #openstack-infra19:11
*** Swami has quit IRC19:12
*** kushal has quit IRC19:12
*** Kennan has quit IRC19:12
*** luv has quit IRC19:12
*** hogepodge has quit IRC19:12
*** lucasagomes has quit IRC19:12
*** pcrews has quit IRC19:12
*** waynr has quit IRC19:12
*** xwizard_ has quit IRC19:12
*** pabelanger has quit IRC19:12
*** asselin has quit IRC19:12
*** akuznetsova has quit IRC19:12
*** katyafervent_awa has quit IRC19:12
*** mordred has quit IRC19:12
*** bknudson has quit IRC19:12
*** mrunge has quit IRC19:12
*** sbadia has quit IRC19:12
*** lindsayk- has quit IRC19:12
*** pavel_bondar has quit IRC19:12
*** htruta has quit IRC19:12
*** itsuugo has quit IRC19:12
*** coreycb has quit IRC19:12
*** f1ller has quit IRC19:12
*** jamespd has quit IRC19:12
*** YorikSar has quit IRC19:12
*** pkarikh has quit IRC19:12
*** mariojv has quit IRC19:12
*** charz_ has quit IRC19:12
*** michaelxin has quit IRC19:12
*** thingee has quit IRC19:12
*** lbragstad has quit IRC19:12
*** mrodden has quit IRC19:12
*** krotscheck1 has joined #openstack-infra19:12
*** lbragstad_ is now known as lbragstad19:12
AJaegerit's 502 gerrit time for me ;(19:12
*** sbadia_ is now known as sbadia19:12
*** mrodden_ is now known as mrodden19:12
*** luv_ is now known as luv19:12
*** pabelanger_ is now known as pabelanger19:12
*** itsuugo_ is now known as itsuugo19:12
*** maishsk has joined #openstack-infra19:12
*** matrohon has quit IRC19:13
*** mrunge has joined #openstack-infra19:13
openstackgerritShamail Tahir proposed openstack-infra/system-config: Adding meetbot and statusbot support for openstack-product  https://review.openstack.org/27536419:13
*** pkarikh has joined #openstack-infra19:13
*** katyafervent_awa has joined #openstack-infra19:14
*** htruta` is now known as htruta19:14
*** akuznetsova has joined #openstack-infra19:14
*** mordred has joined #openstack-infra19:15
*** pavel_bondar has joined #openstack-infra19:15
*** waynr has joined #openstack-infra19:16
*** pcrews has joined #openstack-infra19:16
*** Kennan has joined #openstack-infra19:16
*** boris-42 has joined #openstack-infra19:19
*** esikachev has joined #openstack-infra19:19
anteayaAJaeger: :( time for a break?19:19
AJaegeranteaya: just not multitasking ;)19:20
AJaegerreloads work...19:20
AJaegerjeblair, krotscheck1 : I have a question on https://review.openstack.org/#/c/164927 - uname -m gives i686 for me19:21
*** kdas_ has quit IRC19:21
krotscheck1AJaeger: Ubuntu?19:21
*** dizquierdo has quit IRC19:21
AJaegerkrotscheck1: openSUSE19:22
AJaegeruname -i gives i38619:22
AJaegerkrotscheck1: What OS are you on?19:22
krotscheck1AJaeger: Hrm. I'm on an ubuntu trusty cloud node, and I have x86_6419:22
AJaegerkrotscheck1: try "linux32 uname -i"19:22
clarkbAJaeger: krotscheck1 its going to be install dependent19:23
clarkbif you installed i686 you will get that19:23
AJaegerlinux32 gives you a 32-bit environment...19:23
krotscheck1AJaeger: i68619:23
AJaegerfor -i? Oh...19:23
*** hogepodge has joined #openstack-infra19:24
ianwgreghaynes: cool , was it just then?  assuming latest builds missed it19:24
*** exploreshaifali has quit IRC19:24
*** kushal has joined #openstack-infra19:24
AJaegerkrotscheck1: if we always run it on same environment it won't matter ;) But code would be more robust if we check for i?86 and replace with whatever you want;)19:25
krotscheck1AJaeger: I'm not too worried about it in that particular case. The wheel slave will build wheels for system/version/architecture, and a slave will only resolve from its appropriate system/version/architecture.19:25
*** kushal has quit IRC19:25
clarkbianw: I can trigger fedora 23 after meeting19:25
*** salv-orlando has quit IRC19:25
*** thingee has joined #openstack-infra19:26
krotscheck1AJaeger: If the slave doesn't match exactly, it won't get a mirror.19:26
AJaegerkrotscheck1: ah, ok. Then let's try it...19:26
* AJaeger will +2 but let jeblair hit the +A19:26
ianwclarkb: that would be good, thanks.  i have nothing else outstanding -- the latest dib should have growroot support for f23 so we should have enough disk-space to run a devstack+tempest19:26
krotscheck1jeblair: Well, let's let the jobs pass first ;)19:27
clarkbianw: reviews on https://review.openstack.org/#/c/272384/7 would also help reliability of image builds/uploads19:27
*** esp_ has joined #openstack-infra19:27
*** jesusaurus has joined #openstack-infra19:28
*** infra-red has quit IRC19:29
*** NobodyCa1 has joined #openstack-infra19:30
mrmartinsomeone could help me today to make a backup form ask.o.o's database because I want to fix a bug on ask-staging.o.o and I need to make sure that the bug was caused by an upstream repo branch issue, so it is not an app error.19:30
*** deva_ has joined #openstack-infra19:30
krotscheck1I meant AJaeger ^^19:31
*** dmsimard has joined #openstack-infra19:32
* dansmith throws something rotten at gerrit19:32
AJaegerAh, all dansmith's fault now ;)19:33
timrczxiiro: Hey good to know you're a core.  That's excellent19:33
dansmithAJaeger: I'm throwing in response to the issues, not in order to cause them :P19:33
*** trown|lunch is now known as trown19:33
*** esp_ has quit IRC19:34
zxiirotimrc: yep. I've been a bit slower on reviews lately because my day job is in the middle of a big release cycle in 3 weeks so all my focus unfortunately's been with that. I'll be back to reviewing as much as I can once we're done with that though.19:35
*** NobodyCa1 has quit IRC19:35
mrmartinAJaeger: are you a core?19:35
*** jesusaurus has quit IRC19:35
*** deva_ has quit IRC19:36
*** vgridnev has joined #openstack-infra19:36
AJaegermrmartin: which repo? You know that you can find the cores yourself?19:36
clarkbianw: looks like paths may have changed sufficiently to make the per image log files not work properly. Liekyl something I missed in splitting the logging configs19:36
openstackgerritIan Wienand proposed openstack-infra/system-config: Use diskimage-builder from git for nodepool  https://review.openstack.org/27500619:36
mrmartinAJaeger, I just like to get a backup from ask.o.o's db and move a copy to ask-staging.o.o19:37
*** dmsimard has quit IRC19:37
AJaegermrmartin: that needs infra-root AFAIK - and I'm not one19:37
ianwpabelanger: ^ so another question might be, why did that *not* fail all the other apply tests...19:37
mrmartinoh ok.19:37
*** dtardivel has quit IRC19:38
ianwclarkb: ok, i haven't checked today.  i can take a look in a bit19:38
AJaegermrmartin: clarkb, fungi, jeblair are around and can help.19:38
pabelangerianw: good question, I'd have to look19:38
AJaegermrmartin: but right now is infra meeting...19:38
mrmartinI know.19:38
timrczxiiro: Hey I probably came off as a bit rude, so my apologies there.  I completely understand trying to balance responsibilities.19:38
zxiirotimrc: not at all. I didn't take any offense whatsoever :)19:39
zxiirotimrc: I understand where your coming from, I wish JJB reviews would be quicker myself too.19:40
SpamapSShrews: hey, can you check out https://review.openstack.org/274360 and verify for me that we are, in fact, going to remove munch some day?19:40
*** dmsimard has joined #openstack-infra19:41
*** dmsimard has quit IRC19:41
pabelangerianw: so, puppet-apply did catch the problem it seems however jenkins_dev is only run on trusty.  I think you want to add the change for 275006 some other place19:42
*** sridhar_ram1 has joined #openstack-infra19:43
*** maishsk has quit IRC19:43
krotscheck1jeblair: Shifting into daycare mode. Having some trouble with the project-config builds, something's not syntaxing right.19:43
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add centos7 and fedora-23 to bindep-fallback  https://review.openstack.org/27537619:43
krotscheck1KeyError: 'name'19:43
krotscheck1I'll look at it after 3ish if possible.19:43
*** krotscheck1 has quit IRC19:44
*** _nadya_ has joined #openstack-infra19:44
*** sridhar_ram has quit IRC19:44
*** maishsk has joined #openstack-infra19:44
pabelangerianw: I think you need to update puppet-openstackci19:44
timrczxiiro: I'll start reviewing more in this area if it helps any.  Might be a good way of priotizing what to look at for +2 if there are multiple +1's.19:45
*** dmsimard has joined #openstack-infra19:46
AJaegerkrotscheck1, jeblair: I'm looking at the wheel syntax now..19:47
openstackgerritIgor Belikov proposed openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1'  https://review.openstack.org/27532919:48
timrcclarkb: Also, I went with the "crowd" to Pike's BBQ... really, really bad in terms of brisket.  I had to know though.19:48
*** zxiiro_ has joined #openstack-infra19:49
clarkbtimrc: I have never been19:49
clarkbbut will likely make my own this weekend for sportsball19:49
timrcclarkb: You are not missing anything unless you like disappointment.19:49
Clintand who doesn't like disappointment19:49
timrcI dunno.. alderwood.  I feel sorry for people here that can't get post oak.19:49
clarkbI have done hole in the wall and their brisket sandwich is decent19:49
clarkbtimrc: apple19:49
*** simondodsley_ has joined #openstack-infra19:50
timrcThis brisket sandwhich tasted like sloppy joes.  It was... disconcerting.19:50
bkeroCan we just get some better bbq in Austin?19:50
*** dkranz has quit IRC19:50
prometheanfirebkero: come down to san antonio if you want real bbq19:51
*** flepied has quit IRC19:51
*** zxiiro has quit IRC19:51
*** simondodsley has quit IRC19:51
clarkbeh I just make it myself because then I don't have to go to texas :)19:51
timrcbkero: If you expanded into Central Texas you have many more options :) If you want very good bbq run by !hipster Cooper's in Llano.  Worth the drive.19:51
*** simondodsley_ is now known as simondodsley19:51
bkeroWhatever, that's a short drive19:51
bkero(San Antonio)19:51
*** esker has quit IRC19:51
*** zxiiro_ is now known as zxiiro19:51
bkerotimrc: Sadly I think afterwards I might be headed easterly instead.19:51
prometheanfirestill don't understand why we are having a summit in austin19:51
prometheanfireit's not gonna be a fun time19:51
bkeroThat's where it was the first time?19:52
SpamapSprometheanfire: oh hi, you said hi. How's the gentoo stuff going?19:52
*** michaelxin has joined #openstack-infra19:52
prometheanfireSpamapS: need a couple of +2/+w19:52
prometheanfiresince gate is passing again19:52
prometheanfireor was 12h ago19:52
timrcprometheanfire: It's going to be a grand time for me.  And a 20 minute drive from my house.19:52
*** maishsk has quit IRC19:52
*** flepied has joined #openstack-infra19:53
*** lindsayk has joined #openstack-infra19:53
prometheanfiretimrc: so that's 2 blocks?19:53
prometheanfireI'm going to book my hotel this weekend most likely19:53
*** amrith is now known as _amrith_19:53
prometheanfiregoing for 9 on the map19:53
*** esikachev has quit IRC19:54
*** _amrith_ is now known as amrith19:54
prometheanfireSpamapS: want links?19:54
*** Sukhdev has quit IRC19:54
*** resker has joined #openstack-infra19:54
prometheanfirehttps://review.openstack.org/#/c/273795/ is the important one19:54
*** esikachev has joined #openstack-infra19:55
SpamapSprometheanfire: I'll run the whole queue soon, I think I see some :)19:55
prometheanfireI -1 one waiting on the lined review19:56
*** maishsk has joined #openstack-infra19:56
*** gyee has quit IRC19:57
*** ashleigh_ has joined #openstack-infra19:59
*** ashleighfarnham has quit IRC19:59
*** ashleigh_ has quit IRC20:00
pabelangerfungi: AJaeger: Indeed, 2 talks submitted! So, once voting starts, I'll be sure to email out the talks for people to vote on :)20:01
*** jaypipes has joined #openstack-infra20:01
greghaynesianw: yep, the latest builds likely did miss it20:02
*** Sukhdev has joined #openstack-infra20:02
AJaegerpabelanger: you submitted an old version of my abstract, I updated - and I capitalized Infrastructure in the title this morning. Could you double check, please?20:02
openstackgerritIgor Belikov proposed openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1' for Debian  https://review.openstack.org/27532920:02
*** Sukhdev has quit IRC20:03
*** aysyd has quit IRC20:04
*** Sukhdev has joined #openstack-infra20:04
*** reed_ has joined #openstack-infra20:04
SpamapSfungi, clarkb, jeblair: regarding swift being unsuitable for docs hosting.. this seems like a huge failure of swift. I wonder if we should take a small detour and make sure these things are recorded as bugs.20:04
*** rfolco has quit IRC20:04
AJaegerjeblair: found the bug in the wheel change, currently testing locally...20:05
clarkbSpamapS: I have spent the last couple years trying to explain our use case to swift :)20:05
fungiSpamapS: i don't know that it's entirely a failing of swift20:05
clarkbbut as fungi says its not necessarily swifts fault, we just expected something different of it20:05
*** ilyashakhat has quit IRC20:05
fungimore like our organically-grown documentation management and arrangement does not easily map to what swift provides20:06
SpamapSI'd expect that it would have a way to take a local directory with index.html's pre-created, and host that.20:06
clarkbSpamapS: it does, but then for every container you do that with you hvae to update dns20:06
fungiSpamapS: sure, it does. we could build the entire site and then upload it to swift, no problem20:06
clarkb(granted that particular thing is an implementation detail)20:06
*** tiswanso has quit IRC20:07
fungithe problem domain here is around building and updating, not around how to serve http20:07
prometheanfireSpamapS: about removing the interface files, really I could just not do anything there as they don't exist by default, but if another element ever adds them they will be missed20:07
*** tiswanso has joined #openstack-infra20:07
fungionce we're already stuck maintaining a copy of all the content, it's easier to just apt-get install apache220:07
SpamapSso it sounds like it's as much the CDN failing as swift itself.20:07
clarkbSpamapS: its a "service" failing imo20:08
AJaegeranteaya: you asked me earlier about https://review.openstack.org/#/c/274359 - did you on purpose not vote or hit a 503? ;)20:08
*** jsavak has quit IRC20:08
prometheanfiregerrit just got really slow20:08
SpamapSNo I'd want to build a dir, upload it to a container, and then tell the frontend "that container is now /docs/deployment-guide/"20:08
*** adrienverge has joined #openstack-infra20:08
clarkbwe hvae many services but the overlap between what tehy do and what users want is often not very large20:08
prometheanfireand better now20:09
*** jsavak has joined #openstack-infra20:09
ShrewsSpamapS: will look later tonight20:09
anteayaAJaeger: I just wanted to know if you wanted to comment, which you did, so I will review, thank you20:09
clarkbbut that is based on my biased opinion formed with one sample20:10
*** ociuhandu has quit IRC20:10
fungiSpamapS: from a docs publication standpoint, a lot of the challenges are based around the fact that there is no clear separation between what build processes own which files20:11
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard  https://review.openstack.org/27539420:11
*** achanda has quit IRC20:11
SpamapSclarkb: there's a whole excited dev community over on AWS who host everything in their UI on s3+cloudfront.. so I'm kind of lamenting that swift doesn't provide that.20:11
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool  https://review.openstack.org/26472620:12
*** mrmartin has quit IRC20:12
SpamapSfungi: wait, they're not segmented into dirs?20:12
fungiit's gotten a lot cleaner over the course of the past year or two, but i gather there's still muddiness over multiple jobs updating overlapping/interleaved bits of documents20:12
SpamapSwell damn20:12
clarkbSpamapS: there is no such thing as dirs with swift20:12
AJaegerjeblair: I commented on https://review.openstack.org/#/c/273549 on the two problems I found but now I have another one that confuses me ;( So, giving up and let you fix it if you want - or krotscheck1 later20:12
clarkbSpamapS: just key value pairs essentially20:12
*** HeOS has joined #openstack-infra20:12
SpamapSclarkb: no, there are containers though, and I'm kind of hoping containers can be mapped to paths by a CDN.20:12
SpamapSI am realizing now20:13
* prometheanfire goes to do a rain dance waiting for patches to merge20:13
SpamapSthe problem is the frontend20:13
SpamapSnot swift per se20:13
*** ashleighfarnham has joined #openstack-infra20:13
fungiright, swift does a great job of storing blobs of data and letting you retrieve them again on demand20:13
SpamapSwell and grouping them actually20:13
SpamapSjust, one level group20:13
fungibut then again, so does a filesystem20:13
SpamapSeh.. a filesystem does a much much much better job. But .. it's a static HTML site. You shouldn't need much of that.20:14
fungiand providing access controls around writing to them and so on20:14
SpamapSBut yeah, I am 100% seeing where if you have to actually build up the whole thing.. you may as well just let the FS that you just built do it.20:14
*** moravec has quit IRC20:16
anteayafungi clarkb bluebox is in fairly heavy error state20:16
clarkbSpamapS: as the user you don't know what the shared secret between swift and cdn is so you cna't just map the two20:16
clarkbSpamapS: there is some other non openstack api thing you can hit for that iirc20:16
*** moravec has joined #openstack-infra20:16
clarkbianw: I have image uploads for fedora 23 queued to the rax regions. I notice that we do not hvae fedora 23 in the other clouds, want to add it to them?20:17
ianwclarkb: yep, on my todo list ... just trying to get one working first :)20:18
clarkbianw: thats fair20:18
clarkbianw: wasn't sure if that was a known todo or not20:18
clarkband rax uploads are broken :/20:19
fungialso in some situations we're going to need a real webserver to do things that simply aren't going to be supported by the cdn. actually having apache there gives us greater control for things like proxying to other systems, et cetera20:19
clarkbwhich as an aside go into swift <_<20:19
ianwgreghaynes / clarkb : dropped a query on the disconnect  change ... i'm no gear client expert20:19
greghaynesianw: hrm?20:20
clarkbOpenStackCloudException: Failed at action (create_container) is the error20:20
SpamapSclarkb: well for that I'd probably just have the job that built the site signal a worker somewhere "hey container X is the new /nova" right? But meh.. non openstack API makes me sad too ;)20:21
greghaynesseems like the api said "no"20:21
clarkbgreghaynes: yes I think thats basically it20:21
clarkbunauthorized though so maybe its something with oscc20:21
fungidid we exceed a quota there possibly?20:21
*** jgriffith_away is now known as jgriffith20:21
pabelangerAJaeger: I sure, I had to do some trimming because of the char limit20:21
clarkbits 401 unauthorized20:22
clarkbso I don't think it was timeout or quota20:22
pabelangerAJaeger: you should be able to make edits on the talk directly20:22
AJaegerpabelanger: I did...20:22
clarkbcould be our old friend expiring tokens is back20:22
clarkbor I fixed it in one place but needed two20:22
greghaynesyea, sorry, thats what I mean20:22
AJaegerpabelanger: that's why I asked for double checking20:22
*** HeOS has quit IRC20:22
greghaynestoken timeout ;)20:22
greghaynesWe have way too many types of timeout20:23
fungioh, right. session management for long glance uploads is still meh20:23
greghaynesfungi: Yep, and swift too20:23
fungii meant long uploads into swift for glance to use20:23
pabelangerAJaeger: Ah, okay. Looking now20:23
*** kzaitsev_mb has joined #openstack-infra20:24
fungiglance assigns the location as part of the task, right?20:24
notmorganclarkb: http://logs.openstack.org/27/275327/1/check/gate-tempest-dsvm-full/3c4f1df/logs/apache/keystone.txt.gz that look better from a timestamp perspective?20:24
pabelangerAJaeger: Ya, we are -252 over the char limit now20:24
greghaynesfungi: you just upload it to swift first and then tell glance where it is20:24
notmorganclarkb: that is removing microseconds20:24
fungii'm still slightly fuzzy on the signalling between glance and swift for task-based uploads20:24
pabelangerAJaeger: not sure what will happen20:24
AJaegerpabelanger: oops, let me check...20:24
clarkbSpamapS: so this would be another issue with it that we have found more recently. The client tooling is really hard to use20:24
fungioh, so it's a normal swift upload, nothing special, glance has no idea you're doing anything until you tell it afterward?20:24
ianwgreghayens: 27238420:24
clarkbfungi: yes20:24
clarkbfungi: so problem here is swiftclient doesn't use sessions20:25
greghaynesfungi: Yep. The only specialish aspect is that our files are large enough we have to break them up20:25
clarkbfungi: so in a daemon that runs for days your token expires and it stops working20:25
*** krotscheck1 has joined #openstack-infra20:25
fungigot it. so the expectation is that the consuming system should pick idle times to reauthorize and refresh its token?20:25
notmorganclarkb: sorry, it's on my long list of things to fix...20:25
clarkbnotmorgan: that'll work, though milliseconds are nice if you have them but not a deal breaker20:25
* SpamapS twitches when considering swift image upload in RAX20:25
greghaynesfungi: I think so20:25
AJaegerjeblair, krotscheck1 I fixed the wheel jobs!20:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711720:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354920:25
notmorganclarkb: can't do milliseconds, doesn't have the flexibility afaict20:26
notmorganit's microsecond or no subsecond20:26
*** bexelbie_ has joined #openstack-infra20:26
dmsimardIs there somewhere I could download the centos 7 image that runs in the gate ? I am unable to reproduce an issue that happens in a job there on a local centos image.20:26
krotscheck1AJaeger: Neat, thanks!20:26
clarkbnotmorgan: kk20:26
notmorganclarkb: i'll revisit the swift thing soon.20:26
notmorganSpamapS: ^20:26
notmorganit really isn't far down my list20:26
pabelangerfungi: is -infra revisiting the use case of swift? I think that was mentioned last meeting?20:26
notmorganjust... not at the top atm with some higher prio cleanups20:26
*** bexelbie is now known as Guest4927720:26
*** bexelbie_ is now known as bexelbie20:26
*** ybathia has quit IRC20:26
*** bexelbie- has joined #openstack-infra20:27
notmorgangetting swift on session is a must.20:27
pabelangerdmsimard: you could, but you wouldn't be able to log into it20:27
pabelangerdmsimard: you should be able to build one20:27
SpamapSalso remember there are really two swiftclients in swiftclient20:27
dmsimardpabelanger: why not ? no cloudinit ?20:27
SpamapSthere's the easy one, and the functional one20:27
pabelangerdmsimard: well, we use SSH keys20:27
dmsimardpabelanger: that's fine, I can boot it in a local openstack environment with my key set up20:28
*** salv-orlando has joined #openstack-infra20:28
*** piet has quit IRC20:28
fungipabelanger: i think that was some of the feedback on jhesketh's latest plans was that we may be expending unnecessary effort to map posix filesystem semantics onto swift containers, and that we should revisit better-suited alternatives to make sure the complexity is worthwhile20:28
dmsimardI can try and build it if it's not rocket science, I just thought downloading it would've been simpler :)20:28
AJaegerpabelanger: I'm on it...20:29
timrcprometheanfire: I actually live in Lakeway.  I just drive wrecklessly.20:30
pabelangerdmsimard: look at tools/build-image.sh to setup it up locally.  Easier then having infra expose the images to download20:30
pabelangerdmsimard: under project-config20:30
fungidmsimard: the dib elements and envvars used to build our devstack-centos7 image are listed at http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n49120:30
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492720:31
krotscheck1AJaeger: I found a bug in the first patch in that chain.20:31
dmsimardOk, I'll try that. Thanks.20:31
fungidmsimard: as you can see, it's based on dib's centos7 element primarily20:31
krotscheck1Oh, bloody....20:31
krotscheck1silly rebase20:31
pabelangerdmsimard: DISTRO=devstack-centos7 ./tools/build-image.sh should do the trick20:32
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492720:32
*** piet has joined #openstack-infra20:32
*** tiswanso has quit IRC20:32
clarkbok I can't start tugging on this swift upload thread20:32
clarkbI won't get anything else done this week if I do20:32
clarkbianw: the lazy in my would suggest adding that image to ovh and internap and bluebox20:33
clarkbianw: and worry about rax when we can figure out uploads to them20:33
*** infra-red has joined #openstack-infra20:34
AJaegerkrotscheck1: krotscheck1 you need to rebase the whole series...20:34
krotscheck1AJaeger: I know, working on it.20:34
krotscheck1Gerrit s not cooperating20:34
*** tiswanso has joined #openstack-infra20:35
AJaegerpabelanger: ok to remove the companies from our names? Those are mentioned in the speaker bio anyhow...20:35
* krotscheck1 peers at that patch chain.20:35
* krotscheck1 has no idea how that happened.20:35
AJaegerkrotscheck1: what happened?20:36
krotscheck1AJaeger: It thinks I removed the wheel builder from the macros file.20:36
krotscheck1It was never there to begin with.20:36
pabelangerAJaeger: sure20:37
krotscheck1THat's a change jeblair made20:37
* krotscheck1 sighs20:37
* krotscheck1 reverts all the things.20:37
ianwdmsimard: do you have a rax account ... i could probably build one and share the image with you20:38
*** reed_ has quit IRC20:38
dmsimardianw: don't think I do, it's alright, I'm working on building it :)20:38
*** infra-re_ has joined #openstack-infra20:38
* mgagne opens one and get hit with 50$ infra monthly fee20:38
*** simondodsley has quit IRC20:38
ianwdmsimard: yeah, you probably want to add the devuser element, and set something like DIB_DEV_USER_PASSWORD="..."  DIB_DEV_USER_PWDLESS_SUDO=120:39
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492720:39
AJaegerpabelanger: updated, please double check20:40
clarkbgreghaynes: I think ianw's comment on https://review.openstack.org/#/c/272384/7 may be on to something. We indisciminately remove the db record even though the job may have failed20:40
clarkbgreghaynes: should we set it up to onComplete remove the record instead?20:40
krotscheck1AJaeger: I've reverted ^^ back to jeblair's patch (PS30).20:40
*** infra-red has quit IRC20:41
krotscheck1AJaeger: All patches in the chain seem to have fixed themselves.20:41
HenryGgerrit responsiveness is pants20:42
*** krotscheck1 has quit IRC20:42
*** zeih has joined #openstack-infra20:42
AJaegerkrotscheck1: no, they have not. If you go up, it's broken. Let me do it...20:42
pabelangerAJaeger: +120:42
*** reed_ has joined #openstack-infra20:42
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354920:42
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711720:43
*** eil397 has joined #openstack-infra20:43
*** bexelbie has quit IRC20:44
*** bexelbie- is now known as bexelbie20:44
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711720:47
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Added wheel-release job  https://review.openstack.org/27354920:47
*** sdake has joined #openstack-infra20:47
clarkbthe GC time on gerrit has jumped in the last day20:47
*** resker has quit IRC20:48
*** reed_ has quit IRC20:48
jeblairclarkb: were we going to increase the ram or something?20:48
openstackgerritMerged openstack/diskimage-builder: Add new cloud-init element  https://review.openstack.org/27376420:49
clarkbits something I have mentoned .I think we have to do it to keep up with gerrits ram needs20:49
*** _nadya_ has quit IRC20:49
clarkbbut it is tricky because of the don't change IP or all corps break problem20:49
openstackgerritMerged openstack/diskimage-builder: Fix Gentoo hardened support  https://review.openstack.org/27379520:49
AJaegerpabelanger: great - time to call it done (from my side ;)20:49
*** diana_clarke has joined #openstack-infra20:49
pabelangerAJaeger: now we wait20:50
jeblairclarkb: can we grow this instance?20:50
clarkbjeblair: we can, I am always wary of that because it doesn't allow us to set downtime expectations ahead of time20:50
clarkbbut that is another options20:50
AJaegerpabelanger: yes ;)20:50
fungiwe can grow it?20:51
clarkbfungi: ya we should be able to resize it up a flavor or three20:51
fungidouble-checking. thought they removed that when they transitioned off their legacy environment20:51
clarkbnova supports resize20:51
clarkbbut maybe rax doesn't anymore20:52
fungii know _nova_ does20:52
*** Sukhdev has quit IRC20:52
fungiresize is greyed out in the actions manage drop-down section for review.o.o anyway20:52
*** achanda has joined #openstack-infra20:52
greghaynesclarkb: ianw re: deleting images change - there was much discussion in IRC which I failed at transcribing in that patch (so I should do that). tl;dr is there are several bad options for how to delete an image - we could try and estimate if there is a corresponding delete job we can call but that is likely to be incorrect and err on the side of keeping images, or we can err on the side of always20:52
greghaynescalling delete which will result in some hanging jobs20:52
timrczaro: Can I bribe you with brisket at the Summit to get https://review.openstack.org/#/c/271543/3 merged so https://review.openstack.org/#/c/252094/ merges?20:52
fungiclarkb: the only instances i'm aware you can still resize in rax are trove instances20:53
clarkbfungi: gotcha20:53
clarkbwell then, our option is to breaks HPs firewall rules :)20:53
greghaynesclarkb: ianw we figured it is easier to make deleteImage always call delete, and then in a later patch we can have that inspect a cache of what jobs are listed on the server to know wheter or not we can call the gearman delete20:53
*** mriedem is now known as mriedem_machell20:53
clarkbgreghaynes: so this is a best effort as far as deleting off of disk?20:54
greghaynesclarkb: Yep20:54
fungiso i've been assuming we're stuck with an ip address change if we need more ram. which is not terrible, but for the sake of courtesy we should provide some (a few weeks?) notice20:54
clarkbmy +2 is still applied20:54
jeblairfungi: we can also have the conversation about whether we need to worry so much now that https submission is a thing20:55
jeblair3rd party ci is still a concern of course20:55
fungiyes, great point20:55
anteaya+1 bigger instance20:55
zarotimrc: mmm, brisket bribery.  i'm a fan!20:56
anteayashall I work on an email?20:56
fungiclearly anyone who needs to ask their corporate firewall fascists to open ssh to our server should just give in and use https if possible20:56
*** jgriffith is now known as jgriffith_away20:56
*** dkranz has joined #openstack-infra20:56
fungiwhich for everything but stream-events is fine20:56
* anteaya will just copy paste the last email and let fungi insert new ip addresses20:56
zarotimrc: have you verified that it works?20:56
clarkbfungi: don't forget https://github.com/pypa/pip/pull/343520:58
fungiclarkb: yep, i went so far as to run the unit tests locally, now just need to figure out what amount of tweaking will accommodate null versions20:58
timrczaro: Yeah it's been awhile though.20:59
timrczaro: If that's the criteria to getting this merged, I'll happily re-setup the experiment to verify moving to the latest stable version of gearman-plugin does not break current version of Jankins.20:59
fungiclarkb: which is the easy part. the hard part is deciding whether i push an incremental update based on that commit or a new replacement for that commit20:59
clarkbfungi: I think it depends on the project21:00
clarkbfungi: some wants amends/squahse others want the dirty history21:00
timrczaro: I really want to see these zuul metrics come to life though :)21:00
fungiclarkb: and discard the old pull request and generate another pull request, or figure out if you can update an existing pull request21:00
clarkbfungi: you use the same pull request it automagically happens if you use th esame branch21:00
*** j_king_ is now known as j_king21:01
*** DuaneDeC7 has quit IRC21:01
fungithough to use the same branch i need to push --force or delete and rebranch21:01
fungii guess that's expected in githubland?21:01
clarkbor push a fast forwardable commit21:01
clarkbbut yes force or push fast forward are both normal21:01
fungiffable implies not squashed21:01
clarkbso there are two trains of thought and it dpeends on the project you are PRing21:02
fungii'll push --force and wait for screams21:02
*** thiagop has quit IRC21:02
fungii have a feeling this project's school of thought for a one-line bug fix is "whatever gets the job done"21:02
anteayaI'm working here: https://etherpad.openstack.org/p/WciQEwLsue21:03
clarkbzaro: is there some rough maths we can do to come up with a new server size?21:03
*** jsavak has quit IRC21:03
openstackgerritAdrien Vergé proposed openstack-infra/project-config: Lint YAML files  https://review.openstack.org/27184521:04
openstackgerritAdrien Vergé proposed openstack-infra/project-config: Fix linting errors in YAML files  https://review.openstack.org/27184421:04
*** vgridnev has quit IRC21:04
*** flepied1 has joined #openstack-infra21:04
*** jsavak has joined #openstack-infra21:04
*** salv-orl_ has joined #openstack-infra21:04
*** salv-orlando has quit IRC21:04
*** [1]Thelo has joined #openstack-infra21:04
clarkbit is worth noting I don't think this is the only cause of the 500 errors21:05
clarkbbut I do think it is the cause for the frequent errors as we GC more and pile up threads21:05
anteayaclarkb: can you add details to https://etherpad.openstack.org/p/WciQEwLsue21:05
anteayaclarkb: I'm mostly generalizing21:06
*** flepied has quit IRC21:06
clarkbanteaya: oh I think the message is fine I just wanted to point out that this likely won't fix it all21:06
*** DuaneDeC7 has joined #openstack-infra21:06
anteayaright, I just said we hope this will address the user experience21:06
anteayanot that we think this will fix the underlying cause21:06
anteayaand thanks for looking21:07
*** Thelo has quit IRC21:07
*** [1]Thelo is now known as Thelo21:07
*** salv-orl_ has quit IRC21:08
*** adrienverge has quit IRC21:08
*** dalgaaf has quit IRC21:08
anteayaadded that as I realize my prior sentences could be interpreted either way21:08
*** salv-orlando has joined #openstack-infra21:08
fungiperhaps a qualifier like "most" would help21:08
*** ihrachys has joined #openstack-infra21:09
pabelangerfungi: here is another review that nibalizer +1 workflow but zuul never enqueued.  27367821:09
*** Apoorva has quit IRC21:09
pabelangerfungi: that was 4 days ago21:09
zarotimrc: yeah, i think that's all it takes.21:09
fungipabelanger: was it approved before its depends-on merged?21:09
timrczaro: Okay cool.  Will give me an entry point back into this magical world.21:09
anteayafungi: I'll stick a most in there21:09
pabelangerfungi: I think it was21:10
anteayaadded a most21:10
fungipabelanger: known issue for projects who don't share a change queue for the gate pipeline. just needs workflow +1 (re)applied21:10
pabelangerfungi: Ah, okay. I didn't know that21:10
pabelangerfungi: do you mind toggling it?21:10
zaroclarkb: sorry, i totally forgot about that.  been bogged down trying to fix upstream gerrit bugs.21:10
zaroand the hp cloud thingys21:11
*** infra-re_ has quit IRC21:11
fungipabelanger: i've added a workflow +1 now21:11
clarkbzaro: I am just wondering if 2x is enough or if we should plan for more21:11
pabelangerfungi: danke21:12
fungipabelanger: looks like it was approved a minute after its dependency, but 7 minutes before that dependency merged21:12
zaroclarkb: would be a good try.  but i'm not sure that would be optimum atm.  can maybe give you a better answer tomorrow21:13
bknudson_gerrit seems a little slow (getting occasional proxy errors)21:13
*** dkranz has quit IRC21:13
*** kgiusti has left #openstack-infra21:13
anteayabknudson_: we are discussing moving the server21:13
bknudson_great, thanks21:14
anteayabknudson_: we don't seem to be able to get it to run error free21:14
anteayabknudson_: thanks21:14
jeblairclarkb: should we restart it?21:14
*** annegentle has quit IRC21:14
*** yamahata has joined #openstack-infra21:15
*** reed_ has joined #openstack-infra21:15
*** dkranz has joined #openstack-infra21:16
openstackgerritIan Wienand proposed openstack-infra/project-config: Roll out Fedora 23 to more providers  https://review.openstack.org/27541121:16
*** tzn has joined #openstack-infra21:16
clarkbjeblair: to address the current issues thats probably the best course of action21:16
ianwclarkb: ^ you know i don't think i've tried raw images21:16
clarkbzaro: if you can do that that would be great21:16
ianwi don't have accounts on any of these providers so debugging is harder21:17
*** amrith is now known as _amrith_21:18
AJaegerianw: your change conflicts with the stack at https://review.openstack.org/#/c/274162 - should we mrege that first?21:18
*** diana_clarke has quit IRC21:18
clarkbianw: I am not too worried about the other providers, they mostly just work bceause dhcp21:18
AJaegerian, please see https://review.openstack.org/#/c/274169/121:18
*** diana_clarke has joined #openstack-infra21:18
ianwclarkb: ha ha, famous last words :)21:18
jeblairfungi: topic:wheel_mirror is ready and passing, do you want to do a pass through and +A the system-config ones.  once the mirrors exist we can +A the project-config ones.21:19
*** smarcet has quit IRC21:19
*** tcammann_ has joined #openstack-infra21:19
AJaegerianw: isn't 274169 a duplicate?21:19
clarkbianw: eh its amazing how much better dhcp is for getting reliable networking21:19
fungijeblair: thanks for the heads up, switching back to that now21:19
*** tcammann has joined #openstack-infra21:19
openstackgerritMerged openstack-infra/project-config: Gate on fedora-23 for openstack-infra/system-config  https://review.openstack.org/27367821:19
*** NobodyCa1 has joined #openstack-infra21:20
*** esikachev has quit IRC21:20
*** bhunter71 has quit IRC21:20
clarkbuh do we have working ^ machines for that21:20
ianwAJaeger: hmm, yeah, what's going on with that21:20
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add support for OpenRC to dib-init-system  https://review.openstack.org/27377221:20
openstackgerritMerged openstack-infra/project-config: Add lib-forward-testing for osprofiler  https://review.openstack.org/27447221:21
AJaegerianw: 274162  is not approved yet...21:21
ianwahhh, ok let me check out 274162 and get this all sorted out21:21
AJaegerjeblair: is the stack at 274162 ready? It has 2 +2s and we want to give you the honour to merge it...21:21
*** gyee has joined #openstack-infra21:22
prometheanfireSpamapS: ianw: greghaynes https://review.openstack.org/#/c/273772/ is the last one for now21:22
*** esker has joined #openstack-infra21:22
*** gyee has quit IRC21:22
*** ubuntu has joined #openstack-infra21:23
*** ubuntu is now known as Guest4084821:23
*** annegentle has joined #openstack-infra21:23
*** jsavak has quit IRC21:23
fungiclarkb: er, yeah, i guess i should have looked more closely at that change before approving. granted it already had another workflow +1 and i didn't cr +2, but still my fault. happy to push up a revert and also do the manual steps to un-wedge us if necessary21:23
jeblairAJaeger: i think so, but ianw said something about 274169 at the end of that stack21:24
AJaegerjeblair, ianw: please figure it out together ;)21:24
clarkbfungi: if we don't need much disk then it may just work21:24
*** tcammann_ has quit IRC21:24
fungii have a feeling the puppet apply jobs need at least some nontrivial diskspace to install all the modules21:24
*** tcammann has quit IRC21:24
clarkbfungi: however https://review.openstack.org/#/c/275411/1/nodepool/nodepool.yaml should get us fedora-23 machines with disk21:25
jeblairclarkb: any reason you +2d that instead of https://review.openstack.org/274169 ?21:25
*** NobodyCa1 has quit IRC21:25
clarkbjeblair: only because ianw pushed it just now and pointed it out21:26
*** EricGonczer_ has joined #openstack-infra21:26
jeblairianw: any reason you pushed that rather than pointed clarkb at the older change you already +2d?21:26
*** jsavak has joined #openstack-infra21:27
jeblairi think it would be great if we could review changes rather than all of us just pushing the same change over and over again21:27
clarkbI agree, but we also sometimes do it and it isn't a huge deal (see hpcloud shutdown changes)21:28
ianwjeblair: yeah, i've just got confused as to what was committed, sorry, me cupla21:28
*** esker has quit IRC21:28
anteayasmcginnis: reading the novel, very digestable, well done21:29
ianwok, i abandoned https://review.openstack.org/#/c/275411/21:29
*** sdake has quit IRC21:29
*** jsavak has quit IRC21:29
*** jsavak has joined #openstack-infra21:30
clarkbianw: looks like we are just waiting to switch devstack-centos7 over to dib in ovh, know of any reason to not go ahead and do that?21:31
clarkboh you beat me to the +A21:31
ianwclarkb: i just approved that one through, looks fine.  i just got a bit confused in the stacking of the change i think21:31
jeblairclarkb: that's true, though the older hp shutdown change was materially different than what we ended up merging due to the somewhat catastrophic situation we were in.21:31
*** zaro has quit IRC21:32
anteayasmcginnis: cheesecake ha ha ha21:32
clarkbjeblair: they were functionally identical and oculd have been merged together or in steps21:32
clarkbjeblair: but the end result was the same21:32
*** Apoorva has joined #openstack-infra21:32
clarkb(no more hpcloud in nodepool.yaml)21:32
jeblairclarkb: right, this is a series of things that i thought rather carefully about and put together last week and could have been reviewed and merged at any time by people interested in it, but it has languished21:32
*** jpr has quit IRC21:33
*** jpr has joined #openstack-infra21:33
jeblairat any rate, i'm seeing more instances of us not seeing the right patches21:34
jeblairi think that's true for all of us21:34
AJaegeryolanda and myself could use some help with reviews of project-config - we have 150+ open patches but many merge conflicts...21:35
jeblairAJaeger: merge conflicts are easy, right?  ignore them :)21:35
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add windmill-jobs for ansible-role-zuul  https://review.openstack.org/27541921:35
*** zul has quit IRC21:35
ianwAJaeger: yeah, sorry, i've been slow there, and also slow on devstack since new year21:35
clarkbso I think there are a couple parts to the problem. The sheer volume involved is hard to deal with, but also everyone patches languish as a result21:35
jeblairAJaeger: and i'm going to help you by not reviewing those patches and writing zuulv3 instead so that there are far fewer of them to review in the future.21:35
ianwi feel like i might be getting to the end of fire-fighting images, however21:35
clarkbso you end up in a feedback loop of psuing the same changes21:36
*** ybathia has joined #openstack-infra21:36
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds fairy-slipper build jobs for api-site repo  https://review.openstack.org/26980921:36
*** gyee has joined #openstack-infra21:37
AJaegerjeblair: sure, merges are ignored.21:37
*** zaro has joined #openstack-infra21:37
*** zaro has quit IRC21:38
*** baoli has joined #openstack-infra21:38
greghaynesThere is also a strong negative feedback where the longer a patch sits the more effort it takes to merge (more likely to conflict, harder to fix bugs, more likely to get out of date). There is a really neat tool in russelb's reviewstats for measuring things related to that languish time...21:38
greghaynesI have used it in other projects with some success before21:38
greghayneshttp://russellbryant.net/openstack-stats/infra-openreviews.html being the link21:39
AJaegerjeblair: looking forward to zuulv3... thanks for working on it.21:39
*** zaro has joined #openstack-infra21:39
jeblairhowever, i think right now we're talking about some patches related to an effort we're all working on21:39
clarkbgreghaynes: looks like the pep8 fail for 273703 shouldn't affect the tests21:39
jeblairand we all ought to be aware of21:39
greghaynesclarkb: yea, for some reason the test is timing out21:40
openstackgerritMerged openstack-infra/project-config: Remove devstack-centos7-dib in favor of devstack-centos7  https://review.openstack.org/27416221:40
jeblairso maybe we can just try to keep this in mind that we have several people working on this and communicate a bit more21:40
jeblairmaybe we should set topics better21:40
*** zeih has quit IRC21:40
*** reed__ has joined #openstack-infra21:40
jeblairon that topic21:41
clarkbgreghaynes: I wonder if it isn't detecting the disconnect quickly enough because tcp?21:41
jeblairfungi: you may also want to review "topic:afs-mirror"21:41
*** dkranz has quit IRC21:41
fungijeblair: thanks, adding21:42
greghaynesclarkb: I think I see the issue - the logic change to deleteImage breaks the test21:42
* AJaeger waves good night...21:42
*** ihrachys has quit IRC21:42
greghaynesclarkb: the disconnect happens, we try to submit the delete job to gearman but we are disconnected so that fails so the image sits forever21:42
greghaynesjeblair: ^21:42
ianwjeblair: on the plus side, we got a bit tangled up in those image changes, but the tooling showed us the conflict, it was noticed quickly, we got on the same page and everything is moving ... in other places i've been, this would have probably required about 3 hours of meetings :)21:42
openstackgerritMerged openstack-infra/project-config: Remove unused images  https://review.openstack.org/27416421:43
*** reed__ has quit IRC21:43
clarkbgreghaynes: does it need to be a in try finally block then?21:43
openstackgerritMerged openstack-infra/project-config: Add fedora-23 to bluebox, internap, and ovh  https://review.openstack.org/27416921:43
clarkbgreghaynes: so that db is always cleaned up but we log the exceptional state21:43
greghaynesclarkb: This is a bug in the algorithm we didnt consider21:43
greghaynesclarkb: so I am less thrilled with this solution21:43
*** zeih has joined #openstack-infra21:44
*** reed_ has quit IRC21:44
greghaynesis seems like in the case of a disconnect we will always leave the image hanging on the builder21:44
greghaynesI guess we can just say we will deal with that and periodically clean up?21:44
clarkbgreghaynes: we do have the periodic cron that should resubmit the delete job if the image remains in the db21:45
greghaynesclarkb: oh. So put the delete in a finally21:45
greghaynesclarkb: yes21:45
*** EricGonczer_ has quit IRC21:45
clarkbwell if you put it in a finally then it will get removed from the db and the next cron run won't try to delte it again21:45
greghaynesclarkb: er, in an else21:46
clarkbI think you can keep the code as is since it raises and doesn't remove the image21:46
clarkbit just never submits the job which breaks the test21:46
sc68calrob: lol21:46
greghaynesclarkb: oh! the nodepool loop is not running21:48
greghaynesclarkb: that is the problem21:48
*** sputnik13 has joined #openstack-infra21:48
*** achanda has quit IRC21:48
pabelangerreview.o.o issues? Getting random 503 errors now21:48
*** EricGonczer_ has joined #openstack-infra21:48
anteayaclarkb: I think you make a good point about volume21:49
anteayapabelanger: been happening all day to folks21:50
jeblairianw: true, i am fussing about the last 3% of efficiency :)21:50
pabelangeranteaya: Ah, must be luck then. Only started for me about 5 mins ago21:50
jeblairianw, clarkb: so what is the deal with fedora23 -- do we expect it to work in the other providers?21:50
*** ayoung has quit IRC21:51
jeblairi thought the only issue was it requires dhcp, and those providers have it, so..21:51
clarkbjeblair: yes, we cannot upload anything to rackspce right now but everyone else should be fine21:51
anteayapabelanger: you win the lottery21:51
jeblairclarkb: yeah, and rax is the one without dhcp so it'd be broken anyway, right?21:51
clarkbjeblair: no, the glean stuff should work in rax now. Its purely a uploads to swift not working problem21:51
clarkband the uploads to swift not working appears to still be the same token expiration issue21:52
jeblairoh ok21:52
*** flepied1 has quit IRC21:52
jeblairclarkb: is that a shade bug?21:52
clarkbjeblair: it is a swiftclient bug that shade has not worked around21:52
jeblair(i mean, even so, bug is arguable when we're talking about the proprietary rax upload protocol)21:52
greghaynesYea, its kind of a bug in both :(21:53
clarkbit isn't failing on the proprietary bits, swiftclient as used by shade doesn't make new tokens when it needs to21:53
*** esker has joined #openstack-infra21:53
greghaynesif it is the same deal21:53
clarkbswiftclient supports a way to do that but shade wants to do it the standard way so doesn't do it he way swiftclient supports21:53
jeblairoh is this the thing notmorgan has on his plate?21:53
notmorganjeblair: yeah21:54
jeblaircool, i feel like i'm caught up now, thanks :)21:54
*** flepied has joined #openstack-infra21:54
clarkbif shade were to hand off all of the credential info instead of handing over atoken this should just work21:54
notmorganbut that is generally bad(tm) - for the sake of swiftclient it isn't awful [since it's the "right" approach for now]21:55
notmorganbut simply making swift use session and be able to be constructed with session will solve it21:55
clarkbas an aside this is another reason why consistency between clouds is nice21:55
clarkbit means we don't have to solve the same problem over and over in different code21:55
* notmorgan finishes a series of keystone cleanups and then eyes more occ/ksa work21:56
notmorgannovaclient and swift are top of my list to knock out21:56
*** erlon has quit IRC21:56
*** sdake has joined #openstack-infra21:57
clarkbhrm if we ride out the memory issues for a couple months we could upgrade gerrit to newer ubuntu lts21:57
clarkbdepending on how much lead time we think we need to give our users that might be worth considering21:57
anteayaclarkb: when is it out?21:57
clarkbanteaya: mid april21:57
notmorgananteaya: sometime around when we release ;)21:58
jeblairclarkb: last time i think we were told 3 week minimum but 2 months ideal.21:58
*** DuaneDeC7 has quit IRC21:58
*** flwang1 has joined #openstack-infra21:58
anteayaApril 21st is Xenial's release21:59
clarkbnotmorgan: out of curiousity, couldn't keystone do its own logging rather than rely on apache? and if so would that be preferable for consistency with python and the rest of openstack?21:59
anteayaMitaka is the week of April 4-821:59
*** achanda has joined #openstack-infra21:59
jeblairclarkb: we could do weekly restarts on saturday morning or something22:00
*** shashank_hegde has quit IRC22:00
notmorganclarkb: we could, in general we prefer to let apache handle anything it can esp. things like time on requests.22:00
clarkbnotmorgan: or are there issues with arbitrary threads all writing to the same location?22:00
jeblair(then we will be Real Java Admins (TM))22:00
anteayaand austin is april 25-2922:00
notmorganclarkb: it still needs to be all written to STD...Err.oroutorone of those. we wouldn't write to a file22:00
anteayawould a weekly restart get us through until April?22:00
clarkbjeblair: yes we could, we could also possibly redirect /p/ to git.o.o and reclaim all that memory22:00
notmorganclarkb: we could just make keystone log the timestamp instead of apache too.22:01
*** esker has quit IRC22:01
*** Sukhdev has joined #openstack-infra22:01
clarkbjeblair: we have 30GB total, 12GB are heap for gerrit, ~10GB is caches and the rest is apache serving git repos iirc22:02
*** esker has joined #openstack-infra22:02
notmorganso the apache log would just not apply apache timestamp and keystone could add it, if milliseconds are really thar important it wouldn't be hard to swap that around22:02
*** rossella_s has quit IRC22:02
fungii think the latency difference between gerrit's local and ssh replication is likely irrelevant to most uses of /p (where are we still hitting that in our automation?)22:02
*** trown is now known as trown|outttypeww22:02
*** rossella_s has joined #openstack-infra22:03
notmorgani'd go to %M as the formatter in ErrorLogFormat instead of %{c}t %M22:03
clarkbnotmorgan: I am not sure they are that important, just trying to figure out if we can be consistent22:03
anteayaand does 2.11 work on Xenial?22:03
*** shashank_hegde has joined #openstack-infra22:03
fungijeblair: i like the binary wheel_keytab in hiera. that's convenient (also small!)22:04
*** jsavak has quit IRC22:04
jeblairfungi: yeah, and hieraedit just dtrt when you give it a binary file22:05
*** edmondsw has quit IRC22:05
clarkbanteaya: it should, we already install libs from not the distro when we have to22:05
notmorganclarkb: i'll defer to you, sdague, and -qa folks on that front. i don't care either way. I prefer apache to do any lifting it can. but the next step is move to uwsgi - so we might want keystone to do the timestamp anyway22:05
fungijeblair: hooray for pyyaml making sane choices?22:05
jeblairfungi: yep!22:05
anteayaclarkb: wonderful, I hope it does, the question just floated through my mind22:05
openstackgerritMerged openstack-infra/system-config: Added conditional install statements for kernel header packages  https://review.openstack.org/27320522:06
*** dkranz has joined #openstack-infra22:07
dims_fungi : when you get a chance a pypi upload failed for osprofiler. can you please peek to see how we could fix it? http://logs.openstack.org/7a/7a428885ddb138f3d3a388e72bca276cfa2a5299/release/osprofiler-pypi-both-upload/5c2c80a/console.html#_2016-02-02_22_01_26_31422:07
clarkbdims_: that looks like pypi simply had an error. Our best bet is to re run it22:08
clarkbmaybe check with dstufft to see if things aren't in an unhappy state pypi side22:09
clarkbhttps://status.python.org/ claims it is happy22:10
*** piet has quit IRC22:10
openstackgerritMerged openstack-infra/system-config: Added afs to wheel mirror slaves  https://review.openstack.org/27305422:10
clarkbdims_: I can go ahead and requeue that job22:11
fungiclarkb: won't work22:11
*** jsavak has joined #openstack-infra22:11
fungifrom what dstufft has described in the past, fastly sometimes returns an internal server error on the post even though the upload succeeded22:12
fungiwhich seems to be the case here22:12
ianwjeblair: i expect f23 to work22:12
fungiclarkb: the job will just fail again if you rerun it, but this time because pypi will refuse to let you replace the wheel that's already there22:12
clarkbdoes that mean we just need to upload the sdist?22:12
fungiyeah, i'm about to do that22:12
fungiwas discussing it in -release22:13
clarkbwe could try adding a if 500 then get and if get succeeds then continue step22:15
clarkbelse fail22:15
openstackgerritOndřej Nový proposed openstack/requirements: Bump PyECLib version to 1.0.8  https://review.openstack.org/27518122:15
fungiclarkb: yep, it's been suggested in the past. nobody has found the time to add it yet (or maybe they have and it's lost in our review backlog)22:16
*** esker has quit IRC22:16
clarkbthough its not as simple as just checking name/version since you have to check the whl/sdist path directly22:17
clarkbbut you can use pip for that :)22:17
fungiit may also get more ugly when we start uploading signatures along with wheels and tarballs22:17
clarkbI think pip can check all that for us but I am reading its help output22:18
funginot sure if it's possible for the signature to get uploaded and not the thing that's signed, and what order twine puts them in, and which pypi will let you reupload and which it won't22:18
clarkbapparnelty --no-install is deprecated22:19
clarkbbut can use --download22:20
StevenKpip 8 has 'pip download' too22:20
clarkbpip install --no-deps --download '$name==$version' check the return code and branch on that22:21
fungiheads up: i've approved 270979 just now and will try to keep an eye out for potential apache issues on our mirrors (note that the commit message isn't completely correct--it adds some apache rewrites but not a vhost)22:22
*** rfolco has joined #openstack-infra22:22
*** sdake has quit IRC22:24
*** piet has joined #openstack-infra22:24
*** markmcclain has quit IRC22:25
openstackgerritDina Belova proposed openstack/requirements: Change OSprofiler constraints to >=1.0.0  https://review.openstack.org/27542622:25
*** sc68cal has quit IRC22:25
*** sdake has joined #openstack-infra22:26
*** markmcclain has joined #openstack-infra22:26
dims_clarkb : thanks for looking into it with fungi22:28
anteayafungi: ack22:29
dims_jeblair : dhellmann and i were wondering if there's a way for that job to report failure on the -release irc channel and fungi suggested "irc reporter" for zuul (after zuul v3 gets real)22:29
*** zul has joined #openstack-infra22:29
jeblairAJaeger: when you have a moment, can you look at http://specs.openstack.org/openstack-infra/infra-specs/specs/doc-publishing.html starting at "The openstack-manuals repo builds multiple manuals" and tell me what the current situation looks like there?22:30
anteayadims_: there is the openstackrecheck bot22:30
anteayadims_: that is active in -qa, would it do what you want?22:30
*** annegentle has quit IRC22:30
clarkbwait --download runs setup.py22:30
*** achanda has quit IRC22:30
clarkbso that won't work22:30
jeblairdims_: i think it's a swell idea22:31
*** e0ne has joined #openstack-infra22:31
*** hashar has quit IRC22:31
SergeyLukjanovfungi hi, sorry, I've been unable to join todat22:31
SergeyLukjanovfungi hi, sorry, I've been unable to join today22:31
fungianteaya: the recheck bot watches the gerrit event stream. release job results don't report into gerrit, so it would never see them22:32
fungiat least i think it's watching gerrit22:32
anteayaah sorry22:32
fungii need to confirm my recollection now22:32
mtreinishfungi: it is22:32
*** samuelBartel has joined #openstack-infra22:32
fungiconfirmed! ;)22:32
clarkbtime to check pip download22:33
SergeyLukjanovfungi, I can do renaming in PT time zone Mon-Fri22:33
*** EricGonczer_ has quit IRC22:33
openstackgerritgreghaynes proposed openstack-infra/nodepool: Test for gear disconnect with queued builds  https://review.openstack.org/27370322:33
anteayaSergeyLukjanov: what is PT time?22:33
fungii'm guessing pst/pdt22:34
anteayaI was guessing the same22:34
*** jsavak has quit IRC22:34
anteayawanted to confirm22:34
fungiSergeyLukjanov: sounds like next friday (the 12th?) would be good so that we have more root admins around, since this will be our first rename run after the gerrit 2.11 upgrade22:34
*** shashank_hegde has quit IRC22:35
fungino idea what might need adjusting in our process for 2.1122:35
*** eharney has quit IRC22:35
clarkbha pip download has no no use wheel option22:36
clarkbpip install runs setup.py even with --download22:36
SergeyLukjanov(me calculating tz)22:36
clarkbso maybe using pip to hceck this isn't a good idea22:36
SergeyLukjanovfungi, yeah Friday Feb 12 works good for me22:36
SergeyLukjanovanteaya yeah, I mean PST (that is PT/PST)22:37
*** zeih has quit IRC22:37
anteayaSergeyLukjanov: oh okay yeah pacific time22:37
fungiawesome, i'll bring that up again on tuesday to confirm with everyone22:37
SergeyLukjanovfungi cool, looking forward for it22:38
clarkbfungi: dims_ I thougt pip would make this easy but it builds in too many assumptions around what you want to do with the downloaded files. Our best bet is likely an explicit curl/wget based on the path that things should end up at22:38
fungiwhich unfortunately is a non-guaranteed implementation detail of pypi22:39
fungiso could change at any time22:39
*** thorst has quit IRC22:39
fungibut may be stable enough to use as a safety harness i guess22:39
clarkbI think the /simple index has been fairly consistent22:39
fungiworst case it falls back to the current state22:40
anteayaSergeyLukjanov: so are you in pacific time permanently now?22:40
clarkblet me whip up a thing for that22:40
*** bexelbie_ has joined #openstack-infra22:40
fungiso if we notice that we're back to running into this set of symptoms again, then we can surmise pypi has changed its /simple structure i guess22:41
*** jamielennox|away is now known as jamielennox22:41
*** thorst has joined #openstack-infra22:41
fungioh, and you're right, enough stuff depends on /simple that there very well may be an api guarantee around its structure22:41
*** zul has quit IRC22:42
*** e0ne has quit IRC22:43
anteayadid anyone get a chance to check why so very many bluebox nodes are in error state?22:44
*** armax_ has joined #openstack-infra22:44
*** annegentle has joined #openstack-infra22:44
*** armax has quit IRC22:45
*** armax_ is now known as armax22:45
fungii started looking at the graphs for it when you first mentioned it in here, but have been head-down on other stuff22:45
fungiit looks from grafana like stuff was still running there, but i may have misinterpreted22:46
*** thorst has quit IRC22:46
anteayaI understand22:46
fungioh, the "error node launch attempts" graph22:47
fungiyeah, that's pretty gnarly22:47
zaroclarkb: was gerrit resarted?22:48
clarkbzaro: I don't think so22:49
openstackgerritClark Boylan proposed openstack-infra/project-config: Check twine uploads for success  https://review.openstack.org/27543122:49
clarkbI havne't been in a good place to do that with babies and nodepool distracting me22:49
zaroclarkb: is the 503 issue happening now or has it died down?22:49
anteayazaro: clarkb I haven't seen anyone announce that they did a restart22:49
fungianteaya: it looks from http://grafana.openstack.org/dashboard/db/nodepool-blue-box like there was some sort of state change in bbox around 22:00 utc too22:50
anteayazaro: it is pretty constant, if you look at the channel logs22:50
dstufftclarkb: fungi y'all know twine has a --skip-existing thing right?22:50
fungidstufft: NO!22:50
*** bhunter71 has joined #openstack-infra22:50
fungihow did i miss that?22:50
anteayafungi: looking at the test nodes graph, that looks interesting22:51
*** amitgandhinz has quit IRC22:51
* fungi makes a note to rtfm more often22:51
anteayazaro: also people report 502 and 503 after a restart22:51
anteayazaro: so not all instances are due to GC22:51
dstufftfungi: it's new as of 1.6 (released 2015-09-14)22:52
*** piet has quit IRC22:53
dims_fungi : dstufft is one step ahead :)22:53
*** piet has joined #openstack-infra22:53
openstackgerritSamuel BARTEL proposed openstack-infra/project-config: add project poppy-ui a Horizon Plugin to work with Poppy's CDN API  https://review.openstack.org/26780622:53
dstufftwasn't me! I do very little with twine now adays, that's mostly sigmavirus24_awa :]22:53
*** ayoung has joined #openstack-infra22:53
dstufftfungi: you can't upload a .asc after the fact and you can't reupload a file with an already used name.22:54
clarkbneat, I think we still want to check if the upload succeeded indepdnently so that we can upload the sdist after whl22:54
clarkbbut if we add that then we can rerun at will22:54
dims_thanks sigmavirus24_awa :)22:54
dstufftI think the latest twine always uploads the wheel first if you pass it all in one invocation22:54
clarkbdstufft: we currently do it in two22:54
dstufftyea, also from 1.6.0 -> :feature:`106` Upload wheels first to PyPI22:55
*** dimtruck is now known as zz_dimtruck22:56
dstufftlegacy pypi is not so good with transaction management :[22:56
prometheanfireianw: for some reason I thought that basename took the full path, not just a string22:56
*** lucasagomes_ has quit IRC22:56
clarkbdstufft: do you know if pypi 500s on the first upload if the second one will be attempted in one invocation?22:56
anteayaI'm still working on a cold and am calling it a night, thanks for the day, hope I was helpful, if not there's always tomorrow22:56
clarkbdstufft: it may be best for us to continue to use two simply to work around that22:56
*** dingyichen has joined #openstack-infra22:57
dstufftclarkb: it does not appear if the second one will be attempted22:57
*** shashank_hegde has joined #openstack-infra22:58
ianwprometheanfire: yeah, it's just glorified str.split("/")[-1] :)22:58
*** tiswanso has quit IRC22:59
mgagnefungi our new default security group rules will be effective today for new users. thanks for the feedback =)23:00
fungimgagne: thanks for listening and taking action!23:00
*** thorst has joined #openstack-infra23:00
jeblairthis deserves a cookie!23:01
jeblairme gets cookies23:01
*** lucasagomes has joined #openstack-infra23:03
smcginnisanteaya: Thanks! ;)23:04
*** dizquierdo has joined #openstack-infra23:05
*** dkranz has quit IRC23:05
*** jamielennox is now known as jamielennox|away23:05
*** thorst has quit IRC23:05
clarkbI can restart gerrit now if we are ready to do that23:05
jeblairnodepool is very sad23:05
zaroanteaya: those errors indicate that gerrit is too slow to process requests.  the slowness can be caused by many things, one of them being the GC.23:06
jeblairprobably because of the label deletion23:06
zaroso yes, you might be right23:06
prometheanfirethought I did a git review23:07
ianwjeblair: sad like deleting everything under / sad?23:07
prometheanfirenow I did23:07
clarkbjeblair: yes we removed ubuntu-trusty-check23:07
clarkband the statsd key gen is exploding on it23:07
clarkba restart should correct it23:07
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add support for OpenRC to dib-init-system  https://review.openstack.org/27377223:07
jeblairi thought i wrote a patch to fix that23:07
fungiclarkb: gerrit restart now seems reasonable23:07
clarkbKeyError: 'nodepool.label.ubuntu-trusty-check.nodes.delete'23:07
clarkbwill wait for nodepool to be happy so it is only one thing at a time23:08
fungijeblair: you may have written a patch to fix that and it's just not reviewed :/23:08
jeblairheh, i think it landed, however, it may not be running23:08
fungieasier solution then... nodepool restart time23:08
zaroclarkb: is 'gerrit show-queue' showing a really full queue?23:09
*** peristeri has quit IRC23:09
jeblairrather than try to figure out if it's actually running, which is really hard because of all the local installs i did last week, i'm just going to restart it and if we see it again, take action23:09
*** unicell1 has joined #openstack-infra23:09
clarkbzaro: 5 tasks23:10
*** unicell has quit IRC23:10
*** jpr has quit IRC23:10
*** thorst has joined #openstack-infra23:10
jeblairnope still broken23:11
zaroclarkb: lots of discussion point to gerrit not being able to process queue and then return 503. but loooks like that's definately not case ehre.23:11
*** thorst_ has joined #openstack-infra23:11
clarkbzaro: I think what happens is gerrit goes to allocate more memory and that triggers long and slow gc while that happens apache times out and we 50023:11
clarkbzaro: the more requests we have coming in the more things need memory the worse it gets23:11
clarkbzaro: and apache will cache that 500 error for an interval as well so once it happens many people get that result23:12
*** salv-orlando has quit IRC23:13
*** salv-orlando has joined #openstack-infra23:13
prometheanfireianw: don't want to exit 1 though since that will break other OS's23:14
clarkbjeblair: you ok with me restarting gerrit with nodepool unhappy or should I wait?23:15
*** thorst has quit IRC23:15
jeblairclarkb: go for it23:15
clarkbok restarting now23:16
ianwprometheanfire: hmm, won't that happen anyway?23:16
jeblairclarkb: i just did "nodepool delete 7616039 --now" to delete the one node in that image type23:16
prometheanfireah, you are right, after the edit to remove the other init systems that will happen23:16
zaroclarkb: i think gerrit just died23:16
prometheanfireianw: want me to add the other init's back and add a *) at the end?23:16
clarkbzaro: I just restarted it23:16
prometheanfireI think it could just be debian) ;;23:17
prometheanfireshould be obvious as a noop I hope23:17
ianwprometheanfire: i think so, just have them as blanks.  if we ever get another init system (god i hope not) it should break with a helpful message at that23:17
*** thorst_ has quit IRC23:17
*** tzn has quit IRC23:18
prometheanfirefedora/rh might want this functionality as they don't enable services by default iitv23:18
ianwno, they don't but i think most places we add the service, we enable it there23:18
*** thorst has joined #openstack-infra23:18
clarkbanteaya: the bluebox issues may be related to floating ip exhaustion23:18
ianwglean for example23:18
*** e0ne has joined #openstack-infra23:19
mgagneclarkb you can speed up recovery with "ProxyPass retry=3" in proxy config for gerrit23:19
prometheanfireah, guess I just generified all those specifics23:19
*** baoli has quit IRC23:20
*** dizquierdo has quit IRC23:20
mgagneclarkb works when backend is down then back up, I'm not sure what Gerrit actually returns in the current use case23:20
clarkbmgagne: I am not sure we want to increase the number of requests once gerrit has decided it needs a break23:20
*** rfolco has quit IRC23:20
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add support for OpenRC to dib-init-system  https://review.openstack.org/27377223:20
prometheanfireianw: try that one on for size23:20
* mgagne hammers F523:20
clarkbI worry we will stampede it with requests when it is least happy so the current behacior wile annoying is probably sane23:21
*** regXboi has quit IRC23:21
*** ok_delta has joined #openstack-infra23:21
*** ok___delta has joined #openstack-infra23:21
*** shashank_hegde has quit IRC23:21
clarkbwe have 55 floating ips allocated in bluebox23:22
clarkbmax servers is 3923:22
clarkbwe likely have a leak, I will work on cleaning that up23:23
ianwprometheanfire: before jenkins gets to far, want to just s/DEST=//23:23
prometheanfirethought I did that there23:23
prometheanfireoh, missed it23:23
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add support for OpenRC to dib-init-system  https://review.openstack.org/27377223:23
fungiclarkb: thanks, that might explain the error conditions in the bbox nodepool graphs23:25
*** xyang1 has quit IRC23:25
ianwprometheanfire: ok, LGTM, let jenkins do its thing23:25
prometheanfirejust verified ) ;; works as noop too23:26
*** bexelbie has quit IRC23:26
*** bexelbie_ is now known as bexelbie23:26
fungiamusing. i need to install bzr to be able to run pip's unit tests23:27
*** thorst has quit IRC23:27
prometheanfirefungi: try fossil out23:27
fungithe most amusing part is i'm not sure how i didn't already have bzr installed23:27
prometheanfirewe were using cvs til about 5-6 months ago23:28
*** e0ne has quit IRC23:28
fungiprometheanfire: i don't actually need to interact with bzr. pip is maintained via git. but one of its unit (i use the term loosely) tests wants to create a test package and does something with bzr during the course of that23:29
openstackgerritMerged openstack-infra/irc-meetings: Drop I18N Translation test meeting  https://review.openstack.org/27504523:29
openstackgerritMerged openstack-infra/system-config: Added AFS keytab to wheel slave  https://review.openstack.org/27320623:29
fungiso when running tox -e py27 on a commit for pip, it broke complaining tha it couldn't run bzr23:29
*** annegentle has quit IRC23:30
dstufftfungi: I don't even run the tests locally :[23:30
dstufftunless something is failing23:30
fungioh, heh23:30
dstufftthen I'll do tox -e pywhatever -- -k <testname>23:30
dstufftto run just that test23:30
dstufftnot gonna trick me into installing bzr that easily23:30
fungiwell, in this case i'm trying to tweak my pull request and confirm tests actually pass23:30
openstackgerritMerged openstack-infra/project-config: Make murano tempest job voting and add it to gate pipeline  https://review.openstack.org/27519123:31
openstackgerritMerged openstack-infra/project-config: Add windmill-jobs for ansible-role-zuul  https://review.openstack.org/27541923:31
fungioh, i get you/ you're saying just try rerunning the tests that didn't pass in the ci23:31
openstackgerritMerged openstack-infra/project-config: Test translation imports  https://review.openstack.org/27435923:31
fungimeh, now i actually want to see what's involved in running them ;)23:31
mgagneany updates/news on image update random slowness?23:32
openstackgerritMerged openstack-infra/system-config: Added wheel hosting for AFS mirror slave  https://review.openstack.org/27097923:32
openstackgerritMerged openstack-infra/system-config: Add AFS and Admin keytab to release slave.  https://review.openstack.org/27353823:32
*** krotscheck1 has joined #openstack-infra23:32
dstufftfungi: you might want pytest-xdist and -n 8 or so23:32
dstufftotherwise they take a long time23:32
*** gordc has quit IRC23:33
clarkbfloating IPs are deleting23:33
dstufftI think over an hour if you run the serially23:33
dstufftmaybe less now adays, we started doing some things better23:33
krotscheck1Wow, wheel things are merging!23:33
fungidstufft: cool. i'll check the stats and find out. i have a lot of balls in the air anyway so no biggie if i work on something else while this runs23:33
fungikrotscheck1: yep, brace for impact!23:33
clarkbmgagne: I am not sure I know specificaly what you refer to but we need to get the change that detects gearman client disconnects in and we need to figure out rax uploads once our token expires23:34
* krotscheck1 holds on tight!23:34
krotscheck1ok, so all the puppet things are there. has te slave been built yet?23:35
mgagneclarkb I think jeblair has details. It was a problem he had last week IIRC with random upload speed at OVH or some other providers.23:35
*** lucasagomes has quit IRC23:35
*** shashank_hegde has joined #openstack-infra23:35
prometheanfirehold onto your butts23:36
mgagneclarkb I'm interested in the details because we found similar symptoms on our side. I wish to check if it's caused by the same problem (which we don't know the source yet)23:36
jrollprometheanfire: http://butt.holdings23:38
*** samuelBartel has quit IRC23:39
prometheanfirejroll: this makes me sooo happy23:39
jrollI'll show you my other favorite abuse of new TLDs on friday :P23:39
*** lucasagomes has joined #openstack-infra23:40
clarkbmgagne: for that I am not sure I must've been traveling when it was discussed23:41
*** dizquierdo has joined #openstack-infra23:42
*** bexelbie- has joined #openstack-infra23:44
sridhar_ram1infra team - looking for you help. dvsm functional jobs are failing for the tacker project, see for e.g.  https://review.openstack.org/#/c/231280/23:44
*** shashank_hegde has quit IRC23:44
*** tzn has joined #openstack-infra23:44
sridhar_ram1This seems to have started three days back after the merge of this project-config check-requirements patchset to tacker-horizon https://review.openstack.org/#/c/271980/23:44
*** bexelbie has quit IRC23:44
*** bexelbie- is now known as bexelbie23:44
*** tzn has quit IRC23:45
sridhar_ram1The actual error looks like "Could not satisfy constraints for 'horizon': installation from path or url cannot be constrained to a version"23:45
*** Sukhdev has quit IRC23:45
*** ybathia has quit IRC23:45
clarkbyup thats the problem23:46
sridhar_ram1clarkb: can you parse this mesg for me please ? :)23:46
clarkbsridhar_ram1: https://git.openstack.org/cgit/openstack/tacker-horizon/tree/test-requirements.txt#n12 is what it is mad about23:48
clarkbpip is using constraints when installing and apparently lines like that are not allowed23:48
*** sdake has quit IRC23:48
*** shashank_hegde has joined #openstack-infra23:49
sridhar_ram1clarkb: I see. So those two tarballs lines need to go ?23:49
*** Sukhdev has joined #openstack-infra23:49
sridhar_ram1clarkb: cool.. will take care. Thanks a lot!23:49
*** sridhar_ram1 is now known as sridhar_ram23:50
*** bpokorny_ has joined #openstack-infra23:50
fungikrotscheck1: no slave yet, we're getting there though23:50
armaxanteaya: ping23:50
*** ok_delta has quit IRC23:50
fungiarmax: i think she retired for the night--wasn't feeling well23:51
*** ok___delta has quit IRC23:51
armaxfungi: ah, thanks for letting me know23:51
armaxI’ll try tomorrow23:51
openstackgerritJames E. Blair proposed openstack-infra/git-restack: Correct metadata URLs  https://review.openstack.org/27544423:51
openstackgerritJames E. Blair proposed openstack-infra/git-review: Correct metadata URLs  https://review.openstack.org/27544523:52
*** bpokorny has quit IRC23:53
*** boris-42 has quit IRC23:53
fungidstufft: on a successful run, i got "613 passed, 10 skipped, 1 xfailed, 1 xpassed, 1 warnings in 1371.68 seconds"23:54
fungidstufft: so really didn't take very long23:54
dstufft22 minutes is pretty long tbh23:55
*** jpr has joined #openstack-infra23:56
*** gildub has joined #openstack-infra23:58
*** dizquierdo has quit IRC23:59
fungifaster than nova's unit tests23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!