Tuesday, 2015-03-17

SpamapSsendto(5, "\0REQ\0\0\0\1\0\0\0\6really", 18, MSG_NOSIGNAL, NULL, 0) = 1800:00
SpamapSsendto(5, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 1200:00
SpamapSSo that second packet is the one that fails00:00
*** hrubi has joined #openstack-infra00:00
*** baoli has quit IRC00:00
greghaynesactually, we can just parse the venv requirements thing in devstack into a big source-repositories file and let dib take care of the downloading00:00
*** liusheng has quit IRC00:00
*** baoli has joined #openstack-infra00:00
*** liusheng has joined #openstack-infra00:00
*** r-daneel has joined #openstack-infra00:02
SpamapS  GEARMAN_COMMAND_GRAB_JOB_ALL,          /* W->J -- */00:06
SpamapSlooks like gearman's protocol expanded since gear was written00:06
*** tonytan4ever has quit IRC00:07
clarkbSpamapS: http://gearman.org/protocol/ no they didnt :P00:08
*** gokrokve has joined #openstack-infra00:08
clarkbthats annoying :/00:08
SpamapSclarkb: yeah it looks like a bunch of stuff has landed.. and for even more fun, they still call it "1.0"00:09
SpamapSmap/reduce related looksl ike00:09
*** angela-s has left #openstack-infra00:09
SpamapSit looks quite poorly documented00:10
SpamapSbut the C worker uses GRAB_JOB_ALL by default now00:10
jogogreghaynes: would using pypi-mirror mean we stop using bandersnatch?00:10
* SpamapS realizes it is very late and his children probably miss him00:10
clarkbSpamapS: thats amazingly broken00:10
* SpamapS will bbl00:10
clarkbSpamapS: also go get children00:10
SpamapSclarkb: yeah it's quite sad they'd land stuff w/o docs00:10
SpamapSI've been hunting down docs on the REDUCE stuff for a while now :-P00:11
* SpamapS disappears00:11
clarkbjogo: no00:11
*** erlon has quit IRC00:11
clarkbbandersnatch would be http://mirror.org/simple and wheels would be http://mirror.org/wheels/distro/release/arch or whatever00:11
jogoohhh nice00:11
greghaynesjogo: yep, im just rsyncing over a bunch of wheels and then via the magic of pip extra-index-url we overloay that ontop of the bandersnatch mirror00:13
greghaynes(yep in reference to what clarkb said)00:13
jrollis everyone else getting dcc spam from BlackMustard ?00:13
jogogreghaynes: awesome!00:14
dstufftby the power of greysk--- extra-index-url!00:14
clarkbjroll: no00:14
jrolltons of people are in #freenode, might be worth banning him from openstack channels00:14
*** banix has joined #openstack-infra00:14
openstackgerritIan Wienand proposed openstack-infra/system-config: Add nodepool config-validation environment  https://review.openstack.org/16490100:16
jogogreghaynes: sounds surprisingly fairly straight forward :)00:16
greghaynesjogo: thats whats scary00:16
clarkbjroll: does /ignore cover that in your client?00:16
clarkbjroll: I would suggest you do that instead00:16
jrollclarkb: I can handle myself, just suggesting opportunistically as he seems to be going through channels hitting people with a dcc for SvcHost.exe00:17
*** yamamoto has quit IRC00:18
*** yamamoto has joined #openstack-infra00:18
jrolljust from #freenode I've seen: ##econometrics, ##aws, ##c++-basic, ##infra-talk00:18
*** amotoki has joined #openstack-infra00:19
*** ghostpl_ has quit IRC00:23
*** hdd has quit IRC00:24
fungifwiw i see that all the time from random spambots. they lurk in a channel then /msg or /dcc anybody they see active00:25
*** dizquierdo has quit IRC00:25
fungiit's been commonplace for a few years now00:25
openstackgerritIan Wienand proposed openstack-infra/project-config: Add nodepool config check  https://review.openstack.org/16490400:25
fungiever since they realized that spamming in channels was a quick way to get squashed00:25
*** emagana has quit IRC00:26
*** banix has quit IRC00:26
*** BlackMustard has quit IRC00:27
jogogreghaynes: currently nova's pep8 job takes about 10 minutes at least 8 of those minutes are on installing things. On my local box where I have all the wheels pre-built it takes about00:28
jogo(t'ox -r -epep8' that is)00:28
*** yamamoto has quit IRC00:28
*** juggler has joined #openstack-infra00:29
jogogreghaynes: just under 2 minutes00:29
*** koolhead17 has quit IRC00:30
greghaynesjogo: tripleo was similar, which is why I have the local pypi-mirror install and like keeping it alive00:30
jogohehe nice00:34
lifelessjogo: keystoneclient patch up00:36
*** claudiub has quit IRC00:36
*** shashankhegde has joined #openstack-infra00:36
*** otter768 has joined #openstack-infra00:36
*** nanliu has joined #openstack-infra00:36
jogolifeless: link?00:37
lifelessjogo: its not really a big win though - 220ms down to 160ms.00:37
*** r-daneel has quit IRC00:37
lifelessjogo: https://review.openstack.org/#/c/164066/200:37
jogoahh I see00:37
*** signed8bit_ZZZzz is now known as signed8bit00:37
lifelessits worth doing00:38
lifelessbut its not the end of the story00:38
jogoso take a look at OSC00:38
jogoits even worse00:38
lifelessdo I have to?00:38
jogoit loads all the clients every time I htink00:38
jogolifeless: lol, if you want to be surprised at slow00:38
*** reed has quit IRC00:39
openstackgerritNan Liu proposed openstack-infra/puppet-elasticsearch: Use archive to fetch/checksum instead of wget.  https://review.openstack.org/16490800:39
*** shashankhegde has quit IRC00:39
fungiit loads all the clients every time you think? i guess i didn't notice since i don't think very often00:39
jogofungi: I think so, have to double check. it certainly feels that way00:39
*** xyang1 has quit IRC00:40
lifelessjogo: been there, optimised that - seriously, very happy to share lessons learnt and so forth, but I am much more concerned about our server side effort atm - optimising the client side load experience is straight forward (if tedious)00:40
jogofungi: https://bugs.launchpad.net/python-openstackclient/+bug/1431649/comments/100:40
openstackLaunchpad bug 1431649 in python-keystoneclient "openstackclient is really slow" [Undecided,In progress] - Assigned to Robert Collins (lifeless)00:40
*** otter768 has quit IRC00:40
lifelessjogo: oh btw see the mailing list thread I started for a better timeit fragment00:41
*** Longgeek has joined #openstack-infra00:41
*** stevemar has joined #openstack-infra00:43
jogoI saw but haven't read it yet00:43
jogowill read it later tonight or tomorrow00:43
*** JohnD has joined #openstack-infra00:44
*** kmartin has quit IRC00:47
*** achanda_ has quit IRC00:48
*** nanliu has quit IRC00:50
*** bdpayne has quit IRC00:50
*** salv-orlando has joined #openstack-infra00:50
*** ashleighfarnham has joined #openstack-infra00:51
*** JohnD has quit IRC00:52
*** sputnik13 has quit IRC00:52
*** banix has joined #openstack-infra00:56
*** Qiming has joined #openstack-infra00:56
*** ashleighfarnham has quit IRC00:57
*** Qiming_ has joined #openstack-infra00:57
*** yamamoto has joined #openstack-infra00:59
*** Qiming has quit IRC01:01
*** Longgeek has quit IRC01:01
*** Longgeek has joined #openstack-infra01:01
*** Qiming__ has joined #openstack-infra01:02
*** Qiming__ is now known as Qiming01:02
*** dmellado has quit IRC01:03
*** yolanda has quit IRC01:04
*** garyh has joined #openstack-infra01:05
*** Qiming_ has quit IRC01:05
*** JohnD has joined #openstack-infra01:07
*** ddieterly has quit IRC01:07
*** ddieterly has joined #openstack-infra01:08
*** woodster_ has quit IRC01:10
*** garyh has quit IRC01:10
*** tsg has joined #openstack-infra01:11
*** Qiming_ has joined #openstack-infra01:12
*** tnovacik_ has quit IRC01:12
*** Qiming has quit IRC01:15
*** tsg has quit IRC01:18
*** tiswanso has joined #openstack-infra01:19
*** funzo has quit IRC01:20
*** otter768 has joined #openstack-infra01:21
*** tiswanso_ has joined #openstack-infra01:25
*** yolanda has joined #openstack-infra01:25
*** dmellado has joined #openstack-infra01:26
*** dims has quit IRC01:26
*** tiswanso has quit IRC01:28
*** JohnD has quit IRC01:30
*** banix has quit IRC01:31
tchaypolifeless: do you mean "warning about __init__ importing modules - fast CLIs”? I haven’t seen any other mentions of timeit01:31
*** liusheng has quit IRC01:31
*** JohnT has joined #openstack-infra01:32
tchaypooh, there’s the fragment01:32
*** liusheng has joined #openstack-infra01:32
*** sputnik13 has joined #openstack-infra01:34
*** shashankhegde has joined #openstack-infra01:34
*** dmellado has quit IRC01:35
*** yolanda has quit IRC01:35
*** yolanda has joined #openstack-infra01:36
*** spzala has quit IRC01:38
*** yolanda has quit IRC01:41
*** Longgeek has quit IRC01:41
*** yolanda has joined #openstack-infra01:42
*** banix has joined #openstack-infra01:42
*** woodster_ has joined #openstack-infra01:43
lifelesstchaypo: yes01:44
*** sputnik13 has quit IRC01:44
*** mfink_ has joined #openstack-infra01:46
*** yolanda has quit IRC01:49
*** Longgeek has joined #openstack-infra01:49
asselinanteaya, rhe00 mtreinish FYI, I wasn't impacted by the subunit2sql databases b/c I excluded them. I overwrite this file to an empty one: nodepool/elements/cache-devstack/install.d/99-cache-testrepository-db and I needed to use my own prepare_node.sh scripts (past tense b/c i'm fully dib now)01:51
*** mfink_ has quit IRC01:52
*** mfink_ has joined #openstack-infra01:52
asselinanteaya, rhe00 I should probably add some documentaiton on that. I only have it in the comments now: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/manifests/master.pp#L28901:52
SpamapSwow I've forgotten everything I knew about bzr01:53
anteayaasselin: thank you for following up on that01:54
*** dims has joined #openstack-infra01:54
anteayaasselin: I appreciate it01:54
*** ddieterl_ has joined #openstack-infra01:54
*** tqtran has quit IRC01:54
*** ddieterly has quit IRC01:55
SpamapSclarkb: btw, https://bugs.launchpad.net/gearmand/+bug/92346301:55
openstackLaunchpad bug 923463 in Gearman "GRAB_ALL undocumented and not supported by older versions of server" [Low,Confirmed] - Assigned to Brian Aker (brianaker)01:55
SpamapSclarkb: Found the commit where libgearman was broken to send that by default01:56
*** sputnik13 has joined #openstack-infra01:57
SpamapSterrible commit message.. no idea why that warranted using undocumented protocol features01:57
*** nanliu has joined #openstack-infra01:57
asselinanteaya, there's a few dib files I had to exclude b/c they need access to openstack systems. I will try to document (the moving target). the good news is dib makes it very easy to exclude otherwise work-around.01:58
anteayaasselin: nice01:58
*** baoli has quit IRC01:58
anteayaasselin: it is good to have more background, thank you01:58
*** baoli has joined #openstack-infra01:59
openstackgerritNan Liu proposed openstack-infra/puppet-elasticsearch: Use archive to fetch/checksum instead of wget.  https://review.openstack.org/16490801:59
anteayaasselin: and thanks for all your work making your scripts consumable02:00
*** yolanda has joined #openstack-infra02:00
*** bhunter71 has joined #openstack-infra02:01
*** ddieterly has joined #openstack-infra02:01
*** ddieterl_ has quit IRC02:01
*** dmellado has joined #openstack-infra02:01
*** dims has quit IRC02:03
*** mpopow has quit IRC02:03
*** ivar-laz_ has joined #openstack-infra02:06
*** yamahata has quit IRC02:07
*** Darkwan has joined #openstack-infra02:08
*** asselin_ has quit IRC02:08
*** ivar-lazzaro has quit IRC02:09
*** ivar-laz_ has quit IRC02:10
*** sputnik13 has quit IRC02:10
*** salv-orlando has quit IRC02:11
*** garyh has joined #openstack-infra02:11
*** Qiming_ is now known as Qiming02:12
*** patrickeast has quit IRC02:15
*** JohnT is now known as HomeLate02:16
*** HomeLate is now known as cjh-02:17
*** cjh- is now known as cjh_02:19
SpamapSclarkb: https://code.launchpad.net/~clint-fewbar/gearmand/document-protocol/+merge/253140 btw02:19
SpamapSjeblair: ^^02:19
*** dmorita has quit IRC02:19
*** cjh_ is now known as cjha02:19
*** dmorita has joined #openstack-infra02:19
SpamapSjeblair: fyi C libgearman defaults to GRAB_JOB_ALL.. we can respond to it just fine with null reducers, and refuse SUBMIT_REDUCE_JOB's with a helpful error, something like "We don't support mapreduce"02:20
SpamapSand with that, I go to family time02:20
*** garyh has quit IRC02:21
*** jyuso1 has joined #openstack-infra02:21
*** tsg has joined #openstack-infra02:23
*** baoli has quit IRC02:23
*** baoli has joined #openstack-infra02:24
*** sigmavirus24_awa is now known as sigmavirus2402:24
*** dmorita has quit IRC02:24
*** unicell has quit IRC02:25
*** hdd has joined #openstack-infra02:27
*** coolsvap|afk is now known as coolsvap02:27
*** achanda has joined #openstack-infra02:27
*** sdake__ has quit IRC02:32
*** cjha is now known as kaneeni02:32
*** kaisers1 has joined #openstack-infra02:34
openstackgerritgreghaynes proposed openstack-infra/project-config: WIP: Create jobs for a wheel mirror  https://review.openstack.org/16492702:35
*** kaisers has quit IRC02:35
*** kaneeni is now known as somefag02:35
*** somefag has quit IRC02:35
*** JohnT has joined #openstack-infra02:36
nanliuhmm, confused how this fails precise, but passes on other platforms: https://review.openstack.org/#/c/164908/202:36
nanliuI know it's missing the archive module, not sure if system-config is the right place to add it.02:37
nibalizernanliu: it is02:38
nibalizerbut i did02:38
*** sputnik13 has joined #openstack-infra02:38
*** krtaylor has quit IRC02:38
nibalizerso you can put your change as dependent on mine with Depends-On: I91b7e35ef5e66c3538fd987d3857ec1af455beb1 at the bottom of your commit message02:38
nanliunibalizer, that pr is for puppet-git02:38
nibalizeroh we need archive02:39
nibalizerwait dont we have arrchive?02:39
nanliubut I have no idea how it's passing Jenkins on centos and trusty02:39
nibalizerwell the thing about that is our testing is kindof fail02:39
nibalizerprobably because we test node defs on their native platform02:40
*** tsg has quit IRC02:40
nibalizerso the only host that uses the module your change is in, is precise02:40
nibalizerthat Node-OS: Precise means 'test this on precise only'02:41
*** Ryan_Lane has quit IRC02:41
*** sputnik13 has quit IRC02:42
nanliuah, thanks for the tip and I'll add a license to puppet-git02:43
*** sputnik13 has joined #openstack-infra02:43
nibalizerty ty02:44
nibalizeralso i made a pr against puppet-git02:44
nibalizerprobably not up to your standards but thats a needed feature02:44
openstackgerritNan Liu proposed openstack-infra/puppet-elasticsearch: Use archive to fetch/checksum instead of wget.  https://review.openstack.org/16490802:45
*** shashankhegde has quit IRC02:49
*** krtaylor has joined #openstack-infra02:50
*** JohnT is now known as p1np01nt02:51
*** achanda has quit IRC02:52
*** shashankhegde has joined #openstack-infra02:54
*** gokrokve has quit IRC02:56
*** gokrokve has joined #openstack-infra02:57
*** gokrokve has quit IRC02:57
*** p1np01nt is now known as Exp1r3dP0ps1cle02:58
*** tonytan4ever has joined #openstack-infra02:58
*** Exp1r3dP0ps1cle is now known as d[-_-]b02:58
*** melwitt has quit IRC02:58
*** dkliban is now known as dkliban_afk03:00
*** d[-_-]b has quit IRC03:00
*** sputnik13 has quit IRC03:02
*** dims has joined #openstack-infra03:04
*** RalphWiggum has joined #openstack-infra03:08
*** RalphWiggum is now known as Guest8074903:08
*** salv-orlando has joined #openstack-infra03:11
*** dims has quit IRC03:11
*** boris-42 has quit IRC03:12
*** spzala has joined #openstack-infra03:14
*** harlowja is now known as harlowja_away03:15
*** tiswanso_ has quit IRC03:17
*** banix has quit IRC03:18
*** banix has joined #openstack-infra03:19
*** otter768 has quit IRC03:20
*** garyh has joined #openstack-infra03:22
*** signed8b_ has joined #openstack-infra03:23
*** ddieterly has quit IRC03:25
*** gokrokve has joined #openstack-infra03:25
*** signed8bit has quit IRC03:26
*** gokrokve has quit IRC03:27
*** markvoelker has quit IRC03:29
*** ghostpl_ has joined #openstack-infra03:30
*** gokrokve has joined #openstack-infra03:31
*** garyh has quit IRC03:32
*** hdd has quit IRC03:33
*** dmellado has quit IRC03:33
*** shashankhegde has quit IRC03:34
openstackgerritgreghaynes proposed openstack-infra/project-config: WIP: Create jobs for a wheel mirror  https://review.openstack.org/16492703:35
*** ghostpl_ has quit IRC03:35
*** yolanda has quit IRC03:35
*** david-lyle is now known as david-lyle_afk03:35
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Promote lbaasv2 api job to both the lbaas and neutron check and gate  https://review.openstack.org/16439003:37
*** Alexandra_ has joined #openstack-infra03:39
*** yolanda has joined #openstack-infra03:39
*** dmellado has joined #openstack-infra03:41
*** asettle has quit IRC03:41
*** tsg has joined #openstack-infra03:42
*** eharney has quit IRC03:46
*** coolsvap is now known as coolsvap|afk03:48
*** shashankhegde has joined #openstack-infra03:48
*** Alexandra_ is now known as asettle03:50
*** coolsvap|afk is now known as coolsvap03:55
*** Longgeek has quit IRC03:59
*** achanda has joined #openstack-infra04:01
*** baoli has quit IRC04:02
*** Guest80749 has quit IRC04:04
*** subscope_ has joined #openstack-infra04:04
*** RalphWiggum has joined #openstack-infra04:05
*** RalphWiggum is now known as Guest2211704:05
*** banix has quit IRC04:11
*** spzala has quit IRC04:16
*** mrmartin has joined #openstack-infra04:17
Guest22117DCC SEND STARTKEYLOGGER 0 0 004:22
*** Guest22117 has left #openstack-infra04:22
*** stevemar has quit IRC04:22
*** sarob has quit IRC04:22
*** stevemar has joined #openstack-infra04:23
*** salv-orlando has quit IRC04:23
*** sigmavirus24 is now known as sigmavirus24_awa04:24
*** ddieterly has joined #openstack-infra04:25
*** shashankhegde has quit IRC04:26
*** macjack has joined #openstack-infra04:26
*** macjack has quit IRC04:27
*** ddieterly has quit IRC04:31
*** ayoung has quit IRC04:32
*** garyh has joined #openstack-infra04:32
*** bhunter71 has quit IRC04:34
*** tsg has quit IRC04:35
*** baoli has joined #openstack-infra04:35
*** Tetris- has joined #openstack-infra04:37
*** Tetris- has left #openstack-infra04:38
*** baoli has quit IRC04:40
*** baoli has joined #openstack-infra04:40
*** baoli has quit IRC04:40
*** deepakcs has joined #openstack-infra04:41
*** oomichi has joined #openstack-infra04:41
*** oomichi has quit IRC04:41
*** garyh has quit IRC04:43
*** enikanorov has quit IRC04:43
*** enikanorov has joined #openstack-infra04:43
*** achanda has quit IRC04:46
*** achanda has joined #openstack-infra04:49
*** achanda has quit IRC04:49
*** dboik_ has quit IRC04:49
*** BharatK has joined #openstack-infra04:50
*** macjack has joined #openstack-infra04:51
*** sputnik13 has joined #openstack-infra04:55
*** Longgeek has joined #openstack-infra04:59
*** baoli has joined #openstack-infra05:01
*** baoli has quit IRC05:01
*** Longgeek has quit IRC05:04
*** emagana has joined #openstack-infra05:10
*** unicell has joined #openstack-infra05:13
*** rlucio has joined #openstack-infra05:14
*** gokrokve_ has joined #openstack-infra05:15
rluciohi guys, my ci jobs are failing with resource errors, anyone have a moment to assist?  http://paste.openstack.org/show/192799/05:18
*** gokrokve has quit IRC05:18
*** gokrokve_ has quit IRC05:19
*** salv-orlando has joined #openstack-infra05:20
*** koolhead17 has joined #openstack-infra05:20
*** otter768 has joined #openstack-infra05:21
*** sarob has joined #openstack-infra05:23
*** otter768 has quit IRC05:26
*** ddieterly has joined #openstack-infra05:27
*** sarob has quit IRC05:29
*** ddieterly has quit IRC05:31
*** sabeen1 has quit IRC05:35
*** mrmartin has quit IRC05:35
*** unicell1 has joined #openstack-infra05:37
*** unicell has quit IRC05:37
BharatKrlucio: I am expert in infra... But recently I deployed my CI.05:41
BharatKrlucio: resources means?05:41
BharatKrlucio: RAM or CPUs?05:41
*** subscope_ has quit IRC05:43
rluciobharatk: ah i get the impression it is some sort of process limit05:44
rluciobharatk: so maybe cpu?05:45
rluciobharatk: lemme check the core count on the slave, 1 sec05:45
BharatKrlucio: You are running in upstream right?05:45
BharatKrlucio: Okay05:46
BharatKrlucio: In upstream we have 8 CPUs to the slave machine to run our job.05:46
rluciobharatk: hmm, ok i think this is a  2 processors with 8 cores each05:47
rlucioxeon box05:47
rluciobharatk: i mean, its not top end but its got more power than my desktop05:48
rluciobharatk: do the jenkins jobs for ci create alot more threads for devstack than the vanilla config?05:48
BharatKrlucio: no not like that.05:49
BharatKrlucio: are you install devstack on your environment manually05:49
rluciobharatk: not sure if you saw it but i pasted the dump here: http://paste.openstack.org/show/192799/05:49
*** yamahata has joined #openstack-infra05:49
BharatKrlucio: Yeah I have seen that.05:49
rluciobharatk: no, im using the jenkins job, devstack is installed by devstack-vm-gate.sh i think05:50
rluciobharatk, its ubuntu 14.04 btw05:51
BharatKrlucio: Yes, but were you able to install devstack successfully manually on the same environment05:51
*** nanliu has quit IRC05:51
rluciobharatk: i havent tried, i can certainly run it real quick if you think that is a useful exercise05:52
BharatKrlucio: Give me a movement.05:52
*** mfink_ has quit IRC05:53
*** mfink_ has joined #openstack-infra05:53
BharatKrlucio: yeah it is worth testing manually on the same environment.05:55
BharatKrlucio: I have few steps to run tempest tests also05:55
BharatKrlucio: But in your case devstack installation itself is failing.05:55
BharatKrlucio: It's good to try to install devstack manually on the same environment.05:56
*** mfink__ has joined #openstack-infra05:56
rluciobharatk: ok i'll run it now.. btw i just checked my other slave, which is a differnt cpu config and it fails in the same place05:56
rluciobharatk: maybe some sort of weird config issue?05:57
*** mfink_ has quit IRC05:59
*** asettle has quit IRC05:59
BharatKrlucio: yeah you are right. We can confirm it running devstack manually.05:59
*** yamamoto has quit IRC06:01
BharatKrlucio: There is some problem with environment configuration06:01
*** Longgeek has joined #openstack-infra06:01
BharatKrlucio: From the logs we can figure out that while executing "os.fork()" we got Resource temporarily unavailable06:01
BharatKrlucio: that means, http://stackoverflow.com/questions/13009632/getting-strange-behavior-from-terminal-and-iterm-bash-fork-resource-temporar06:02
BharatKrlucio: It cannot allocate sufficient memory to copy the parent's page tables and allocate a task structure for the child.06:02
BharatKIt was not possible to create a new process because the caller's RLIMIT_NPROC resource limit was encountered. To exceed this limit, the process must have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability.06:02
*** baoli has joined #openstack-infra06:02
*** baoli has quit IRC06:06
*** hichihara has quit IRC06:08
*** yfried_ has joined #openstack-infra06:11
*** tonytan4ever has quit IRC06:11
*** tonytan4ever has joined #openstack-infra06:11
rluciobharatk: yea i encountered similar info earlier.. seems silly though the box lists 82483544 kB of ram06:11
rluciobharatk:bharatk: damn, devstack vanilla fails on some apache user not authorized error06:14
rluciobharatk: while uploading images to glance06:15
*** tonytan4ever has quit IRC06:15
*** mtanino has quit IRC06:21
*** sarob has joined #openstack-infra06:25
*** achanda has joined #openstack-infra06:26
*** ddieterly has joined #openstack-infra06:28
*** dboik has joined #openstack-infra06:29
BharatKrlucio: oh... :( Give another try.06:30
*** sarob has quit IRC06:30
*** baoli has joined #openstack-infra06:32
*** baoli has quit IRC06:32
*** ddieterly has quit IRC06:32
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/16495606:33
*** baoli has joined #openstack-infra06:33
*** baoli has quit IRC06:33
rluciobharatk: ok gimme a few min06:34
*** signed8b_ has quit IRC06:34
*** Longgeek has quit IRC06:35
*** Longgeek has joined #openstack-infra06:35
*** ashishb has joined #openstack-infra06:39
openstackgerritQiming Teng proposed openstack-infra/project-config: new-project: stackforge/python-senlinclient  https://review.openstack.org/16496306:41
*** gokrokve has joined #openstack-infra06:41
*** amotoki has quit IRC06:43
*** sdake has joined #openstack-infra06:44
*** stevemar has quit IRC06:44
*** salv-orlando has quit IRC06:44
*** mrunge has joined #openstack-infra06:45
*** unicell1 has quit IRC06:47
*** sdake has quit IRC06:48
*** mrunge has quit IRC06:49
*** mrunge has joined #openstack-infra06:49
*** amotoki has joined #openstack-infra06:49
StevenKpleia2: Update for meeting. I am still plugging away, trying to convince maven-release-plugin to build so maven-sortpom-plugin can build, so zanata-parent can.06:50
*** unicell has joined #openstack-infra06:51
StevenK[INFO] Error building POM (may not be this project's POM).06:52
StevenKReason: POM 'org.apache.maven.plugins:maven-surefire-plugin' not found in repository: System is offline.06:52
*** yamahata has quit IRC06:53
StevenKWhich is a lie, maven-surefire-plugin is in the repo. :-(06:53
*** yamahata has joined #openstack-infra06:53
*** unicell has quit IRC06:58
*** garyh has joined #openstack-infra07:00
*** mrda is now known as mrda-away07:00
*** unicell has joined #openstack-infra07:02
*** gokrokve has quit IRC07:02
*** gokrokve has joined #openstack-infra07:02
*** markvoelker has joined #openstack-infra07:04
*** garyh has quit IRC07:05
*** gokrokve has quit IRC07:07
*** Hal has joined #openstack-infra07:08
*** Hal is now known as Guest6820707:08
*** markvoelker has quit IRC07:08
*** Guest68207 has quit IRC07:12
*** emagana has quit IRC07:14
*** scheuran has joined #openstack-infra07:16
*** yamahata has quit IRC07:20
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Update pip-missing-reqs to pip-check-reqs  https://review.openstack.org/16497407:21
*** yfried_ is now known as yfried|afk07:22
*** otter768 has joined #openstack-infra07:22
*** FingerPistol has joined #openstack-infra07:23
*** afazekas has joined #openstack-infra07:25
*** otter768 has quit IRC07:27
*** ddieterly has joined #openstack-infra07:28
*** FingerPistol has quit IRC07:29
*** yfried|afk is now known as yfried_07:30
*** achanda has quit IRC07:31
*** ddieterly has quit IRC07:33
*** baoli has joined #openstack-infra07:34
*** AJaeger has joined #openstack-infra07:35
*** AJaeger has joined #openstack-infra07:35
*** AJaeger has quit IRC07:35
*** GumBall has joined #openstack-infra07:36
openstackgerrityolanda.robla proposed openstack-infra/system-config: Parameterize url to clone puppet modules  https://review.openstack.org/16337607:36
*** baoli has quit IRC07:38
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/16495607:39
*** yfried_ is now known as yfried|afk07:41
*** mpavone has joined #openstack-infra07:41
*** rkukura_ has joined #openstack-infra07:45
openstackgerrityolanda.robla proposed openstack-infra/puppet-graphite: Add ability to configure retention policy  https://review.openstack.org/16167907:46
*** rkukura has quit IRC07:47
*** rkukura_ is now known as rkukura07:47
*** armax has quit IRC07:48
rluciobharatk: good news, we were on the right track, it was a file descriptor limit issue07:49
BharatKrlucio: Cool....07:49
rluciobharatk: ulimit -Hn showed 4096 file descriptors max for the user, which is too low07:50
BharatKrlucio: now you are able to run devstack, through jenkins right.07:50
openstackgerrityolanda.robla proposed openstack-infra/puppet-cgit: Add configurable setting for MaxClients  https://review.openstack.org/16088507:50
rluciobharatk: i edited /etc/security/limits.conf and upped the stack user's limits to 10k07:50
BharatKrlucio: oh great point. :)07:50
rluciobharatk: well, one slave failed with an internal server error when trying to create and upload a disk image07:51
BharatKrlucio: what is the error you got?07:51
rluciobharatk: the other slave failed with a fd limit error later on07:52
rluciobharatk: guess 10k isnt enough!07:52
BharatKrlucio: Just now I ran devstack on ubuntu 14.0407:53
BharatKrlucio: It ran successfully07:53
BharatKrlucio: there I executed "ulimit -Hu"07:54
rluciobharatk: yea i actually just made a vm on my desktop and installed it there, seems to be working fine07:54
BharatKrlucio: I got "63718"07:54
*** soren has joined #openstack-infra07:54
rluciobharatk: yea the article i found was saying for db servers or web servers you might need 100k +07:55
BharatKrlucio: Oh okay. I doubt I have the output is in K or M?07:55
rluciobharatk: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/07:56
BharatKrlucio: That means here we need to check in start user.07:58
BharatKrlucio: *stack user07:58
rluciobharatk: right08:00
*** AJaeger has joined #openstack-infra08:00
rluciobharatk: by default the limit was 4096 on my boxes08:01
AJaegerfungi, something is wrong with check_irc_access.py. It should fail with exit(1) but does not - see http://logs.openstack.org/31/163331/4/check/gate-project-config-irc-access/1940784/console.html08:01
*** AJaeger has quit IRC08:01
*** markus_z has joined #openstack-infra08:01
*** yfried|afk is now known as yfried_08:01
*** oomichi_ has joined #openstack-infra08:02
BharatKrlucio: Yeah but how can we set the ulimit for new users. I mean "stack" user08:02
*** dtantsur|afk is now known as dtantsur08:03
*** HeOS has quit IRC08:03
rluciobharatk: you can add an entry for a specific user in /etc/security/limits.conf08:03
*** tnovacik_ has joined #openstack-infra08:03
BharatKrlucio: cool... is that working?08:04
rluciobharatk: i added "stack soft nofile 80000 / stack hard nofile 100000"08:04
rluciobharatk: just started a build with these new limits, i'll let you know when they finish08:04
BharatKrlucio: cool ... thanks.08:04
*** asselin_ has joined #openstack-infra08:04
*** markvoelker has joined #openstack-infra08:05
*** garyh has joined #openstack-infra08:06
*** AJaeger has joined #openstack-infra08:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix logic for check_irc_access failure  https://review.openstack.org/16497908:08
AJaegerfungi: I found something ^ and would appreciate carefull review08:08
*** AJaeger has quit IRC08:08
*** asselin_ has quit IRC08:09
*** markvoelker has quit IRC08:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix logic for check_irc_access failure  https://review.openstack.org/16497908:10
*** ildikov has quit IRC08:11
*** yfried_ is now known as yfried|afk08:11
*** ashishb has quit IRC08:12
*** Ala has joined #openstack-infra08:13
GheRiveromorning all08:13
*** macjack has quit IRC08:15
*** mfink__ has quit IRC08:15
*** garyh has quit IRC08:16
*** e0ne has joined #openstack-infra08:17
*** kaisers1 has quit IRC08:18
*** che-arne has quit IRC08:24
*** mfink_ has joined #openstack-infra08:26
*** yfried|afk is now known as yfried_08:27
*** ashishb has joined #openstack-infra08:27
*** ivar-lazzaro has joined #openstack-infra08:28
*** ddieterly has joined #openstack-infra08:29
*** salv-orlando has joined #openstack-infra08:31
*** hashar has joined #openstack-infra08:31
*** ashishb has quit IRC08:32
*** ivar-lazzaro has quit IRC08:32
*** ddieterly has quit IRC08:34
*** baoli has joined #openstack-infra08:35
*** e0ne is now known as e0ne_08:36
*** arxcruz has joined #openstack-infra08:36
*** tkelsey has joined #openstack-infra08:37
openstackgerrityolanda.robla proposed openstack-infra/system-config: Configure puppetmaster server from manifest  https://review.openstack.org/16467008:38
*** ildikov has joined #openstack-infra08:39
*** arxcruz has quit IRC08:39
*** baoli has quit IRC08:39
*** nilasae has joined #openstack-infra08:41
*** yfried_ is now known as yfried|afk08:42
*** dims has joined #openstack-infra08:42
*** Hal has joined #openstack-infra08:44
*** Hal is now known as Guest2426408:44
*** arxcruz has joined #openstack-infra08:44
*** jlanoux has joined #openstack-infra08:46
*** dims has quit IRC08:48
*** ashishb has joined #openstack-infra08:51
*** andreykurilin_ has joined #openstack-infra08:53
*** yfried|afk is now known as yfried_08:55
*** boris-42 has joined #openstack-infra08:59
*** woodster_ has quit IRC09:00
*** Somay has joined #openstack-infra09:01
*** Somay has quit IRC09:03
*** jistr has joined #openstack-infra09:04
*** Somay has joined #openstack-infra09:04
*** mrunge_ has joined #openstack-infra09:04
*** jcoufal has joined #openstack-infra09:05
*** yfried_ is now known as yfried|afk09:05
*** markvoelker has joined #openstack-infra09:06
*** mrunge has quit IRC09:08
rluciobharatk: ok well, this is where i am stuck now, seen this before by chance? http://paste.openstack.org/show/192830/09:09
BharatKrlucio: that is glance not configured well09:10
BharatKrlucio: do you have logs of this run09:10
*** markvoelker has quit IRC09:11
BharatKrlucio: look at glance logs once09:11
*** skolekonov has joined #openstack-infra09:12
*** mpavone has quit IRC09:12
rluciobharatk: yea i have logs, one sec09:12
BharatKrlucio: okay09:12
*** Somay has quit IRC09:13
BharatKrlucio: it I am not wrong, that is ubuntu 14.04 right?09:13
*** zz_johnthetubagu is now known as johnthetubaguy09:14
rluciobharatk: yes its 14.0409:17
*** garyh has joined #openstack-infra09:17
openstackgerritChristian Berendt proposed openstack-infra/project-config: Add new project faafo to Stackforge  https://review.openstack.org/16466809:17
rluciobharatk: http://paste.openstack.org/show/192832/09:17
*** ghostpl_ has joined #openstack-infra09:18
*** GumBall has quit IRC09:18
BharatKrlucio: are you using swift as glance backend09:18
rluciobharatk: apparently yea09:19
rluciobharatk: whatever the default is09:19
BharatKrlucio: I think default is not swift just a file09:19
BharatKrlucio: anyway can you send your localrc file?09:19
rluciobharatk: http://paste.openstack.org/show/192833/09:21
*** salv-orlando has quit IRC09:21
*** salv-orlando has joined #openstack-infra09:21
rluciobharatk: yea... this is probably the core issue.  isn't this autogenerated?09:22
*** kaisers has joined #openstack-infra09:23
BharatKrlucio: yeah this is auto generated.09:23
*** otter768 has joined #openstack-infra09:23
BharatKrlucio: anyway even swift should work. Can you paste swift logs?09:23
BharatKrlucio: just search for errors in swift logs.09:26
*** yfried|afk is now known as yfried_09:26
*** garyh has quit IRC09:27
*** mpavone has joined #openstack-infra09:27
*** otter768 has quit IRC09:27
*** sarob has joined #openstack-infra09:28
rluciobharatk: oh i put the errors in that previous paste, if you scroll down09:28
BharatKrlucio: sorry I will see.09:29
BharatKrlucio: Same error09:29
*** e0ne_ has quit IRC09:30
rluciobharatk: i thought i had read somewhere a while ago that you can get swift errors if your machine isn't up to snuff09:30
BharatKrlucio: Resource temporarily unavailable09:30
*** ddieterly has joined #openstack-infra09:30
rluciobharatk: yea09:30
BharatKrlucio: in line # 71 you can find that...09:31
BharatKrlucio: check the ulimit of swift user09:31
BharatKrlucio: brb09:31
rluciobharatk: ah interesting09:33
*** sarob has quit IRC09:33
*** ddieterly has quit IRC09:34
*** yfried_ is now known as yfried|afk09:36
*** dizquierdo has joined #openstack-infra09:36
rluciobharatk: i changed ulimit from user to '*' so that is for all users09:38
rluciobharatk: so all users should have 100k fd limit now09:39
rluciobharatk: re-running  again09:39
*** ssam2 has joined #openstack-infra09:40
*** ZZelle has joined #openstack-infra09:43
*** jedimike has joined #openstack-infra09:45
*** markus_z has quit IRC09:47
*** scheuran has quit IRC09:47
*** mpavone has quit IRC09:47
*** markus_z has joined #openstack-infra09:47
*** mpavone has joined #openstack-infra09:48
*** andreykurilin_ has quit IRC09:48
*** dims has joined #openstack-infra09:50
rluciobharatk: bah same error /shrug09:54
*** HeOS has joined #openstack-infra09:54
rluciobharatk: i gotta go for now, i'll look more into this later.  thanks for all your help though09:54
*** rlucio has quit IRC09:55
*** andreykurilin_ has joined #openstack-infra09:56
*** scheuran has joined #openstack-infra09:59
*** andreykurilin_ has quit IRC10:01
SergeyLukjanovnibalizer, jeblair, sorry about merging https://review.openstack.org/#/c/164725/ it was looks like no disagreements ;)10:01
SergeyLukjanovnibalizer, jeblair, and fact that I hate arrow alignment in puppet was helping me to merge it :)10:02
*** mpavone has quit IRC10:03
*** mpavone has joined #openstack-infra10:06
*** markvoelker has joined #openstack-infra10:07
*** claudiub has joined #openstack-infra10:08
*** Qiming has quit IRC10:11
*** markvoelker has quit IRC10:12
*** mfink_ has quit IRC10:13
*** teran has joined #openstack-infra10:13
*** yfried|afk is now known as yfried_10:16
*** e0ne has joined #openstack-infra10:17
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:no-underscore-dangle  https://review.openstack.org/16150310:21
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:strict  https://review.openstack.org/16150410:21
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:no-console  https://review.openstack.org/16150510:21
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:space-unary-ops  https://review.openstack.org/16150610:22
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:consistent-return  https://review.openstack.org/16150710:22
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:no-shadow  https://review.openstack.org/16150810:23
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:space-infix-ops  https://review.openstack.org/16150910:24
openstackgerritMerged openstack-infra/storyboard-webclient: Enabled eslint:no-use-before-define  https://review.openstack.org/16151010:24
openstackgerritMerged openstack-infra/storyboard-webclient: Added null check for story creator_id.  https://review.openstack.org/16487710:24
*** yfried_ is now known as yfried|afk10:26
*** garyh has joined #openstack-infra10:28
*** rlandy has joined #openstack-infra10:29
*** sarob has joined #openstack-infra10:29
*** mrunge_ is now known as mrunge10:30
*** ddieterly has joined #openstack-infra10:31
*** achanda has joined #openstack-infra10:31
*** claudiub has quit IRC10:32
openstackgerritJulien Danjou proposed openstack-dev/pbr: Handle PEP426 markers  https://review.openstack.org/16501510:32
*** pc_m has joined #openstack-infra10:33
*** sarob has quit IRC10:35
*** ddieterly has quit IRC10:35
*** achanda has quit IRC10:36
*** dtantsur is now known as dtantsur|brb10:37
*** _nadya_ has joined #openstack-infra10:37
*** garyh has quit IRC10:38
*** macjack has joined #openstack-infra10:38
*** e0ne is now known as e0ne_10:40
*** e0ne_ is now known as e0ne10:44
*** Qiming has joined #openstack-infra10:47
*** coolsvap is now known as coolsvap|afk10:53
*** alex_xu has joined #openstack-infra10:55
alex_xuianw: Hi, are you around, could you let me know whether this https://review.openstack.org/#/c/160068/4 make sense to you? if not, I will update it. Thanks in advance!10:57
*** Bsony has joined #openstack-infra11:00
*** Bsony has quit IRC11:00
*** Bsony has joined #openstack-infra11:01
openstackgerritPeter Hamilton proposed openstack/requirements: Update PyKMIP version in requirements  https://review.openstack.org/16444911:07
*** e0ne is now known as e0ne_11:08
*** Somay has joined #openstack-infra11:08
*** mfink_ has joined #openstack-infra11:08
*** markvoelker has joined #openstack-infra11:08
*** jedimike has quit IRC11:10
*** notnownikki has joined #openstack-infra11:10
*** ashishb has quit IRC11:12
*** imcsk8 has quit IRC11:14
*** markvoelker has quit IRC11:14
*** imcsk8 has joined #openstack-infra11:14
*** claudiub has joined #openstack-infra11:17
*** e0ne_ has quit IRC11:17
kaisersHello! Is it ok to ask general tempest question here or is there a better channel?11:19
*** _nadya_ has quit IRC11:19
*** ominakov has joined #openstack-infra11:22
*** cdent has joined #openstack-infra11:22
openstackgerrityolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created  https://review.openstack.org/16502911:22
*** otter768 has joined #openstack-infra11:24
*** e0ne has joined #openstack-infra11:27
*** ominakov has quit IRC11:28
*** otter768 has quit IRC11:28
openstackgerrityolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created  https://review.openstack.org/16502911:30
*** sarob has joined #openstack-infra11:30
*** macjack has quit IRC11:30
*** sarob has quit IRC11:35
*** markvoelker has joined #openstack-infra11:38
*** mrmartin has joined #openstack-infra11:40
*** Qiming_ has joined #openstack-infra11:42
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Temp set saharaclient src juno job to non-voting  https://review.openstack.org/16503811:43
openstackgerrityolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created  https://review.openstack.org/16502911:43
*** ldnunes has joined #openstack-infra11:44
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper  https://review.openstack.org/15668111:45
*** baoli has joined #openstack-infra11:45
*** Qiming has quit IRC11:46
*** mtanino has joined #openstack-infra11:46
*** baoli has quit IRC11:47
*** baoli has joined #openstack-infra11:47
*** deepakcs has quit IRC11:48
openstackgerritJaroslav Henner proposed openstack-infra/jenkins-job-builder: Add checking of multijob phase condition.  https://review.openstack.org/16466911:49
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added request statistic  https://review.openstack.org/16469611:49
*** baoli has quit IRC11:50
*** koolhead17 has quit IRC11:51
*** ominakov has joined #openstack-infra11:57
*** Longgeek has quit IRC11:58
*** woodster_ has joined #openstack-infra11:59
*** amotoki has quit IRC11:59
*** e0ne is now known as e0ne_12:00
*** dprince has joined #openstack-infra12:01
*** yamamoto has joined #openstack-infra12:04
*** coolsvap|afk is now known as coolsvap12:04
*** stevemar has joined #openstack-infra12:05
*** e0ne_ is now known as e0ne12:06
*** oomichi_ has quit IRC12:06
*** weshay has joined #openstack-infra12:08
*** ihrachyshka has joined #openstack-infra12:10
*** radez_g0n3 is now known as radez12:13
*** zz_dimtruck is now known as dimtruck12:14
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Branches, milestones and projects ids validation in tasks  https://review.openstack.org/16212612:14
*** dkranz has quit IRC12:16
*** dims has quit IRC12:16
*** dims has joined #openstack-infra12:17
*** dtantsur|brb is now known as dtantsur12:18
Somayhi clarkb : I have made the changes you mentioned here. https://review.openstack.org/#/c/163137/ Please review :)12:18
*** bswartz has quit IRC12:19
*** jaypipes has joined #openstack-infra12:20
*** aysyd has joined #openstack-infra12:22
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added story types  https://review.openstack.org/16341212:23
*** ominakov has quit IRC12:24
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added story types  https://review.openstack.org/16341212:25
*** dimtruck is now known as zz_dimtruck12:25
*** _nadya_ has joined #openstack-infra12:25
*** jistr is now known as jistr|class12:26
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added story types  https://review.openstack.org/16341212:26
*** salv-orlando has quit IRC12:30
*** sarob has joined #openstack-infra12:31
*** ddieterly has joined #openstack-infra12:32
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added request statistic  https://review.openstack.org/16469612:32
*** adalbas has joined #openstack-infra12:32
*** ominakov has joined #openstack-infra12:35
*** spredzy is now known as spredzy|afk12:36
*** sarob has quit IRC12:36
*** ddieterly has quit IRC12:37
openstackgerritPeter Hamilton proposed openstack/requirements: Update PyKMIP version in requirements  https://review.openstack.org/16444912:38
*** gordc has joined #openstack-infra12:39
*** adalbas has quit IRC12:41
*** ominakov_ has joined #openstack-infra12:43
*** ominakov has quit IRC12:45
*** banix has joined #openstack-infra12:48
*** dkliban_afk is now known as dkliban12:48
*** garyh has joined #openstack-infra12:49
openstackgerritMerged openstack-infra/shade: Fix up and document input parameters  https://review.openstack.org/16018412:50
openstackgerritMerged openstack-infra/shade: Add support for keystone projects  https://review.openstack.org/15994112:52
openstackgerritTeemu Patja proposed stackforge/python-jenkins: Add a get_nodes method  https://review.openstack.org/16306812:52
openstackgerritMerged openstack-infra/shade: Small fixes found working on ansible modules  https://review.openstack.org/16214012:52
openstackgerritMerged openstack-infra/shade: Disable dogpile.cache if cache_interval is None  https://review.openstack.org/16196312:52
openstackgerritMerged openstack-infra/shade: Add basic unit test for shade.openstack_cloud  https://review.openstack.org/15967412:52
openstackgerritMerged openstack-infra/shade: Handle image name for boot from volume  https://review.openstack.org/16469112:52
*** adalbas has joined #openstack-infra12:53
*** nelsnels_ has quit IRC12:55
*** bswartz has joined #openstack-infra12:56
*** bknudson has quit IRC12:57
*** mriedem_away is now known as mriedem12:58
*** garyh has quit IRC12:59
cdentsdague: could you be so kind as to lay your eyes upon: https://review.openstack.org/#/c/163865/12:59
*** _nadya_ has quit IRC12:59
*** salv-orlando has joined #openstack-infra13:00
*** bknudson has joined #openstack-infra13:01
openstackgerritMonty Taylor proposed openstack-infra/shade: Allow for passing cache class in as a parameter  https://review.openstack.org/16214813:01
*** Qiming_ is now known as Qiming13:03
*** ChuckC has quit IRC13:04
*** ChuckC_ has joined #openstack-infra13:04
*** ddieterly has joined #openstack-infra13:05
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Restore query for horizon bug 1265057  https://review.openstack.org/16505913:06
openstackbug 1265057 in grenade "gate/check-grenade-dsvm: Horizon front page not functioning!" [Undecided,Confirmed] https://launchpad.net/bugs/126505713:06
*** ddieterly has quit IRC13:07
gordcsdague: could you re-approve this as well? :) https://review.openstack.org/#/c/155473/13:07
*** ChuckC has joined #openstack-infra13:07
*** ChuckC_ has quit IRC13:07
sdaguemriedem: do you have a fix for the horizon thing?13:07
mriedemsdague: nope13:08
mriedemlooking at recent commits13:08
mriedemdoug-fish: can you help here? ^13:08
openstackLaunchpad bug 1265057 in grenade "gate/check-grenade-dsvm: Horizon front page not functioning!" [Undecided,Confirmed]13:08
*** julim has joined #openstack-infra13:09
sdaguewhen did that merge?13:09
sdagueyeh, all the cells jobs are fail right now13:09
sdagueok, that explains things13:09
doug-fishmriedem: yeah, looking13:09
*** amotoki has joined #openstack-infra13:09
mriedemsdague: is there a reason why we run horizon in that cells devstack job?13:11
openstackgerritMerged openstack-infra/project-config: Add functional test job for python-ceilometerclient  https://review.openstack.org/16386513:11
dhellmannmordred, lifeless: Can we sync up on the state of pbr today? We have a lot of unreleased changes in master that I'd like to understand and jd__ has started working on the pip requirements environment rules stuff there, too.13:11
sdaguemriedem: because that cells job is terrible in all ways13:11
*** dustins has joined #openstack-infra13:11
mriedemsdague: well, it did find the problem with volumes in novaclient :)13:12
mriedemso it's not all bad13:12
*** ChuckC has quit IRC13:12
*** tnovacik_ has quit IRC13:13
*** dkranz has joined #openstack-infra13:14
*** dboik has quit IRC13:14
*** dustins has quit IRC13:15
sdaguemore importantly, why are we running any exercise in that cells job except the volumes one13:15
*** yamamoto has quit IRC13:15
*** changbl has quit IRC13:16
sdagueI'm just going to start deleting exercises in devstack, no one should be using these for testing13:16
mriedemwell, volumes and secgroups are probably about the only useful thing for that job, not sure what client-args is13:17
mordreddhellmann: yes please. I believe we need one more patch from lifeless (or someone) to fix master13:17
mriedembut we don't need to test sahara and swift and horizon for cells13:17
dhellmannmordred: ok, I wouldn't want to actually release at this point in the cycle so I'm mostly trying to understand where things stand. What's the missing patch?13:18
sdaguemriedem: see... we have these actual tests, which are maintained, in this other project :)13:18
sdaguedevstack exercises are not tests, and shouldn't be used as such13:19
*** liusheng has quit IRC13:19
*** liusheng has joined #openstack-infra13:19
mriedemmy only counterpoint would be that volumes thing13:19
*** nelsnelson has joined #openstack-infra13:19
*** PaulMurray has joined #openstack-infra13:20
mriedemthat tempest wasn't tickling13:20
*** tnovacik_ has joined #openstack-infra13:20
sdagueyeh, so that was discovered because we have 0 testing on the clients13:20
sdaguewhich is it's own issue13:20
*** peristeri has joined #openstack-infra13:20
*** nelsnelson has quit IRC13:22
*** banix has quit IRC13:22
mordreddhellmann: it's the thign to make the semver stuff not break everything13:22
*** yamamoto has joined #openstack-infra13:22
*** nelsnelson has joined #openstack-infra13:22
mordreddhellmann: I've already forgotten what it is about the semver patches that broke everything - but I think it was that patch series being angry with people for having legacy things installed or something13:22
mordreddhellmann: I also agree - releasing now == bad13:23
*** ayoung has joined #openstack-infra13:23
sdaguedhellmann: btw, until that cells / horizon thing is unwound, all the req patches are blocked13:23
*** mtanino has quit IRC13:24
dhellmannmordred: oh, right. I wonder if we should just revert that work and move it to a feature branch? It's not clear to me what the goal there was to begin with.13:24
openstackgerritMerged openstack-infra/elastic-recheck: Restore query for horizon bug 1265057  https://review.openstack.org/16505913:24
openstackbug 1265057 in grenade "gate/check-grenade-dsvm: Horizon front page not functioning!" [Undecided,Confirmed] https://launchpad.net/bugs/126505713:24
dhellmannsdague: ok, I don't know about the cells/horizon thing, do you think that's something I might be able to help with or are there folks already working it?13:24
*** _shaps_ has joined #openstack-infra13:25
*** Somay has quit IRC13:25
*** otter768 has joined #openstack-infra13:25
sdagueso, honestly, right now this is my fix for it - Sean Dague proposed openstack-dev/devstack: remove horizon exercises  https://review.openstack.org/16506313:25
mordreddhellmann: I think the idea was that if there was another thing we needed to release and had to do it on the stable branch, that we'd revert, put semver in a feature branch and make the current stable branch master again13:25
mordredlifeless: any chance you'll have time to finish the semver work any time soon ^^ ?13:25
*** sdake has joined #openstack-infra13:25
*** spredzy|afk is now known as spredzy13:25
sdaguebut, I'm not going to babysit that, and people keep adding jobs to devstack, so my faith in that landing on first go are small13:26
*** tnovacik_ has quit IRC13:26
dhellmannsdague: seems reasonable13:26
*** eharney has joined #openstack-infra13:27
*** alexpilotti has joined #openstack-infra13:27
*** sdake__ has joined #openstack-infra13:27
dhellmannmordred, lifeless: ok. Now that we have the processes set for the other oslo libs, I want to start including pbr in our regular reviews. It's pretty inconvenient that master is unreleasable, because it ends up meaning no one wants to do anything with the lib because there's no point. :-/13:27
*** otter768 has quit IRC13:29
*** hashar has quit IRC13:29
openstackgerritMonty Taylor proposed openstack-infra/shade: Add 'rebuild' to shade  https://review.openstack.org/16106713:29
*** hashar has joined #openstack-infra13:30
mordreddhellmann: ++ totally agree13:30
*** bhunter71 has joined #openstack-infra13:30
*** salv-orlando has quit IRC13:31
*** sdake has quit IRC13:31
*** kaisers has left #openstack-infra13:31
*** alexpilotti has quit IRC13:32
*** kaisers has joined #openstack-infra13:32
*** alexpilotti has joined #openstack-infra13:32
*** sarob has joined #openstack-infra13:32
dhellmannmordred, lifeless : I wonder if some of the constraints checking would be better off in a stand-alone tool for suggesting a new tag to use for a project, instead of being enforced in pbr itself.13:32
*** _nadya_ has joined #openstack-infra13:32
*** hdd has joined #openstack-infra13:33
mordreddhellmann: I also believe I agree with that13:33
*** ChuckC has joined #openstack-infra13:33
mordreddhellmann: also, I just re-replied to the list - my first one was wrong13:33
*** ociuhandu has joined #openstack-infra13:33
mordreddhellmann: I think we do need to do the compat shim you suggested, because pbr doesn't (and can't) do stable releases13:33
mordredso tip of pbr really _must_ support existing stable versions of thigns in the field13:34
mordredwhich means we need to keep support for reading requirements-py* - but moving forward we should get the projects using markers instead13:34
*** tiswanso has joined #openstack-infra13:35
*** dboik has joined #openstack-infra13:36
*** banix has joined #openstack-infra13:36
dhellmannmordred: yeah that's another argument in favor of having pbr just "do what I tell you" and having these semver rules be applied by something else13:36
*** LinuxJed_ has joined #openstack-infra13:36
dhellmannopenstack is all edge cases13:36
*** ddieterly has joined #openstack-infra13:36
dhellmannmordred: I'll start  ML thread about fixing pbr master, since the time difference with lifeless is leaving him out of this conversation.13:37
*** sarob has quit IRC13:38
openstackgerritMonty Taylor proposed openstack-infra/shade: Add 'rebuild' to shade  https://review.openstack.org/16106713:38
*** tiswanso has quit IRC13:38
*** hdd has quit IRC13:38
*** tiswanso_ has joined #openstack-infra13:39
ZZellefungi, hi13:40
mordreddhellmann: sweet. thanks13:40
*** prad has joined #openstack-infra13:42
*** johnthetubaguy is now known as zz_johnthetubagu13:43
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added events for tags  https://review.openstack.org/16507413:43
*** yamamoto has quit IRC13:43
*** LinuxJedi has quit IRC13:44
*** claudiub has quit IRC13:44
*** LinuxJed_ is now known as LinuxJedi13:44
*** soren has quit IRC13:45
*** Somay has joined #openstack-infra13:45
dstufftmordred: coutndown until pbr falls into postel's tarpit13:45
dstufft(like all the other packaging tools :V)13:46
openstackgerritMerged openstack-infra/shade: Add methods to create and delete networks  https://review.openstack.org/15792313:47
*** sdake has joined #openstack-infra13:48
mordreddstufft: everything should go into the tarpits13:48
mtreinishclarkb: sort of, we've got a workaround to do the migration in a different way which shouldn't trigger the space issue: https://review.openstack.org/16483813:50
mtreinishonce that lands I'll push a bugfix release and we'll try again13:50
mordredmtreinish: is that patch working now?13:50
*** zz_dimtruck is now known as dimtruck13:51
mtreinishmordred: yeah, I got it to pass the migration unit tests, and I inspected it locally it looked fine13:51
*** zz_johnthetubagu is now known as johnthetubaguy13:51
*** sdake__ has quit IRC13:51
*** banix has quit IRC13:52
*** sabeen has joined #openstack-infra13:53
*** koolhead17 has joined #openstack-infra13:55
*** koolhead17 has quit IRC13:55
*** koolhead17 has joined #openstack-infra13:55
*** yamamoto has joined #openstack-infra13:56
*** Qiming has quit IRC13:56
*** Qiming has joined #openstack-infra13:56
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added request statistic  https://review.openstack.org/16469613:58
*** amitgandhinz has joined #openstack-infra13:59
*** eharney has quit IRC13:59
*** garyh has joined #openstack-infra14:00
*** dboik_ has joined #openstack-infra14:00
openstackgerritMonty Taylor proposed openstack-infra/shade: Add 'rebuild' to shade  https://review.openstack.org/16106714:00
*** gokrokve has joined #openstack-infra14:01
mtreinishjeblair, mordred: btw, this was the patch from yesterday to run the migrations in the background: https://review.openstack.org/16473614:03
*** pblaho_ has joined #openstack-infra14:03
*** dboik has quit IRC14:03
*** r-daneel has joined #openstack-infra14:03
mtreinishthe only bit missing from the puppet before we push the new release is fixing the syntax error on the mysqldump (which unfortunately isn't printed in the failure output)14:03
*** pblaho_ has quit IRC14:04
*** pblaho has quit IRC14:05
*** jistr|class is now known as jistr14:05
*** pblaho_ has joined #openstack-infra14:05
*** pblaho_ has quit IRC14:06
*** pblaho_ has joined #openstack-infra14:06
*** esker has joined #openstack-infra14:07
*** sigmavirus24_awa is now known as sigmavirus2414:07
*** pblaho_ has quit IRC14:08
*** garyh has quit IRC14:09
*** mtanino has joined #openstack-infra14:10
*** nilasae has quit IRC14:10
*** pblaho__ has joined #openstack-infra14:10
*** salv-orlando has joined #openstack-infra14:11
*** salv-orlando has quit IRC14:11
*** salv-orlando has joined #openstack-infra14:11
*** che-arne has joined #openstack-infra14:11
*** tnovacik_ has joined #openstack-infra14:12
mkodereranteaya: ping14:12
*** mrmartin has quit IRC14:12
openstackgerritMerged openstack-infra/system-config: Add instructions for vgcreate to sysadmin docs  https://review.openstack.org/15968114:14
openstackgerrityolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created  https://review.openstack.org/16502914:15
*** kaisers has quit IRC14:16
*** hdd has joined #openstack-infra14:17
openstackgerritMerged openstack-infra/system-config: Update docs for bup user home in /opt/backups  https://review.openstack.org/15993314:17
*** mattfarina has joined #openstack-infra14:18
openstackgerritMerged openstack-infra/system-config: Add changing to launch/ directory to a README  https://review.openstack.org/15964714:18
mordredmtreinish: looks great14:19
pabelangermorning, is there any etherpads that describe the ansible effort for -infra?14:20
*** mfink_ has quit IRC14:21
*** BharatK has quit IRC14:22
*** wenlock has joined #openstack-infra14:22
fungiZZelle: what's up?14:23
fungipabelanger: mordred said he'd whip up a spec14:23
*** mfink_ has joined #openstack-infra14:24
pabelangerfungi, okay, no rush.  I was just checking to see if any thoughts had been written down14:24
doug-fishmriedem: do you know offhand where the code is for the test that is failing and resulting in https://bugs.launchpad.net/horizon/+bug/1265057 ?14:25
openstackLaunchpad bug 1265057 in grenade "gate/check-grenade-dsvm: Horizon front page not functioning!" [Undecided,Confirmed]14:25
doug-fishpretty sure I know how to fix it14:26
mriedemdoug-fish: devstack, but it's being removed https://review.openstack.org/#/c/165063/14:26
doug-fishah, well that's a way to fix it too14:26
mriedemwe have the same test in tempest, so having it in the devstack exercises is pointless14:27
doug-fishI'm a bit surprised tempest isn't broken as well14:27
openstackgerritMonty Taylor proposed openstack-infra/system-config: Manage hiera.yaml on all hosts  https://review.openstack.org/16487014:28
openstackgerritMonty Taylor proposed openstack-infra/system-config: Set $group at node scope for hiera lookups  https://review.openstack.org/16484214:28
openstackgerritMonty Taylor proposed openstack-infra/system-config: Setup hiera lookups to do common,group,fqdn  https://review.openstack.org/16484514:28
mordredpabelanger: it's def on my todo list - hoping to get to it either today or tomorrow14:29
alex_xuianw: good morning, are you around? hope to get your option on https://review.openstack.org/#/c/160068/ thanks in advance14:29
openstackgerritMerged openstack-infra/system-config: Add configurable graphite url on zuul status  https://review.openstack.org/16217714:30
openstackgerritMerged openstack-infra/system-config: grammar updates for gerrit documentation  https://review.openstack.org/16018514:30
doug-fishokay, I'm not surprised now that tempest is working urllib2 follows directs.  curl doesn't (at least not without a flag)14:30
openstackgerritMerged openstack-infra/system-config: Add requirement to get approval to post comments  https://review.openstack.org/15255014:30
ZZellefungi, hi could you look at https://review.openstack.org/124410 ?14:30
*** tsg has joined #openstack-infra14:30
pabelangermordred, sounds good14:30
mordredpabelanger: althoguh with the three patches of nibalizer's that I just re-pushed above - the story becomes kinda simple14:31
*** timcline has joined #openstack-infra14:31
mordredpabelanger: a) split current common.yaml into per-host or per-group files (with some things still in common.yaml)14:32
mordredpabelanger: b) have ansible start copying the hiera files that are for a given host as step one before it runs puppet on that host14:32
mordredpabelanger: c) switch to puppet apply, because now each host will have its own hiera info14:33
*** zz_jgrimm is now known as jgrimm14:33
mordredpabelanger: d) move from the puppet-cert ansible inventory to the openstack ansible inventory14:33
mordredI think that's it14:33
*** sarob has joined #openstack-infra14:33
pabelangerYa, that is nice.14:34
mordredwhich I love, beacuse it's super simple transition and can be done stepwise14:34
*** achanda has joined #openstack-infra14:34
*** garyh has joined #openstack-infra14:35
fungistep z) once we have working dnssec somewhere, start having ansible check ssh host keys against sshfp records so we can finally stop relying on tofu14:36
*** _nadya_ has quit IRC14:37
mordredfungi: well, step y) write an extension for nova which can do an ssh ping of a host from the nova-compute process to the hypervisor over the on-machine connection and return the ssh host key14:37
mordredfungi: and get our cloud providers to deploy it14:37
openstackgerritMerged openstack-infra/system-config: Add the Gerrit replication plugin paramater 'defaultForceUpdate'  https://review.openstack.org/15513514:37
*** emagana has joined #openstack-infra14:37
fungitrue. then we can just continue to trust the nova api endpoint, which we have to anyway14:38
*** sarob has quit IRC14:38
mordredalthough I think the step would be "create server, query nova for hostkey, add hostkey to dns"14:38
mordredwhich would be killer cool14:38
mordredfungi: what does one need to do to openssh to get it to check sshfp records?14:39
jrollmordred: fungi: glyph did a big hacky thing recently to verify a cloud server's host keys securely14:39
jrollif you're interested I can maybe get him to do a writeup or something14:40
*** kaisers has joined #openstack-infra14:40
mordredjroll: yes - I would very much like to know what he did - because as of right now, I can  not think of any path that would be valid14:40
fungimordred: step one, have them served via dnssec-signed zones, step 2 have a local resolver which actually checks dnssec, step 3 turn on VerifyHostKeyDNS in your openssh client config14:40
mordredfungi: awesome. adding VerifyHostKeyDNS to ansible's openssh use should be easy14:41
jrollmordred: I'll poke him. when I say hacky I mean HACKY. but it seemed like it was workable14:41
dstufftyou could create a SSH certificate too14:41
dstufftinstead of DNSSEC which is kinda borken14:41
fungitrue, that's an alternative. maintain a ca which is trusted by your openssh client and sign all the host keys with it, then have them provide that ceritficate on connect14:42
mordredwe'd have to inject a cert on server create14:42
openstackgerritMerged openstack-infra/puppet-subunit2sql: Run subunit2sql migrations in the background  https://review.openstack.org/16473614:43
fungiyeah, the trick there is distributing the certs to the servers and distributing the ca cert to the clients14:43
fungias opposed to dns which solves the distribution problem14:43
mordredyup. I think I'm fine with just grabbing the host_key from the nova api when we create it14:43
*** achanda has quit IRC14:43
*** baoli has joined #openstack-infra14:44
*** gokrokve_ has joined #openstack-infra14:44
dstufftwouldn't that just be having whatever process is spinning up servers grab the host key from the nova API, ssh into the server with that host key, and signing the auto generated ssh key?14:44
mordreddstufft: yes. once the nova api can give us the host key14:45
*** gokrokve has quit IRC14:45
dstufftyea, I mean using the ssh certificates instead of DNSSEC, DNSSEC is kinda grody, it operates at a layer that things aren't really well equiped to deal with non-networking failures14:46
fungii mean, we sort of still have the tofu problem with sshfp or certs as well (but only once globally though rather than once for every client)14:46
*** _nadya_ has joined #openstack-infra14:46
fungithat is until nova can tell us the host key fingerprint14:46
fungiright now we ask nova to boot us an instance, then we ssh into it to configure it. it's that step where the risk comes in14:48
fungibecause our build tooling simply has to assume that the initial ssh it performs to that ip address is not subjected to a mitm attack14:48
openstackgerritMonty Taylor proposed openstack-infra/shade: Add 'rebuild' to shade  https://review.openstack.org/16106714:49
fungiperhaps that's something we could hack around via nova console, but that would be yucky14:49
openstackgerritSean Dague proposed openstack/requirements: Cap Oslo libraries for kilo  https://review.openstack.org/16265614:49
mordredfungi: yeah - I think writing a nova extension for M will be a nice thing there14:49
jeblairsdague: so the devstack-dsvm-cells job is asymmetric (horizon does not run it).  is horizon being completely removed from being tested in that job, or should we add it to horizon?14:50
dstufftdoesn't nova have the ability to add user data when spawning a server?14:50
mordreddstufft: sure- but we don't use it really14:50
mordreddstufft: and I'd like to keep it that way14:50
dstufftyou could use it to have the spawning process inject a freshly created SSH key14:50
mordredyes - we coudl - the problem is that it's a very brittle system and hard to debug when it breaks14:51
sdaguejeblair: honestly, I personally think we should remove the job entirely. It was setup poorly from the start.14:51
mordredwhich is why we mostly avoid it14:51
jeblairsdague: is someone working on that?14:51
sdaguenot that I know of14:51
*** spzala has joined #openstack-infra14:51
*** baoli has quit IRC14:51
dstufftmordred: that's reasonable. Just an idea anyways :D14:51
sdaguemy current fix is to remove the horizon exercise14:52
jeblairsdague: okay.  will that remove horizon from the test path?14:52
jeblairokay, i won't propose any layout changes then14:52
*** dimtruck is now known as zz_dimtruck14:53
*** baoli has joined #openstack-infra14:53
*** gokrokve_ has quit IRC14:53
*** gokrokve has joined #openstack-infra14:56
*** david-lyle_afk is now known as david-lyle14:57
*** nanliu has joined #openstack-infra14:58
*** mrmartin has joined #openstack-infra14:59
*** asselin_ has joined #openstack-infra15:00
*** kmartin has joined #openstack-infra15:00
*** SumitNaiksatam has joined #openstack-infra15:00
*** zz_dimtruck is now known as dimtruck15:03
*** julim has quit IRC15:04
*** jaypipes has quit IRC15:06
*** kgiusti has joined #openstack-infra15:06
*** erikmwil_ has joined #openstack-infra15:06
*** erikmwilson is now known as Guest7875715:06
*** erikmwil_ is now known as erikmwilson15:06
*** erikmwilson_ has joined #openstack-infra15:06
*** soren has joined #openstack-infra15:06
*** tiswanso_ has quit IRC15:07
*** julim has joined #openstack-infra15:07
*** erikmwilson has quit IRC15:08
*** erikmwilson has joined #openstack-infra15:08
*** e0ne is now known as e0ne_15:08
openstackgerritMonty Taylor proposed openstack-infra/system-config: Manage hiera.yaml on all hosts  https://review.openstack.org/16487015:09
openstackgerritMonty Taylor proposed openstack-infra/system-config: Set $group at node scope for hiera lookups  https://review.openstack.org/16484215:09
openstackgerritMonty Taylor proposed openstack-infra/system-config: Setup hiera lookups to do common,group,fqdn  https://review.openstack.org/16484515:09
*** dhritishikhar has joined #openstack-infra15:13
*** banix has joined #openstack-infra15:13
*** yamamoto has quit IRC15:14
*** tiswanso has joined #openstack-infra15:14
*** tiswanso_ has joined #openstack-infra15:15
*** ominakov_ has quit IRC15:15
*** crc32 has joined #openstack-infra15:17
kaisersHi! Does anybody have a min to give me a hint or two on how best skip a tempest test (started via tox/testr)?15:17
*** tiswanso has quit IRC15:18
*** eharney has joined #openstack-infra15:18
kaisersI know i can select a test via regexp parameter, that works fine. But can i also exclude a single test by that method?15:18
*** crc32 has quit IRC15:18
*** e0ne_ has quit IRC15:18
kaisersAs it is regexp, probably yes. :-D15:19
*** crc32 has joined #openstack-infra15:19
*** ssam2 has quit IRC15:20
*** ssam2 has joined #openstack-infra15:20
*** e0ne has joined #openstack-infra15:24
*** dannywilson has joined #openstack-infra15:24
jeblairkaisers: the folks in #openstack-qa might have an answer15:24
*** weshay has quit IRC15:25
*** otter768 has joined #openstack-infra15:26
kaisersjeblair: thanks, will try! :)15:26
*** crc32 has quit IRC15:28
*** dimtruck is now known as zz_dimtruck15:29
*** otter768 has quit IRC15:30
*** zz_dimtruck is now known as dimtruck15:31
*** changbl has joined #openstack-infra15:32
anteayamkoderer: how can I help?15:33
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose tracked branch for rebase when submitting  https://review.openstack.org/15887715:34
hasharjeblair: good morning :)  I have a couple Zuul patches that could use your approval when you get some spare time :D15:34
hasharjeblair: one is a state change occuring before the actual operation is started ( https://review.openstack.org/#/c/128921/ )15:34
*** sarob has joined #openstack-infra15:34
hasharjeblair: the other is some oddities with .git dirs that can be either files or directories ( https://review.openstack.org/#/c/123437/ )15:35
hasharboth have been running on wikimedia for a few months now15:35
jeblairhashar: thanks!  today is meeting day, so it may not be today, but i'll try to review them soon.  :)15:35
*** dhritishikhar has quit IRC15:35
hasharjeblair: also joshua has a bunch of Zuul refactoring changes that would be nice additions.15:36
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose tracked branch for rebase when submitting  https://review.openstack.org/15887715:36
hasharjeblair: have good meetings! I just wanted to make sure the changes are under your radar15:36
openstackgerritDmitry Tantsur proposed openstack/requirements: Add ironic-discoverd to projects.txt  https://review.openstack.org/15627015:37
*** tiswanso_ has quit IRC15:38
*** tiswanso has joined #openstack-infra15:39
anteayahashar: on your radar, under your radar means you don't see them, on your radar means you do15:39
*** tiswanso has quit IRC15:39
*** yfried|afk is now known as yfried_15:40
*** sarob has quit IRC15:40
hasharanteaya: ahhh I always wondered why one said "under"  that didn't make sense to me!15:40
* hashar blushes15:40
anteayahashar: :) perfectly understandable15:40
anteayaI can't even speak another language well enough to make any sense let alone try an idiom15:40
hasharyou should try :)15:41
anteayaha ha ha15:41
hasharif you have a nice audience, you will be corrected till you are flawless!15:41
anteayathat is true15:41
anteayathe audience makes all the difference15:41
anteayaa buddy of mine used to hold open stage once a month, we had about 70 of the same people all the time15:42
anteayaeveryone was so gentle kind and polite, some people became really good performers as a result of the support of the audience15:42
anteayamiss them15:42
*** tsg has quit IRC15:43
*** Somay has quit IRC15:45
jeblairmordred: i have reviewed your ansible puppet apply stack, however, i don't actually see how it will be used, particularly what the relationship is between "hiera_hostvars" and "hirea" and where those things will come from.15:46
jeblairmordred: i think there must be extra information in that conversation between you and clarkb that i did not read that didn't make it into the commit messages or comments15:46
mordredjeblair: yes - actually, ignore that whole stack15:46
mordredjeblair: pretty much the whole thing gets invalidated by yesterday's conversation15:46
mordredjeblair: I should have hit abandon - sorry for wasting your time15:46
mordredjeblair: the replacement will be WAY smaller and simpler15:47
jeblairmordred: okay, well, you still might want to read my -1.  :)15:47
mordredjeblair: I will15:47
mordredjeblair: I'm sure it will be a fun -115:47
jeblairmordred: you don't have to abandon it (i mean, unless that's the right thing to do).15:48
mordredjeblair: I believe it will be the right thing to do for much of it - but I'll clean it up in just a jiff15:48
jeblairmordred: you marked it wip, i just thought you already had the replacement up.15:48
mordredjeblair: nope. not yet - although you should look at the nibalizer hiera patches - those are worth looking at15:49
mordredGheRivero: your fix for the ansible inventory private option - does that work for you with current shade and os-client-config? if so, it's a cleaner patch than my patch15:49
anteayamordred: jeblair does post some of the most fun -1s15:49
*** yfried_ is now known as yfried|afk15:50
*** yamahata has joined #openstack-infra15:50
GheRiveromordred: no it doesn't. Needs a small tweak in shade. I was waiting for all your patches to land today and summit it.15:51
mordredI made this a little while ago: https://github.com/ansible/ansible/pull/10362/files ... but I think your approach may be cleaner15:51
*** prad has quit IRC15:51
*** prad has joined #openstack-infra15:51
*** ajmiller has joined #openstack-infra15:52
openstackgerritMerged openstack-infra/os-loganalyze: Update regex for better HTML detection.  https://review.openstack.org/16489115:52
openstackgerritMonty Taylor proposed openstack-infra/system-config: Manage hiera.yaml on all hosts  https://review.openstack.org/16487015:52
openstackgerritMonty Taylor proposed openstack-infra/system-config: Set $group at node scope for hiera lookups  https://review.openstack.org/16484215:52
openstackgerritMonty Taylor proposed openstack-infra/system-config: Setup hiera lookups to do common,group,fqdn  https://review.openstack.org/16484515:52
*** asselin_ has quit IRC15:52
dougwighello infra, good morning.  i need a few cores to peek at a new gate job for lbaas and neutron (non-voting for this week), for the new lbaas v2 feature: https://review.openstack.org/#/c/164390/15:52
*** tiswanso has joined #openstack-infra15:53
*** tiswanso has quit IRC15:53
*** tiswanso has joined #openstack-infra15:53
*** unicell has quit IRC15:54
anteayadougwig: WIP pending the right hooks landing in neutron-lbaas15:54
dougwigahh, they landed and i forgot to update.  one sec.15:54
anteayacan you post a comment confirming this has happened with a link?15:54
anteayaoh sorry that was the depends-on15:55
anteayawhich I see has merged15:55
jeblairmordred, nibalizer: why does elasticsearch not have a $group?  does it have no hiera info?15:55
*** Qiming has quit IRC15:55
dougwigalso commented.15:55
*** nilasae has joined #openstack-infra15:56
anteayadougwig: non-voting jobs don't go in the gate15:56
mordredjeblair: I just may have missed it15:56
openstackgerritGhe Rivero proposed openstack-infra/shade: Allow passing private option  https://review.openstack.org/16511915:56
dougwiganteaya: ok.15:56
GheRiveromordred: ^ that's the change needed in shade for ansible inventory to work with my patch15:57
*** liusheng has quit IRC15:57
*** yfried|afk is now known as yfried_15:57
*** liusheng has joined #openstack-infra15:58
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Promote lbaasv2 api job to both the lbaas and neutron check and gate  https://review.openstack.org/16439015:58
dougwiganteaya: thank you15:58
*** radez is now known as radez_g0n315:59
anteayaI'll fix the commit message unless you want to, you can do it in the gui15:59
anteayadougwig: ^15:59
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Promote lbaasv2 api job to both the lbaas and neutron check queue  https://review.openstack.org/16439016:00
dougwiganteaya: updated, thanks.16:00
anteayathank you16:01
anteayadougwig: you regressed16:01
anteayayou fixed the commit message and got the old patch back16:01
anteayatake a minute I'm not in a hurry16:01
*** soren has quit IRC16:02
anteayaI'm tired of being in a hurry16:02
anteayaand just as a public service announcement I would like to say that I think Tim Bell is one of the awesomest people I have ever had the pleasure to work with16:03
dougwiganteaya: that's odd, i used the gerrit link to edit the commit. can that revert the code?16:04
dougwigthe commit message, i mean.16:04
*** tonytan4ever has joined #openstack-infra16:04
*** ashleighfarnham has joined #openstack-infra16:05
*** ominakov has joined #openstack-infra16:05
anteayaif you did git review -d 164390 you pull down the patchset to your working env16:05
anteayaif you used the gui in gerrit, well it depends what patchset you were looking at when you did it, but I do believe the gui shows the lastest patchset in the gui16:06
*** Longgeek has joined #openstack-infra16:06
anteayaso if you used the gui to edit the commit message for patchset 4 then yes, it is very odd if you ended up with code from patchset 3 in patchset 516:06
*** kmartin has quit IRC16:06
*** amotoki has quit IRC16:06
*** yamamoto has joined #openstack-infra16:07
*** kmartin has joined #openstack-infra16:07
anteayabut yes in patchset 5 the gate jobs are back which were removed in patchset 416:07
anteayaso I am confused16:07
mrmartinfungi: r u here?16:07
dougwigi might've needed a browser refresh, i'll bet.  i'll fix as soon as the lbaas meeting is over.  sorry about that, learned something new.16:07
*** kaisers has quit IRC16:07
*** dalgaaf has joined #openstack-infra16:07
anteayadougwig: perhaps that was it16:08
anteayaif you had patchset 3 commit message in your gui and edited the commit message on that that would explain what I am looking at for patchset 516:08
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Promote lbaasv2 api job to both the lbaas and neutron check queues  https://review.openstack.org/16439016:08
*** BobH has joined #openstack-infra16:08
dougwiganteaya: fixed16:09
fungimrmartin: yeah, i'm here. what's up?16:09
*** baoli has quit IRC16:09
fungior down, perhaps?16:09
*** spredzy is now known as spredzy|afk16:09
mrmartinfungi: I cannot attend to infra meeting today, but the ask site seems to be ok16:09
*** dboik_ has quit IRC16:09
anteayadougwig: that is something I can support, well done16:09
mrmartinso we need to go on with testing, and when it gets a green light, we can continue the migration16:10
mrmartinso this part of the askbot task is accomplished nicely16:10
*** Longgeek has quit IRC16:10
fungimrmartin: okay, great. we may not need to discuss scheduling the cut-over maintenance during the meeting since i can tackle that on my own as long as there are a few people around to test it out afterward16:10
fungiso we can just collectively figure out a time when it will be convenient so that you and others are around to make sure it looks good and don't need a rollback16:11
mrmartinI like to see at least 2weeks of test window, because it is a very visited site, and I like to avoid any issues we can with a proper testing16:11
*** skolekonov has quit IRC16:11
pleia2good morning16:11
anteayamorning pleia216:12
fungimrmartin: perfectly fine. if you need a database/files refresh from production into the replacement server in the next couple of weeks, let me know16:12
mrmartinfungi: thanks, the actual db is enough16:12
*** ominakov has quit IRC16:13
pabelangerI don't think it is, but does nodepool offer some sort of 2nd stage invoke for a script?  Basically, trying see how I could reboot an image build, then have nodepool continue the image build before taking a snapshot16:13
elarsonpleia2: fwiw, here is what I've done so far from the new project steps regarding setting up oslo.cache http://paste.openstack.org/show/192920/16:13
openstackgerritTomas F proposed openstack-infra/jenkins-job-builder: Add combine-queued-commit support  https://review.openstack.org/16082916:13
pabelangerI'm going to say, it is not possible currently16:13
*** yfried_ is now known as yfried|afk16:14
pleia2elarson: thanks, I'll pass this along to shelly_16:14
*** kaisers has joined #openstack-infra16:14
pabelangerokay, ready-script might work16:15
pleia2elarson: want to put this into an etherpad.openstack.org pad instead so it can be edited/noted?16:15
pabelangerWow, okay, I really need to re-read nodepool, there is also a reset script too16:15
*** soren has joined #openstack-infra16:17
openstackgerritMerged openstack-infra/project-config: Temp set saharaclient src juno job to non-voting  https://review.openstack.org/16503816:17
*** esker has quit IRC16:17
elarsonpleia2: https://etherpad.openstack.org/p/oslo.cache-setup-steps16:18
anteayamorning zaro16:18
pleia2elarson: great, thanks!16:18
dougwiganteaya: ugh, i botched the grammar in that new commit message, but i'm guessing it's not worth churning jenkins over.16:19
openstackgerrityolanda.robla proposed openstack-infra/project-config: Update puppet-setproxy to belong to Gozer group  https://review.openstack.org/16482016:20
*** sputnik13 has quit IRC16:20
yolandaanteaya, thx for the review, fixed16:20
anteayadougwig: not for me, it says the important things16:20
anteayayolanda: thank you16:20
yolandai just copied acl from another stackforge one, but you are right16:20
anteayayolanda: figured as much, now if you needed it that is fine, I just wanted to ensure you had what you wanted16:20
yolandanow it looks fine to me16:21
*** salv-orlando has quit IRC16:21
anteayayolanda: thanks I'll look again16:22
mtreinishfungi: any thoughts on: https://review.openstack.org/#/c/164838/ it's the mordred workaround for the db migration16:22
yolandathx for the quick feedback16:23
pabelangerYa, reset not what I expected in nodepool16:23
pabelangerreally need some sort of 2 stage bootstrapServer function16:23
anteayayolanda: sure16:23
*** crc32 has joined #openstack-infra16:23
*** yfried|afk is now known as yfried_16:24
pabelangerjeblair, ^ regarding the purpose of the reset flag for images.  Trying to understand its purpose16:24
anteayayolanda: the only other thing is that right now you have release permissions given to the whole team, we usually prefer that those permissions are given to one person who is in a -release group, such as gozer-release in this case16:25
anteayayolanda: do you want tagging permissions on all of core or just one person?16:25
yolandawell, honestly haven't thought about it, but it's ok for me to follow the general guidelines16:26
jeblairpabelanger: it sort of looks like it's not used?16:26
anteayaokay, I'll give you an inline comment, sorry I missed it the first time16:26
pabelangerjeblair, Yup, that is what I am seeing too16:26
yolandaanteaya, so i can amend and specify a different team there16:26
jeblairpabelanger: i think we can remove it :)  it goes back to the initial commit; i may have copied something from devstack-gate, or it may have been an incomplete thought at the time.16:27
yolandamm, although, just thinking... it will be fine for me to have release perms for that project, as i created. But we are created a generic gozer team, if we want to reuse for some other projects, release person should be different16:27
*** jaypipes has joined #openstack-infra16:27
*** arxcruz has quit IRC16:27
anteayayolanda: your call, so if you don't want a release group say so in a comment16:28
anteayait saves the next reviewer having to have the same conversation16:28
yolandaok, i'll answer the review16:28
yolandathx for the guidance there, first time in these kind of commits16:28
anteayathanks for being around to reply16:28
pabelangerjeblair, okay, I'll get a patch up to remove it.  But curious what you think of some sort of 2 stage bootstrap process?  For the purpose of rebooting between stages?16:28
fungimtreinish: 164838 seems fine. any reason not to go ahead and approve?16:29
*** Somay has joined #openstack-infra16:29
mtreinishfungi: nope go for it, I'll push a release after it merges16:29
mtreinishwhich should trigger the migration to run again16:30
*** unicell has joined #openstack-infra16:30
*** soren has quit IRC16:31
jeblairpabelanger: BobBall was working on that, and may still have patches for it.  i'm not sure if the DIB work would be an easier way to deal with that.16:31
yolandaanteaya, ok, answered16:31
anteayaso clarkb when you are around, I don't see any enable_swift patches to review, does that mean that this is working?16:31
*** unicell1 has joined #openstack-infra16:31
*** sigmavirus24 is now known as sigmavirus24_awa16:31
clarkbanteaya unsure this morning has been a bit crazy for me16:32
jeblairclarkb: i merged the os-loganalyze change16:32
*** bdpayne has joined #openstack-infra16:32
clarkbjeblair perfect we can check thatbwith nodepool coverage logs16:32
anteayaclarkb: you okay?16:32
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Remove unused 'reset' setting from images  https://review.openstack.org/16513616:33
*** packet has joined #openstack-infra16:33
*** Bsony has quit IRC16:33
clarkbanteaya yup, MIL had scheduled surgery early today but on our way to pick her up the car died, so just dealt with car, eating fast then back to hospital16:34
*** GumBall has joined #openstack-infra16:34
anteayaclarkb: oh go go16:34
*** scheuran has quit IRC16:34
anteayaclarkb: hope everything works out well16:34
*** unicell has quit IRC16:35
anteayaclarkb: just be offline16:35
pabelangerjeblair, perfect, thanks. Found the review, going to see what I can do to bring it back to life16:35
*** sarob has joined #openstack-infra16:35
*** yamamoto has quit IRC16:36
jeblairpabelanger: okay, let's have a discussion about whether it can be done with DIB instead before you spend too much time on it16:37
pabelangerjeblair, Sounds good to me16:37
*** Somay has quit IRC16:38
jeblairpabelanger: because iirc, it's _very_ complex, and i'd like to switch to using DIB completely when it's finished/ready and seriously consider dropping snapshot-based images16:38
SpamapSmtreinish: you know, on that hand-coded migration, you could also not lose that microsecond of data by doing a table merge between test_runs_old and test_runs before dropping test_runs_old.16:39
SpamapSmordred: ^16:39
*** melwitt has joined #openstack-infra16:39
SpamapSnot sure we care, but just sayin. :)16:39
anteayazaro: so looking again at https://review.openstack.org/#/c/155463/16:39
pabelangerYa, I haven't looked into DIB yet, should see how they work.  But ya, was trying to see how to get multi-staged snapshots, but DIB sounds like right way moving forward16:39
clarkbif car had not broken down this wouldve been a mostly normal day oh well we have a rental now so its taken care of16:39
anteayaclarkb: glad you are in good shape16:40
anteayaclarkb: no fires I know of, take your time16:40
mtreinishSpamapS: meh, I'm fine with it. It gives it some flair :)16:40
*** SumitNaiksatam has quit IRC16:40
anteayazaro: what does it take to get the expected tarball on http://tarballs.openstack.org/ci/gerrit/16:40
*** enikanorov has quit IRC16:40
*** sarob has quit IRC16:41
*** enikanorov has joined #openstack-infra16:41
mtreinishSpamapS: I'm more worried about just getting this migration done with, so I can start working on the next one :)16:41
openstackgerritMerged openstack-infra/subunit2sql: Hard code the sql for mysql on migration 1679b5bc102  https://review.openstack.org/16483816:41
*** tnovacik_ is now known as tnovacik16:41
zaroanteaya: it requires an approval to a change in openstack-infra/gerrit project, so approval of this change will do: https://review.openstack.org/#/c/155448/16:41
SpamapSmtreinish: yeah, it's sort of like the cashier at 7-11 not caring that you don't have $0.01 and just giving you back whole dollars as change. They'll deal.16:41
anteayahaving said that I look at the logstash queue and see the graph has nosedived16:42
SpamapSworld works better when there's a smoothing factor in place. :)16:42
*** mrmartin has quit IRC16:42
anteayazaro: does that one not need to be rebased?16:42
*** mrmartin has joined #openstack-infra16:42
mtreinishfungi, jeblair, mordred: ^^^ ok it's landed I'll push a release now16:42
*** baoli has joined #openstack-infra16:43
zaroanteaya: the dependent change has already merged, so i don't think so.16:43
openstackgerritMerged openstack-infra/project-config: Create nova-zvm-virt-driver project on StackForge  https://review.openstack.org/16368716:43
anteayawell outdated with a red line looks like a rebase is required to my understanding of what outdated with a red line means16:43
zaroanteaya: just did a recheck so we'll see.16:43
clarkbhttp://logs.openstack.org/72/164472/2/check/nodepool-coverage/12bef67/cover/nodepool_nodedb.html it renders properly bow16:44
clarkbanteaya jeblair ^16:44
clarkbso now just need new images with index fix then rerun for new logs16:44
anteayaclarkb: wow, that is pretty16:44
anteayaah cool16:45
anteayayou planning on running them when you return?16:45
*** Somay has joined #openstack-infra16:45
jeblairclarkb: what's the index fix?16:45
anteayanot creating a new index file if there is one already16:45
*** mrmartin has quit IRC16:47
jeblairthat merged yesterday, so should be in today's images?16:47
*** yamamoto has joined #openstack-infra16:47
*** gordc has quit IRC16:48
anteayaI think so16:48
*** GumBall has left #openstack-infra16:48
clarkbif they built16:49
jeblairlooks like we made new bare-trusty images on hpcloud-b1 and b3, and all three rax.16:49
jeblairb2 b4 and b5 don't have bare-trusty images from today16:49
clarkbso good chance then, should recheck a nodepool change and roll the dice?16:49
*** harlowja_away is now known as harlowja16:49
*** yfried_ is now known as yfried|afk16:50
*** yamamoto has quit IRC16:50
jeblairhrm.  why does hpcloud-b2.bare-trusty not appear in the image build log at all?16:51
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Story events may now be filtered by event_type.  https://review.openstack.org/16046216:51
*** tqtran has joined #openstack-infra16:53
*** dhritishikhar has joined #openstack-infra16:53
*** alexpilotti has quit IRC16:53
*** salv-orlando has joined #openstack-infra16:53
*** markus_z has quit IRC16:55
*** Ala has quit IRC16:56
*** unicell has joined #openstack-infra16:56
*** tnovacik has quit IRC16:57
*** unicell1 has quit IRC16:58
*** bswartz has quit IRC16:58
*** gordc has joined #openstack-infra16:58
jeblairthe servers for those images went into ERROR state16:58
mtreinishok the release is pushed (waiting on upload to pypi), now comes the waiting game for puppet to pick it up (and hope backgrounding the migration works)16:58
openstackgerritMerged openstack/requirements: Bump cassandra-driver to 2.1.4  https://review.openstack.org/15111816:59
mtreinishseems like a good time for me to find food16:59
*** ociuhandu has quit IRC16:59
openstackgerritKhai Do proposed openstack-infra/system-config: Fix copying of jquery.js file for gerrit  https://review.openstack.org/16514517:00
*** ParsectiX has joined #openstack-infra17:01
*** Bsony has joined #openstack-infra17:03
*** yfried|afk is now known as yfried_17:03
*** jlanoux has quit IRC17:04
openstackgerritJames E. Blair proposed openstack-infra/project-config: Update forge-upload job to use tags  https://review.openstack.org/16401617:05
*** radez_g0n3 is now known as radez17:05
openstackgerritJames E. Blair proposed openstack-infra/project-config: Update forge-upload job to use tags  https://review.openstack.org/16401617:05
*** tonytan4ever has quit IRC17:05
*** ajmiller_ has joined #openstack-infra17:06
anteayanote a user who has been dccing me just joined -meeting17:06
*** hashar is now known as hasharAway17:06
anteayaI just blocked them17:06
*** dustins has joined #openstack-infra17:06
anteayafor myself I don't have ops on meeting17:06
anteayajeblair: ^17:07
*** ajmiller has quit IRC17:07
pleia2ah, I got them too, it's been a long time since I've seen that particular dcc vulnerability being exploited17:07
*** koolhead_ has joined #openstack-infra17:07
fungiyeah, just report them to staff in #freenode17:07
anteayafigured it was just me, didn't see the point in making noise17:07
anteayayou are okay with them being in meeting?17:08
pleia2freenode staff can ban them from the network17:08
*** dmorita has joined #openstack-infra17:08
pleia2then we don't need to do one-off bans in every channel17:08
anteayaI was going by your advice from last night which was when being dcc'd just block them17:08
fungithey'll just rejoin with another nick and ip address if you ban them17:08
pleia2that too17:08
anteayawell I got hit all weekend for my login credentials17:09
anteayanot interested in speaking up in freenode and making myself a bigger target17:09
*** koolhead17 has quit IRC17:09
anteayaand I'm sure they don't take kindly to pms, not from someone they don't know17:10
*** esker has joined #openstack-infra17:10
*** dboik has joined #openstack-infra17:10
*** coolsvap is now known as coolsvap|afk17:11
*** Ryan_Lane has joined #openstack-infra17:12
*** mpavone has quit IRC17:12
*** arosen1 has joined #openstack-infra17:12
anteayayes so I blocked it17:12
anteayawas just saying it joined -meeting17:12
arosen1sorry just curious if you guys have seen this gcc error that I seem to be hitting in our CI when it trys to install pycparser : http://paste.openstack.org/show/192936/17:14
arosen116:00:07     x86_64-linux-gnu-gcc: internal compiler error: Killed (program cc1)17:14
*** Somay has quit IRC17:14
*** jistr has quit IRC17:14
*** Somay has joined #openstack-infra17:15
*** dtantsur is now known as dtantsur|afk17:16
anteayanibalizer: I had a question on https://review.openstack.org/#/c/162830/ an extra line kind of question17:17
fungiarosen1: check the end of dmesg -T for signs of the kernel oom killer shooting gcc in the head17:17
*** dimtruck is now known as zz_dimtruck17:18
fungiarosen1: or possibly syslog if you archived that but have no access to the machine where it ran any longer17:18
*** ihrachyshka has quit IRC17:19
*** armax has joined #openstack-infra17:21
*** salv-orl_ has joined #openstack-infra17:21
*** patrickeast has joined #openstack-infra17:22
adam_gany chance of some project-config'er eyes on https://review.openstack.org/#/c/163632/ ? trying to close an ironicclient CI hole17:22
*** zz_dimtruck is now known as dimtruck17:23
zaroanteaya: recheck passed for 15544817:24
arosen1fungi: i have the kernel log here: http://logs2.aaronorosen.com/85/164885/1/check/dsvm-tempest-full-congress-pg-nodepool/4e6db51/logs/kern_log.txt.gz17:25
*** salv-orlando has quit IRC17:25
openstackgerritMerged openstack-infra/git-review: Support authentication in run_http_exc  https://review.openstack.org/12441017:25
arosen1ah also the syslog let me look in that17:25
arosen1fungi: this: http://logs2.aaronorosen.com/85/164885/1/check/dsvm-tempest-full-congress-pg-nodepool/4e6db51/logs/syslog.txt.gz#_Mar_16_22_59_36 ?17:26
*** johnthetubaguy is now known as zz_johnthetubagu17:26
*** vhoward has left #openstack-infra17:26
arosen1Just curious why this would start occuring now though since this has been working for a while with no changes. I guess this is always the nature of the beast17:26
fungiarosen1: i see it at the end of your kern_log.txt "Out of memory: Kill process 13320 (cc1)"17:27
*** otter768 has joined #openstack-infra17:27
arosen1fungi:  hrm the Ci runs with 4gigs of ram17:27
fungiarosen1: but if you search the kern_log.txt for occurrences of the word "kill" you'll see that it was very busy killing all sorts of other things prior to that too17:27
arosen1fungi: how much ram do you guys use for your jenkin slaves?17:28
fungiarosen1: 8gb17:28
pleia2devstack needs 8 these days17:28
*** gokrokve has quit IRC17:28
arosen1ah okay i'll bump it up to 8 and see17:28
*** sigmavirus24_awa is now known as sigmavirus2417:28
*** tonytan4ever has joined #openstack-infra17:28
*** notnownikki has quit IRC17:29
*** arosen has joined #openstack-infra17:29
*** otter768 has quit IRC17:31
*** unicell has quit IRC17:32
*** unicell has joined #openstack-infra17:33
*** packet has quit IRC17:33
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Add support for running puppet apply  https://review.openstack.org/16354017:33
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Remove hiera data option  https://review.openstack.org/16515617:33
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Copy in matching hiera files  https://review.openstack.org/16515717:33
*** achanda has joined #openstack-infra17:34
mordredclarkb, nibalizer, jeblair: ^^ there you go - that's the full implementation needed based on yesterday's conversation17:34
openstackgerrityolanda.robla proposed openstack-infra/puppet-cgit: Add configurable setting for MaxClients  https://review.openstack.org/16088517:34
mordredwell, needs one more patch in system-config17:34
*** BobBall is now known as BobBall_AWOL17:34
mordredbut that's the support code17:34
*** e0ne has quit IRC17:37
openstackgerritMerged openstack/requirements: Cap Oslo libraries for kilo  https://review.openstack.org/16265617:37
*** sarob has joined #openstack-infra17:37
openstackgerrityolanda.robla proposed openstack-infra/system-config: Expose jenkins_gitfullname and jenkins_gitemail  https://review.openstack.org/16169517:37
*** sputnik13 has joined #openstack-infra17:37
ttxLooks like we have some node attrition processing the post queue17:37
ttx7 hours lag there17:37
ttxis that a known issue ?17:38
ttxfungi, jeblair ^17:38
ttxadam_g: will you be around for the cross-project meeting ? We'll discuss your openstack-spec17:38
anteayattx I confirm what you are seeing in the post queue17:39
clarkblooks like I will miss infra meeting. only thing I have is ansible is puppeting well and swift logs still going17:39
ttx3 hours ago it was a 4 hour lag so it may just be stuck17:39
fungittx: yeah, a while back there was some request that the check pipeline be prioritized over the post pipeline, since changes now need to pass check before they can enter the gate17:39
adam_gttx, yeah, was planning on it17:40
ttxadam_g: great thx17:40
fungittx: so now post jobs only run if there's no pending demand from gate or check pipelines17:40
fungiwhich today you can see on the zuul job queues graph has been pretty steady demand17:40
anteayaclarkb: thanks, take care of what you need to17:40
ttxfungi: hmm, wondering if that will affect my tarball generation17:41
ttxfungi: I suspect those will use the pre-release pipe17:41
fungittx: yeah, pre-release and release are on equal footing with check i believe17:41
*** HeOS has quit IRC17:41
ttxit will just bork my tools (which compare the produced tarball with the last post-job tarball for similarity)17:41
ttxoh well17:41
openstackgerritamrith proposed openstack-infra/project-config: Increase trove gate job timeout  https://review.openstack.org/16516117:41
jeblairfungi, ttx: actually pre-release/release == high == gate17:41
jeblaircheck == normal17:42
ttxMy tools are probably too paranoid17:42
fungiaha, even better17:42
fungianyway, not as low as post was my point17:42
*** reed has joined #openstack-infra17:42
*** sarob has quit IRC17:42
anteayacheck is normal, gate is high, post is low: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml17:42
ttxjeblair: yeah, I ususally notice the lag due to my similar-tarballs check17:42
ttxI guess I could just remove that step17:43
ttxand trust the system17:43
openstackgerritSwaminathan Vasudevan proposed openstack-infra/devstack-gate: Configure sub-nodes for Neutron DVR aiopcpu  https://review.openstack.org/15852517:43
ttx(scary but)17:43
jeblairttx: trust no one17:43
*** afazekas has quit IRC17:43
fungittx: the computer is your friend17:43
openstackgerritMonty Taylor proposed openstack-infra/system-config: Move playbooks out of the puppet module  https://review.openstack.org/16353717:43
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add some in-tree ansible group vars  https://review.openstack.org/16353817:43
ttxok, back in a couple hours17:44
*** _nadya_ has quit IRC17:44
anteayaI think the problem is that post is low priority17:44
fungiyep. by popular demand17:45
*** hdd has quit IRC17:45
anteayareally, I must have missed that one17:45
*** koolhead_ has quit IRC17:45
*** angela-s has joined #openstack-infra17:46
fungidevs wanted faster turn-around on check results when we were running under load17:46
anteayawell hope they're happy17:46
*** koolhead17 has joined #openstack-infra17:47
fungihappened back in september via https://review.openstack.org/12482617:47
openstackgerritMonty Taylor proposed openstack-infra/system-config: Manage hiera.yaml on all hosts  https://review.openstack.org/16487017:47
openstackgerritMonty Taylor proposed openstack-infra/system-config: Set $group at node scope for hiera lookups  https://review.openstack.org/16484217:47
openstackgerritMonty Taylor proposed openstack-infra/system-config: Setup hiera lookups to do common,group,fqdn  https://review.openstack.org/16484517:47
*** ghostpl_ has quit IRC17:48
*** tnovacik has joined #openstack-infra17:49
anteayaah so my read was that the balance between check and (merge-check|silent) needed to be readjusted, keeping gate ahead of check17:50
anteayaI don't see there was a whole lot of evaluation on how this would effect the post pipeline17:50
*** radez is now known as radez_g0n317:51
anteayait seems to be doing so17:51
*** dprince has quit IRC17:51
*** ParsectiX has quit IRC17:51
*** dimtruck is now known as zz_dimtruck17:52
mtreinishjeblair, fungi, mordred: so it looks like the puppet pulled in the new release, but I don't see anything about the migration exec on puppetboard. Did it start it up correctly?17:52
*** ParsectiX has joined #openstack-infra17:52
*** dizquierdo has quit IRC17:52
fungimtreinish: i'll see if i can tell17:53
*** ParsectiX has quit IRC17:53
*** ParsectiX has joined #openstack-infra17:53
openstackgerritSwaminathan Vasudevan proposed openstack-infra/devstack-gate: Configure sub-nodes for Neutron DVR aiopcpu  https://review.openstack.org/15852517:54
*** funzo has joined #openstack-infra17:54
*** dangers_away is now known as dangers17:55
*** gyee has joined #openstack-infra17:55
*** zz_dimtruck is now known as dimtruck17:56
fungimtreinish: subunit2sql 0.4.1 is definitely installed on subunit-worker01 (the puppet log is a little vague when it says "changed from 0.4.0 to latest")17:57
*** dangers is now known as dangers_away17:57
mtreinishfungi: oh, I meant on logstash.o.o, that's where the migration is being run17:57
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose tracked branch for rebase when submitting  https://review.openstack.org/15887717:58
*** dangers_away is now known as dangers17:58
mtreinishfungi: fwiw, we will have to restart the subunit-gearman worker on subunit-worker01 when the migration finishes17:59
fungiMar 17 17:25:02 logstash puppet-agent[17404]: (/Stage[main]/Subunit2sql::Server/Exec[backup_subunit2sql_db]) Scheduling refresh of Exec[upgrade_subunit2sql_db]17:59
fungiso presumably it finished since then. where is that logging to, if anywhere?18:00
mtreinishfungi: cool, I guess that means it at least started the exec. Is the process still running?18:00
mtreinishfungi: I had it write a log file to /var/log/subunit* (one sec I'll look up the full path)18:00
fungifound it18:01
mtreinishfungi: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/manifests/server.pp#n5318:01
fungithe file's empty though18:01
mtreinishfungi: could it still be running?18:01
*** baoli has quit IRC18:01
mtreinishwe background it to avoid the ansible puppet apply timeout18:01
fungiyeah, i'm checking18:01
*** gokrokve has joined #openstack-infra18:02
*** alexpilotti has joined #openstack-infra18:02
fungimtreinish: nothing's jumping out at me in the process list. what would the executable name be?18:03
fungijust making sure i'm not overlooking it18:03
fungiyeah, i filtered for subunit, sql, db... nothing18:03
*** hdd has joined #openstack-infra18:04
*** banix has quit IRC18:04
fungiso i guess if it started, is no longer running and logged nothing, success?18:04
*** shardy has quit IRC18:04
mtreinishfungi: hmm, I'm skeptical, let me check the schema18:04
*** ildikov has quit IRC18:04
mtreinishfungi: nope, still on the old schema. I guess the auto migrate magic still doesn't quite work18:05
openstackgerritRamy Asselin proposed openstack-infra/nodepool: Fix Nodepool DIB Uploads to Glance w/o region_name  https://review.openstack.org/16231318:05
mtreinishfungi: do you want to try manually kicking off the migration?18:05
*** baoli_ has joined #openstack-infra18:05
fungimtreinish: sure, what's the synyax?18:06
anteayaso I don't know as post should be in the high precedence category18:06
openstackgerrityolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created  https://review.openstack.org/16502918:06
mtreinishfungi: subunit2sql-db-manage --config-file /etc/subunit2sql.conf upgrade head18:06
anteayabut I don't think it should stay in low18:06
fungimtreinish: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.mysql_upgrade.sql'18:07
mtreinishfungi: heh, you know I had a feeling that was going to be the case18:08
mtreinishfungi: sigh, I screwed up the python code to find the path18:08
*** tiswanso has quit IRC18:08
jeblairdo puppet execs run with bash?18:08
fungianteaya: post and periodic are the only heavy users of node resources in low, and periodic is just at certain times (usually corresponding with a lower volume of activity anyway) so if we moved post to normal then low vs normal would basically have no meaning18:09
mtreinishjeblair: oh, yeah that's a good point18:09
mtreinishfungi: can you do a locate for the file: 1679b5bc102c_add_subsecond_columns_to_test_runs_table.mysql_upgrade.sql18:10
*** _shaps_ has quit IRC18:10
jeblairmtreinish, fungi: it might just be running the command itself, without a shell, which means the shell redirect won't work.18:10
mtreinishbecause that's what it should be loading, but the code to figure out the path really screwed up...18:10
jeblairmtreinish, fungi: if it is using a shell, i believe the redirect included is a bash-ism.18:10
jeblairfungi: oh, but you said the path exists, right?18:10
mtreinishjeblair: yeah, I just kinda of assumed &> would work18:11
jeblairmtreinish, fungi: that makes me lean toward it using a non-bash shell18:11
fungijeblair: yeah the file exists but is empty18:11
fungishould probably use >whatever.log 2>&118:12
jeblairyeah, i think that will work18:12
fungithat ought to work in posix-compliant bourne18:12
*** dprince has joined #openstack-infra18:12
mtreinishfungi: ok, I can push a patch up to do that instead, one sec18:12
*** yfried_ is now known as yfried|afk18:12
fungimtreinish: to answer your question, /usr/local/lib/python2.7/dist-packages/subunit2sql/migrations/versions/1679b5bc102c_add_subsecond_columns_to_test_runs_table.mysql_upgrade.sql18:13
mtreinishfungi: ok, now the question is do we want to just hardcode that path in the migration py file in the meantime while I push out a fix18:13
jeblairi looked at nodepool logs, and the providers are doing decently well right now; not an enormous number of errors18:14
*** dboik has quit IRC18:14
*** sdake__ has joined #openstack-infra18:14
*** dangers is now known as dangers_away18:14
*** dangers_away is now known as dangers18:14
*** radez_g0n3 is now known as radez18:15
*** notnownikki has joined #openstack-infra18:15
fungimtreinish: i can. just a sec18:15
*** achanda has quit IRC18:16
mtreinishfungi: it would be something like: http://paste.openstack.org/show/192954/18:16
*** baoli_ has quit IRC18:16
*** e0ne has joined #openstack-infra18:16
*** ParsectiX has quit IRC18:16
fungimtreinish: thanks, that helps18:17
*** ParsectiX has joined #openstack-infra18:17
gordcdtroyer: hey Dean, when you have time, can you re-approve https://review.openstack.org/#/c/155473/ (had a false merge conflict)18:17
anteayafungi: so we keep post in low status so that low has meaning?18:17
dtroyergordc: done18:17
*** sdake has quit IRC18:18
mtreinishfungi: sorry for being stupid in that patch, I didn't even think about there being a '.' in the path :)18:18
fungianteaya: no, check is in normal so that it doesn't have to fight with post (in low) for resources18:18
gordcdtroyer: thanks!18:18
anteayawell the big change was when experimental moved from low to normal18:18
*** pblaho__ is now known as pblaho18:19
*** pblaho has quit IRC18:19
*** pblaho has joined #openstack-infra18:19
anteayaI'm just trying to figure out a way for ttx to not be up at night18:20
fungimore nodes18:20
*** maulermark has joined #openstack-infra18:20
anteayasounds comforting18:20
*** yfried|afk is now known as yfried_18:21
*** ghostpl_ has joined #openstack-infra18:21
*** achanda has joined #openstack-infra18:21
*** pblaho has quit IRC18:21
fungii mean, it's cool to discuss which jobs should be prioritized over which other jobs when we don't have sufficient resources, but in the end there will always be people who disagree on whether the jobs they care about should get priority vs the jobs someone else cares about18:22
*** pblaho has joined #openstack-infra18:22
mordredyeah - the main way to fix that is to always have capacity :)18:22
mordredAlways. Be. Closing.18:23
anteayattx: I tried18:23
fungimtreinish: it's running now18:23
mtreinishfungi: cool thanks18:23
* mordred is excited about our new subunit2sql overlord18:23
*** ParsectiX has quit IRC18:24
*** ParsectiX has joined #openstack-infra18:24
*** tiswanso has joined #openstack-infra18:24
* pleia2 fingers crossed18:24
nibalizerjeblair: mordred should 'puppet-apply transition' get a bullet point under priority efforts in the meeting?18:25
mtreinishall this just to fix the preseeding so it works reliably and we can start to use it for reals18:25
fungianteaya: one possible compromise is to make multiple prioritized versions of some of the independent pipelines and then decide on subsets of jobs to split between them18:26
mordrednibalizer: it's not a priority effort, actually - and I've been a bad boy for working on it18:26
mordrednibalizer: however - did you see the patches I pushed up?18:26
fungianteaya: for example make a priority-post pipeline at high precedence and put the branch-tarball jobs in that18:26
*** e0ne is now known as e0ne_18:27
fungiof course, then you'd have documentation updates lagging behind tarballs, but that may or may not be a problem18:27
mordrednibalizer: I've got patches covering all of teh ansible side of teh coin - and I updated your patch to add groups for all of the things18:27
nibalizermordred: reviewing now18:27
*** arosen has quit IRC18:28
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Fix terrible path finding code in 1679b5bc102 migration  https://review.openstack.org/16517918:28
mtreinishjeblair, fungi: ^^^ there is the real fix for the stupid mistake18:28
*** tiswanso has quit IRC18:29
greghaynesmordred: trying to resurrect pypi-mirror for wheels, found https://review.openstack.org/#/c/103360/ but having a hard time figuring out where the old pypi-mirror.yaml config file is being generated (im guessing there was some puppet somewhere?)18:29
greghaynesyour name is on that review so you get the highlight ;)18:29
mordredgreghaynes: uhm ... yeah ... I believe it was just in puppet18:29
*** sputnik13 has quit IRC18:30
greghaynesso system-config?18:30
mordredI think so, yeah18:30
mordredI cannot remember where it would have been though18:30
*** pblaho has quit IRC18:30
greghaynesyeaa, and my git log grepping hasnt been too good there18:30
mordredmaybe clarkb or fungi remember18:30
jesusaurusgreghaynes: look at the history of modules/openstack_project/manifests/mirror_slave.pp18:31
greghaynesjesusaurus: tyty18:31
mordredgreghaynes: I think it was VERY small18:31
*** tiswanso_ has joined #openstack-infra18:31
*** tiswanso_ has quit IRC18:31
greghaynesjesusaurus: oh hey, theres a pypi_mirror.pp in that dir18:32
*** e0ne_ has quit IRC18:32
anteayafungi: does zuul3 take care of that functionality?18:32
nibalizergreghaynes: if you checkout 0b7898161f0adcdfebeefc4ed73b41a103f606af then git grep pypi-mirror you should be good18:32
jesusaurusgreghaynes: I specifically see an ancient reference to puppet:///modules/openstack_project/jenkins-pypi-mirror.yaml18:32
*** sputnik13 has joined #openstack-infra18:32
mordredanteaya: probably not as much18:32
*** tiswanso_ has joined #openstack-infra18:33
*** bswartz has joined #openstack-infra18:33
anteayamordred: okay18:33
mordredanteaya: although collapsing down to one node type will help some18:33
fungianteaya: the underlying issue is the gearman protocol has a grand total of three precedence values18:33
anteayafungi: ah18:33
*** Slower has quit IRC18:33
openstackgerrityolanda.robla proposed openstack-infra/system-config: Expose jenkins_gitfullname and jenkins_gitemail  https://review.openstack.org/16169518:33
mordredsince it will spread the total resources we have18:33
jesusaurusgreghaynes: also, 'git log -S' ftw18:33
fungiwhich sort of ties our hands when it comes to prioritizing things18:33
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Fix db upgrade console redirection to be posix compliant  https://review.openstack.org/16518318:33
anteayathree shall be the number, and the number shall be threee18:33
mordredanteaya: :)18:33
fungibut yeah, fewer node types will help there18:33
fungianteaya: five is right out18:33
mtreinishjeblair, fungi: ^^^ and there is the console redirection fix18:33
greghaynesjesusaurus: yep, looks like you found the file18:33
anteayamakes me look around for a holy hand grenade18:34
*** dboik has joined #openstack-infra18:34
jesusaurusgreghaynes: sweet18:34
*** baoli has joined #openstack-infra18:34
greghaynesnow to increase puppet knowledge to slightly above 018:34
anteayaI just always feel bad when ttx is stuck away into the wee hours waiting for patches to become available18:34
*** radez is now known as radez_g0n318:35
anteayattx so all I can give you is my empathy18:35
*** ddieterly has quit IRC18:35
anteayattx not worth much but its all I've got18:35
*** dboik has quit IRC18:36
anteayadhellmann: I see you are at it again, did tickled pink make the list?18:36
*** dboik has joined #openstack-infra18:36
dhellmannanteaya: tickled pink?18:36
dougwighello infra.  need one more core on this new neutron check job (lbaasv2): https://review.openstack.org/#/c/164390/18:37
*** baoli_ has joined #openstack-infra18:37
dhellmannanteaya: oh, the emotion in the release notes? heh, no, but we'll take a pull request if you want to add it :-)18:37
*** gokrokve has quit IRC18:38
dhellmannanteaya: the latest release notes script is still working its way into the release-tools repo: https://review.openstack.org/#/c/161265/18:38
anteayaI offered a patch18:38
anteayadhellmann: dims reviewed it18:38
* dhellmann goes to look for it18:38
anteayaha ha ha18:38
anteayadhellmann: https://review.openstack.org/#/c/162651/18:39
*** packet has joined #openstack-infra18:39
*** cebruns has quit IRC18:39
*** e0ne has joined #openstack-infra18:39
*** cebruns has joined #openstack-infra18:39
*** baoli has quit IRC18:39
anteayathanks dims18:39
*** tnovacik has quit IRC18:40
anteayadougwig: when shopping for reviews state the repo in question, it saves time18:40
dhellmanndims, anteaya : ah, I see one from dims, too. I should just approve those and add them to my patch against release-tools18:40
*** radez_g0n3 is now known as radez18:40
*** cdent has quit IRC18:40
anteayaI hadn't seen dims18:41
*** dboik has quit IRC18:41
dougwiganteaya: thanks, that'd be project-config18:41
* anteaya looks18:41
anteayadougwig: thanks18:41
*** dboik has joined #openstack-infra18:41
dougwiganteaya: you already +2'ed.  :-)18:41
*** dboik has quit IRC18:41
anteayaI did18:41
anteayadims: nice job18:41
anteayadhellmann: mine aren't in alphabetical order18:42
nibalizermordred: with jeblair im wondering why some but not all nodes have a group in 16484218:42
anteayadhellmann: want me to fix that18:42
dhellmannanteaya: nah, I've +2aed it so I'll fix that when I move them to the release-tools version18:42
*** mpopow has joined #openstack-infra18:42
anteayadhellmann: okay, sorry about that, I will do better next time18:42
anteayaand thanks18:42
dhellmannanteaya: we're less worried about that sort of thing than you are with the infra config files ;-)18:43
*** SumitNaiksatam has joined #openstack-infra18:43
dimsdhellmann: haha18:43
anteayawell not me, ajeager18:43
*** Somay has quit IRC18:43
*** dboik has joined #openstack-infra18:43
anteayaI have to say I'm glad I don't live with him18:43
dhellmannanteaya: that was the collective "you" :-)18:43
mordrednibalizer: because what's group do a node with a single instance of itself need to be in?18:44
*** dangers is now known as dangers_away18:44
anteayaI never move anything in the house, ever18:44
mordrednibalizer: I mean, we could totally do group-per-server18:44
anteayawhen I bring it home that is where it lives until it leaves18:44
mordrednibalizer: I just don't  know what group to make the others18:44
*** dangers_away is now known as dangers18:44
*** dboik has quit IRC18:45
*** sdake has joined #openstack-infra18:45
anteayaif he alphabetized my spice drawer there would be trouble18:45
*** yfried_ is now known as yfried|afk18:45
*** dboik has joined #openstack-infra18:45
nibalizeroh thats a solid point i just didn't get it18:45
*** maulermark has quit IRC18:45
mordrednibalizer: be sure to catch jeblair's comment on 164870 - I covered it up by uploading a new patchset18:45
mordrednibalizer: also, that patch is failing and I ahve not investigated why18:46
*** Swami has joined #openstack-infra18:46
*** mrunge has quit IRC18:46
*** maulermark has joined #openstack-infra18:46
mordrednibalizer: there it is: 2015-03-17 18:25:21.531 | Error: Could not find dependency Class[Openstack_project::Server] for File[/etc/puppet/hiera.yaml] at /home/jenkins/workspace/gate-infra-puppet-apply-trusty/openstack-infra/system-config/modules/openstack_project/manifests/base.pp:22018:47
SwamiNeed some information on multi-node setup in the gate18:47
SwamiIs there anyone who can help me answer some questions.18:47
openstackgerritSpencer Krum proposed openstack-infra/system-config: Refactor o_p::base into o_p::template  https://review.openstack.org/16283018:47
nibalizeranteaya: thanks, fixed18:47
SwamiFor the neutron multi-node setup I need to create an additional interface for the Dataplane network.18:48
*** sdake__ has quit IRC18:48
anteayanibalizer: thank you18:49
anteayaSwami: clarkb is mostly away today18:49
*** nanliu has quit IRC18:49
anteayawhy don't you ask your questions and he or others can do their best to reply18:49
anteayahe reads backscroll18:49
*** bhuvan has joined #openstack-infra18:49
Swamianteaya: thanks18:50
funginibalizer: do we need provider=>shell too in https://review.openstack.org/165183 ?18:50
SwamiAs I mentioned above to have the multi node setup for the neutron-dvr job, I need to know how I can setup an additional dataplane interface and provide the IP.18:50
anteayaSwami: you're welcome18:50
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: Import updates to release_notes.py from oslo-incubator  https://review.openstack.org/16518618:51
maulermarkLooking for an example Apache config for setting up HTMLIFY log formatting with WSGI pretty-log.conf18:52
*** Guest24264 has quit IRC18:52
*** jcoufal has quit IRC18:53
*** gokrokve has joined #openstack-infra18:53
fungimaulermark: i recognize some of those words. can you rephrase your question to make it clearer how it relates to the channel topic?18:53
anteayaSwami: have you spent anytime looking at our configuration files? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n176018:54
nibalizerfungi: i dont think so, since you can do redirection on the posix provider18:54
nibalizerjust not shell built ints18:54
Swamianteaya: yes18:54
nibalizerim wondering why its backgrounding the process though18:54
Swamianteaya: I have posted a patch, but I had a couple of questions.18:54
anteayaSwami: okay and what is your current understanding?18:54
*** ssam2 has quit IRC18:54
funginibalizer: perfect--thanks!18:54
anteayaSwami: oh well let's look at the patch, what is the url?18:54
funginibalizer: because it takes 30+ minutes to complete and we don't want to bump the exec timeout18:54
Swamianteaya: My current understanding is in the single node case there is no dataplane interface. It just has a public interface and local flat-interface.18:55
anteayaSwami: looking at code really helps18:55
nibalizerfungi: the exec timeout is set to 0 - never timeout18:55
maulermarkfungi:  I was directed here by an OpenStack resource.  Our log files are not displaying correctly on the web.  We need to format them.18:55
anteayaSwami: let's have the link to the patch please18:55
Swamianteaya: https://review.openstack.org/#/c/158525/18:55
*** nanliu has joined #openstack-infra18:55
Swamianteaya: take a look at the patch and let me know if you have any clues. I had added a comment in there for my question.18:56
*** kgiusti has quit IRC18:56
funginibalizer: interesting. has that changed? we always used to see puppet agent terminate long-running processes it initiated in the past18:56
funginibalizer: long-running foreground processes that is18:56
maulermarkfungi:  I've installed os-loganalyze and configured Apache but our logs don't display as expected.18:57
* SpamapS runs to coffee before meeting18:57
fungimaulermark: okay, so you're looking for the apache configuration we use on logs.openstack.org then. just a sec and i can link it18:57
*** tkelsey has quit IRC18:57
nibalizerwell if the '&' has been there since before then its not part of this change, it suprises me but isn't an issue18:58
anteayaSwami: okay yes, it appears you need clarkb to clarify his comment to you18:58
maulermarkfungi:  YES!18:58
*** gordc has quit IRC18:58
fungimaulermark: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logs.vhost.erb18:58
fungimaulermark: keep in mind that's an erb template we use to generate the apache vhost config from a puppet module, but should have the relevant bits for you to compare against at least18:58
morganfainbergclarkb, anteaya, the DB2 CI has formally asked to be re-enabled but they will not be voting/commenting.18:59
maulermarkfungi:  Thank you!18:59
Swamianteaya: thanks for your help, I will for clarkb's comment.18:59
fungimeeting time!19:00
pleia2meeting time19:00
morganfainbergclarkb, anteaya, they want to be able to work on solving their issues (hey we got some attention here). so they can be in compliance19:00
*** sarob has joined #openstack-infra19:00
anteayamorganfainberg: do you want them to be re-enabled?19:00
morganfainberganteaya, i am ok with them being re-enabled. they seem to be on the level19:02
morganfainberganteaya, and we have the path for escalation if they start misbehaving19:02
*** Rockyg has joined #openstack-infra19:02
anteayavery good19:02
anteayatristanC: ping for infra meeting now happening in #openstack-meeting19:03
lifelessdhellmann: yes, pbr master should be fixed. Its basically just adjusting the version output to match the current pip behaviour. Its near the top of my pile, but anyone should be able to do it19:03
SpamapSwe meeting?19:03
lifelessmorganfainberg: ^19:03
lifelessmordred: ^19:03
SpamapSdoh wrong window oops19:03
morganfainbergoh hi19:03
dhellmannlifeless: maybe you would reply to my email with the details of what needs to be fixed so someone else motivated can try it?19:03
*** dustins has quit IRC19:04
lifelesssure thing19:04
dhellmannlifeless: (we may not be in sync on the ML; I realized since you were offline when I was asking before I should send email)19:04
dhellmannlifeless: thanks19:04
*** yfried|afk is now known as yfried_19:05
*** ivar-lazzaro has joined #openstack-infra19:06
*** tim_o has joined #openstack-infra19:08
*** che-arne has quit IRC19:08
*** ddieterly has joined #openstack-infra19:09
ctlaughlooking for some nodepool help -- after making the change to cap the subunit2sql version, I was able to get nodepool to complete building an image.  Now, it is not starting new instances.  'nodepool list' doesn't show anything, and watching the debug log, it just shows this over and over: http://paste.ubuntu.com/10616976/.  I'm not sure where to look next...19:10
*** dustins_ has joined #openstack-infra19:10
*** dustins_ is now known as dustins19:10
ctlaugh^^ nodepool.log: http://paste.ubuntu.com/10616979/19:11
*** signed8bit has joined #openstack-infra19:11
*** smarcet has joined #openstack-infra19:12
*** ihrachyshka has joined #openstack-infra19:13
fungimtreinish: seems to _still_ be running btw19:15
*** yfried_ is now known as yfried|afk19:15
*** radez is now known as radez_g0n319:16
*** packet has quit IRC19:16
*** dangers is now known as dangers_away19:16
asselinctlaugh, what do you see in nodepool image-list?19:16
*** dangers_away is now known as dangers19:16
ctlaughasselin: http://paste.ubuntu.com/10617024/19:17
mtreinishfungi: yeah, I expect it will be **SLOW** but that it's still running is a good sign, because it was after 20min before we ran out of tmp disk space19:17
*** soren has joined #openstack-infra19:17
*** dangers is now known as dangers_away19:18
asselinctlaugh, strange...19:18
ctlaughasselin: I've had lots of strange recently...19:18
*** hodos has joined #openstack-infra19:19
*** dangers_away is now known as dangers19:19
ctlaughasselin: Is there somewhere else to look for possible errors / more debug logging in nodepool to enable?19:19
ctlaughasselin: I'm not seeing anything that indicates a problem, other than it not thinking it has a deficit?19:19
asselinctlaugh, can you paste more output of nodepool debug log19:19
*** dboik has quit IRC19:20
*** dboik has joined #openstack-infra19:20
*** yamahata has quit IRC19:22
ctlaughasselin: http://paste.ubuntu.com/10617045/19:23
*** HeOS has joined #openstack-infra19:25
*** jbryce has joined #openstack-infra19:25
asselinctlaugh, nodepool list shows nothing?19:25
ctlaughasselin: Here's the nodepool.yaml, in case I fouled something up there: http://paste.ubuntu.com/10617057/19:25
ctlaughasselin: Output of 'nodepool list': http://paste.ubuntu.com/10617060/19:26
*** jcoufal has joined #openstack-infra19:26
*** Slower has joined #openstack-infra19:26
asselinctlaugh, not sure but min ready > max-servers...19:27
asselinctlaugh, try chaning min ready to 119:27
*** marun has quit IRC19:28
ctlaughasselin: It's creating one now19:28
ctlaughasselin: Deficit: d-p-c: 0 (start: 0 min: 1 ready: 1 capacity: 1)19:29
*** marun has joined #openstack-infra19:29
*** dboik has quit IRC19:31
maulermarkfungi:  I must not have something configured correctly.  Logs are mis-formatted.
*** dboik has joined #openstack-infra19:31
*** tsg_ has joined #openstack-infra19:32
ctlaughasselin: thank you, so what does max-servers do?  I thought min-ready controlled the # of instances to have running, but wasn't sure how max-servers fit in.19:32
fungimaulermark: yeah, looking at the page source it seems like it's not applying any formatting at all--it's just the raw file being served as if it were html (so no line breaks)19:32
asselinctlaugh, max servers is the maximum number of instances that should be created on that particular provider19:33
fungimaulermark: you might check your apache access or error logs for any signs that something is wrong19:33
*** achanda has quit IRC19:33
*** ParsectiX has quit IRC19:33
ctlaughasselin: ok, and I guess now, it's important that it's more than min-ready...19:33
ctlaughasselin: thank you, again, for the help!19:34
fungimaulermark: also we're running that on ubuntu 12.04 which uses apache 2.2, so if you're on a different apache release that config may need to be adjusted19:34
*** ParsectiX has joined #openstack-infra19:34
asselinctlaugh, yeah, maybe a bug in nodepool math logic, so it gave up....19:34
maulermarkfungi:  This is Ubuntu 14.04.1 LTS19:34
asselinmaulermark, fungi https://review.openstack.org/#/c/164889/19:35
asselinthis worked for me on 14.04 log server ^^19:35
maulermarkfungi: Apache2 2.4.719:35
*** tgohad has joined #openstack-infra19:36
fungimaulermark: yeah, so you probably need to do additional things in the apache config. my guess is that it isn't being funneled through the wsgi thingy at all19:36
*** tsg_ has quit IRC19:39
*** yfried|afk is now known as yfried_19:39
openstackgerritMerged openstack-infra/infra-specs: Added backup/recovery description to askbot migration spec  https://review.openstack.org/16069319:40
*** kutija has joined #openstack-infra19:41
*** yamahata has joined #openstack-infra19:41
*** samueldmq has joined #openstack-infra19:42
* mkj_wk does the my-tests-work-under-git-1.7-again dance...19:42
*** banix has joined #openstack-infra19:44
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Removed username.  https://review.openstack.org/16519319:45
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Removed Nickname  https://review.openstack.org/16519419:45
*** andreykurilin_ has joined #openstack-infra19:45
*** ParsectiX has quit IRC19:46
maulermarkfungi:  I made the changes you referred to but still no luck.  It looks like it is bypassing WSGI all together.19:46
*** ParsectiX has joined #openstack-infra19:46
fungimaulermark: it's possible one of the other third-party ci operators has already trodden this ground19:46
*** ddieterly has quit IRC19:47
fungircarrillocruz: you had been working on implementing some third-party ci stuff on trusty recently, yeah?19:47
*** ddieterly has joined #openstack-infra19:47
* fungi is trying to remember who else19:47
* asselin also working on trusty third party ci19:48
fungiasselin: have you tried getting os-loganalyze working on apache 2.4?19:48
*** dboik has quit IRC19:48
fungimaulermark: is struggling19:48
pleia2I think I saw a patch earlier that was related19:49
*** dustins has quit IRC19:49
asselinI was just working ont hat yesterday19:49
*** dboik has joined #openstack-infra19:49
asselinI will test some more....just got to the apache is workikng part19:49
pleia2ah yes, asselin's patch :) https://review.openstack.org/#/c/164889/19:49
*** dustins has joined #openstack-infra19:49
*** mpopow has quit IRC19:50
maulermarkpleia2:  I implemented 164889 but still not working...19:50
*** bswartz has quit IRC19:50
*** dustins has quit IRC19:52
*** dustins has joined #openstack-infra19:52
*** ddieterly has quit IRC19:53
*** sputnik13 has quit IRC19:53
*** ddieterly has joined #openstack-infra19:53
*** radez_g0n3 is now known as radez19:54
*** dboik has quit IRC19:54
*** dustins has quit IRC19:54
*** ZZelle_ has joined #openstack-infra19:55
*** dboik has joined #openstack-infra19:55
ctlaughasselin: one last time I'll bug you today :)  I think something is amiss between zuul and gearman... I can see changes queuing up in zuul, and I see jenkins slaves (both show idle), but no tests start running.  http://paste.ubuntu.com/10617183/   I don't see anything that looks like an error in debug.log or zuul.log19:55
*** Ryan_Lane has quit IRC19:57
*** tnovacik has joined #openstack-infra19:58
*** tiswanso_ has quit IRC19:58
*** sparkycollier has joined #openstack-infra19:59
*** dboik_ has joined #openstack-infra20:00
*** dhritishikhar has quit IRC20:00
pleia2as a heads up, I'm going to be at PuppetCampSF next Tuesday (so, no meeting for me), and then the following week I'm heading back east to take care of some family stuff, so I'll be out (that's the week leading up to passover/easter)20:00
jeblairtristanC, anteaya: we can try to timebox the next meeting20:00
*** tiswanso_ has joined #openstack-infra20:01
*** dimtruck is now known as zz_dimtruck20:01
tristanCjeblair: heh, well else the bottom of the agenda is never reached :)20:01
*** tsg_ has joined #openstack-infra20:01
*** mpopow has joined #openstack-infra20:01
notnownikkijeblair, if you get a few minutes for https://review.openstack.org/#/c/163734/ and https://review.openstack.org/#/c/163402/ i'd really appreciate it, especially the first one that needs your almighty hand :)20:01
tristanCanteaya: so the proposition in the etherpad looks very good to me20:01
tristanCanteaya: but as the elections are starting in two weeks, this discussion is really for next ones20:02
asselinmaulermark, I just checked that os_log_analyze does work with those changes20:02
*** dboik has quit IRC20:02
*** tgohad has quit IRC20:02
anteayajeblair: okay thanks20:03
anteayatristanC: thank you20:03
maulermarkasselin:  Thanks for confirming.20:03
anteayatristanC: yes it will have to be, I had hoped we could use it this time, for no other reason but to help you and pleia2 as the mailing list if flooded20:03
asselinmaulermark, here's what I did: https://github.com/rasselin/os-ext-testing/tree/logserver (see instructions at the bottom of the README)...but adjust to use the 14.04 patch20:04
anteayatristanC: sorry I didn't do a better job helping you20:04
*** samueldmq has quit IRC20:04
anteayaI feel like I've left you with a big mess20:04
pleia2you didn't make the mess :)20:04
anteayaand I wanted you to have some space to get comfortable20:04
anteayano but I didn't clear the space20:04
*** melwitt_ has joined #openstack-infra20:04
anteayawhich I had hoped I could20:04
anteayano idea this would come up as fast as it did20:05
*** melwitt has quit IRC20:05
anteayaI'll do what I'm allowed to do to help youo20:05
*** melwitt_ is now known as melwitt20:05
*** carl_baldwin has joined #openstack-infra20:06
anteayapleia2: have you re-enabled a gerrit account yet?20:06
asselinmaulermark, also check to see if the file is badly formatted in the source20:06
tristanCanteaya: afaik, the mailing list volume is ok, I use a "PTL Candidacy" in subject filter and my candidacies folder is still very manageable that way20:06
anteayapleia2: because I would like account id 9751 re-enabled: http://paste.openstack.org/show/192969/20:06
*** dustins has joined #openstack-infra20:06
anteayapleia2: perhaps fungi can walk you through it20:06
anteayatristanC: glad to hear it20:07
*** jcoufal has quit IRC20:07
asselinctlaugh, not sure....I'd try to restart some of the services.....20:07
fungipleia2: ossh -p 29418 review gerrit set-account --active 975120:08
*** fifieldt has quit IRC20:08
fungier, my aliases crept in there20:08
smarcetfungi: hello, where i could preview this https://review.openstack.org/#/c/165199/?20:08
fungipleia2: ssh -p 29418 review.openstack.org gerrit set-account --active 975120:08
*** yfried_ is now known as yfried|afk20:08
pleia2sorry, was distracted20:09
fungismarcet: we probably need to add docs jobs to the openstackid project first. checking20:09
pleia2fungi, anteaya: done20:10
smarcetfungi: k tnx20:10
anteayapleia2: thank you20:10
anteayamorganfainberg: ibm db2 ci is now activated, their behaviour is up to you, if they dont' meet the infra requirements and your expectations, tell me and they go dark again20:11
*** hdd has quit IRC20:12
*** bswartz has joined #openstack-infra20:14
anteayaI sent an email to -announce20:14
*** unicell has quit IRC20:14
*** unicell has joined #openstack-infra20:14
*** ParsectiX has quit IRC20:15
*** ildikov has joined #openstack-infra20:15
*** yfried|afk is now known as yfried_20:16
fungismarcet: it doesn't have the necessary jobs yet, but i'm writing a change to the project-config repo to add them now. it'll take a few minutes because i'm mostly trying to pay attention in the tc meeting which is going on right now20:18
*** mrda-away is now known as mrda20:18
*** teran has quit IRC20:19
*** achanda has joined #openstack-infra20:19
*** whoops has joined #openstack-infra20:20
*** sdake__ has joined #openstack-infra20:21
*** fifieldt has joined #openstack-infra20:21
*** dboik_ has quit IRC20:22
smarcetfungi: ok np, thx for your help :)20:22
*** dboik has joined #openstack-infra20:22
*** dangers is now known as dangers_away20:22
*** dangers_away is now known as dangers20:22
*** hdd has joined #openstack-infra20:23
*** sputnik13 has joined #openstack-infra20:23
*** zz_dimtruck is now known as dimtruck20:24
*** sputnik13 has quit IRC20:24
*** sdake has quit IRC20:25
*** r-daneel has quit IRC20:25
*** dboik has quit IRC20:26
*** dboik has joined #openstack-infra20:26
*** julim has quit IRC20:27
*** sputnik13 has joined #openstack-infra20:29
*** sarob has quit IRC20:32
*** mpopow has quit IRC20:32
*** dboik has quit IRC20:35
*** dboik has joined #openstack-infra20:36
*** imcsk8 is now known as imcsk8|lunch20:38
*** dangers is now known as dangers_away20:39
*** dangers_away is now known as dangers20:39
*** dhritishikhar has joined #openstack-infra20:41
*** dprince has quit IRC20:41
*** Bsony has quit IRC20:42
*** boris-42 has quit IRC20:42
*** hasharAway is now known as hashar20:43
*** sigmavirus24 is now known as sigmavirus24_awa20:44
*** achanda has quit IRC20:44
*** _nadya_ has joined #openstack-infra20:45
*** achanda has joined #openstack-infra20:45
*** sdake has joined #openstack-infra20:45
mtreinishfungi: is the migration still going?20:48
fungimtreinish: yep20:48
fungiERROR:jenkins_jobs.builder:Duplicate entry found: '{pipeline}-neutron-dsvm-api{b20:48
*** dhritishikhar has quit IRC20:48
fungier, mispaste20:48
fungiERROR:jenkins_jobs.builder:Duplicate entry found: '{pipeline}-neutron-dsvm-api{b20:48
* fungi curses his terminal20:49
*** dimtruck is now known as zz_dimtruck20:49
pleia2linux paste buffers :(20:49
*** _nadya_ has quit IRC20:49
*** sdake__ has quit IRC20:49
fungiwell, i was intentionally pasting but had accidentally copied stray newlines where my terminal wrapped20:51
anteayatristanC: so openstackclient just got added to project.yaml in the governance repo20:52
anteayatristanC: so you need to find out if they are included in this round of ptl elections20:52
*** hodos has quit IRC20:53
dtroyeranteaya: fwiw, we've not had anyone step up to run before the proposal, so either way I don't expect much action there20:53
anteayadtroyer: you darling you20:54
*** Ryan_Lane has joined #openstack-infra20:54
*** mestery has quit IRC20:55
openstackgerritMonty Taylor proposed openstack-infra/infra-specs: Add spec for puppet apply work  https://review.openstack.org/16521720:57
*** carl_baldwin has quit IRC20:57
*** dkliban is now known as dkliban_afk20:58
greghaynesA spec!20:58
*** mestery has joined #openstack-infra20:59
*** alexpilotti has quit IRC21:00
stevemardtroyer will be osc ptl 4 life21:01
*** sputnik13 has quit IRC21:01
mordredgreghaynes: look! I wrote a spec!21:01
mordredgreghaynes: clarkb will be very confused that I've done such a thing21:01
*** alexpilotti has joined #openstack-infra21:02
*** teran has joined #openstack-infra21:02
*** sigmavirus24_awa is now known as sigmavirus2421:03
*** harlowja has quit IRC21:03
*** alexpilotti has quit IRC21:03
*** harlowja_ has joined #openstack-infra21:03
*** carl_baldwin has joined #openstack-infra21:04
*** skraynev has quit IRC21:04
*** alexpilotti has joined #openstack-infra21:06
*** skraynev has joined #openstack-infra21:06
*** zz_dimtruck is now known as dimtruck21:06
*** alexpilotti has quit IRC21:06
*** mpavone has joined #openstack-infra21:07
anteayastevemar: if all three of you are here, we could have a vote now21:07
*** ddieterly has quit IRC21:08
*** ddieterly has joined #openstack-infra21:08
stevemaranteaya, shouldn't be hard to wrangle all three of us up21:08
*** sputnik13 has joined #openstack-infra21:08
stevemarmight want to wait til after x-project meeting for dhellmann21:09
*** sputnik13 has quit IRC21:09
anteayastevemar: fair enough21:09
*** carl_baldwin has quit IRC21:09
*** mmedvede has quit IRC21:11
*** carl_baldwin has joined #openstack-infra21:11
*** carl_baldwin has quit IRC21:12
*** dhp has quit IRC21:12
*** smarcet has quit IRC21:13
*** imcsk8|lunch is now known as imcsk821:14
*** aysyd has quit IRC21:14
anteayapleia2: so zaro needs this patch in before we upgrade our os for gerrit on saturday, it is a system-config patch: https://review.openstack.org/#/c/165145/21:16
*** esker has quit IRC21:16
*** ldnunes has quit IRC21:17
*** mmedvede has joined #openstack-infra21:18
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Removed Nickname  https://review.openstack.org/16519421:18
*** BobH has quit IRC21:18
*** HeOS has quit IRC21:19
*** sparkycollier has quit IRC21:19
*** HeOS has joined #openstack-infra21:20
pleia2anteaya: I know, but I'm not too familiar with the javascript files in gerrit so it's going to take me a few minutes to review (it's in my queue)21:20
anteayaall it does is change the url to get the jquery file rather than the jquery.min file21:21
anteayano actual js to review21:21
greghaynesmordred: Nice idea for that spec21:21
openstackgerritEric Larson proposed openstack-infra/project-config: Create oslo.cache project  https://review.openstack.org/16522421:21
pleia2anteaya: yes, but I should probably make sure it exists at least :)21:21
anteayayes yes that would be good21:21
pleia2I'll try to get to it soon21:21
*** rfolco has quit IRC21:22
anteayaokay as you will, just trying to provide context21:22
anteayaI find commit messages sometimes to bear little relation to what the patch is actually doing21:23
anteayaand more what the dev was thinking about at the time21:23
*** BobH_ has joined #openstack-infra21:24
reeduhm, new gerrty doesn't like python ssl libs shipped with Ubuntu 14.1021:24
anteayamordred: if you actually care about the scaling election tools proposal: https://etherpad.openstack.org/p/scaling-election-tooling21:25
*** tiswanso_ has quit IRC21:25
anteayareed: what are you seeing?21:25
*** dimtruck is now known as zz_dimtruck21:26
*** dkranz has quit IRC21:26
reedanteaya, InsecurePlatformWarning21:26
*** dangers is now known as dangers_away21:26
reedlinking to https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning21:26
*** dangers_away is now known as dangers21:27
*** carl_baldwin has joined #openstack-infra21:27
*** eharney has quit IRC21:27
anteayareed: well jeblair is in the cross-project meeting, I'll let him address it when he is free21:28
*** otter768 has joined #openstack-infra21:29
*** notnownikki has quit IRC21:29
*** zz_dimtruck is now known as dimtruck21:30
*** carl_baldwin has quit IRC21:31
anteayaI rechecked this nodepool change as clarkb suggested to test swift logs: https://review.openstack.org/#/c/139704/ the logs I have a console output for all build on rax-drw, which jeblair said had the updated images21:31
anteayaso far, so good21:31
anteayaI'm going for a walk21:32
*** otter768 has quit IRC21:33
mordredanteaya: looking21:33
*** hyakuhei has joined #openstack-infra21:33
openstackgerritMerged openstack-infra/subunit2sql: Fix terrible path finding code in 1679b5bc102 migration  https://review.openstack.org/16517921:34
mordredgreghaynes: there are patches up in the puppet-apply topic that has most of that implemented already21:34
mtreinishfungi: ^^^ I guess I should hold off on pushing a release for until the current migration finishes21:34
fungimtreinish: yeah, it's still going (or else it's hung--hard to know without attaching an strace from another shell, which i'll do in a sec)21:35
fungianteaya: pleia2: zaro: fwiw you should consider that change against both precise and trusty. the distro-packaged jquery.min.js is not a symlink to jquery.js on precise21:36
fungithey are distinct21:37
*** bhuvan has quit IRC21:37
fungithat may be fine though, just pointing it out21:37
*** pc_m has quit IRC21:37
anteayaif we need it for our upgrade to trusty21:37
anteayawhat is our usecase for making sure it works with precise?21:37
mtreinishfungi: heh, I'm sure it's still moving, it's looping over >67M rows21:37
*** SwimmingDolphin has joined #openstack-infra21:37
openstackgerritSteve Baker proposed openstack-infra/project-config: Use ssh publisher for heat-templates-buildimages  https://review.openstack.org/16323321:38
openstackgerritSteve Baker proposed openstack-infra/project-config: heat-dsvm-functional enable build pip mirror  https://review.openstack.org/16522721:38
openstackgerritKhai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon  https://review.openstack.org/16522821:39
*** mattfarina has quit IRC21:39
*** signed8bit has quit IRC21:41
*** dangers is now known as dangers_away21:42
*** dangers_away is now known as dangers21:42
*** crc32 has quit IRC21:43
*** banix has quit IRC21:45
*** yfried_ is now known as yfried|afk21:45
tristanCanteaya: I can't see any reason why not including them...21:46
fungimtreinish: strace on the python process is waiting on a read(), but maybe that's just because all the work is happening on the trove side of things21:46
*** dboik has quit IRC21:46
anteayatristanC: right it is your decision, yours and pleia2's21:46
*** dboik has joined #openstack-infra21:46
anteayatristanC: I'm just letting you know their status has changed21:46
*** LinuxJedi has quit IRC21:47
*** bhuvan has joined #openstack-infra21:47
clarkbanteaya it totally worked http://logs.openstack.org/04/139704/3/check/nodepool-coverage/1f5cd29/cover/ thats cool21:47
anteayaclarkb: yay21:48
anteayaclarkb: where are you now?21:48
anteayahome or hospital?21:48
tristanCanteaya: thanks, that's very appreciated!21:48
anteayatristanC: :)21:48
anteayaclarkb: ah21:49
clarkbalmost home21:49
anteayalots of time then21:49
mtreinishfungi: yeah it's probably just waiting on something on the db side21:49
mtreinishfungi: I think there is a mysql command you can run to see the status of things going on21:49
anteayaclarkb: so right now we have I think 3 images that didn't build21:49
anteayaclarkb: jeblair didn't know why yet21:49
mtreinishfungi: show processlist maybe?21:49
anteayaclarkb: and we are open to what the next steps are for swift logs21:49
anteayaafter we get all the images building the way they are supposed to21:50
clarkbI think rollout to more jobs21:50
*** SwimmingDolphin has quit IRC21:50
anteayahave any in mind?21:50
clarkbprobably dsvm jobs21:50
harlowja_qq, anyone know why https://review.openstack.org/#/c/164836/ has a jenkins -1 but no errors (all green in the check list)21:50
*** esker has joined #openstack-infra21:51
anteayaclarkb: let's hear from jhesketh when he arrives what he wants to do next21:51
mtreinishharlowja_: it's the requirements check21:51
anteayaclarkb: I announced I'm going for a walk now, so I had better21:51
clarkbanteaya enjoy21:51
mtreinishharlowja_: you've got to hit toggle ci to see the message, the formatting is different so the js which makes the grid doesn't know what to do with it21:51
*** soren has quit IRC21:52
*** mdenny has joined #openstack-infra21:52
*** gokrokve has quit IRC21:52
*** FreeCandy has joined #openstack-infra21:53
harlowja_toggle ci u say, how's that done21:53
*** FreeCandy has quit IRC21:53
*** FreeCandy has joined #openstack-infra21:53
fungianteaya: the reason for making it work with precise is that it will totally have to if we approve it before the maintenance21:53
*** FreeCandy has quit IRC21:53
*** FreeCandy has joined #openstack-infra21:54
*** FreeCandy has quit IRC21:54
mtreinishharlowja_: there is a little button at the very bottom of the review21:54
*** FreeCandy has joined #openstack-infra21:54
*** FreeCandy has quit IRC21:54
*** FreeCandy has joined #openstack-infra21:54
*** FreeCandy has quit IRC21:54
*** asettle has joined #openstack-infra21:55
pleia2jquery.js and jquery.min.js are very different files on our precise production instance (I've now mentioned this in the review)21:56
*** esker has quit IRC21:56
*** esker has joined #openstack-infra21:56
mordredpleia2: you mean that jquery.min.js is a different version of jquery?21:56
*** LinuxJedi has joined #openstack-infra21:57
*** ChuckC has quit IRC21:57
pleia2mordred: no, I mean on the filesystem, there's a huge diff between jquery.js and jquery.min.js, merging https://review.openstack.org/#/c/165145/ will replace our currently served jquery.min.js with a new jquery.min.js-symlinked-to-jquery.js and I don't know what that'll do21:57
mordredoh  my - why would we symlink jquery.min.js to jquery.js?21:58
pleia2I don't know :)21:58
pleia2the commit message could be clearer21:58
mordredjquery.min.js should be a minified version of jquery.js21:58
zaroyes, that's true21:59
*** ayoung has quit IRC21:59
*** erikmwil_ has joined #openstack-infra21:59
*** erikmwilson is now known as Guest8460421:59
*** erikmwil_ is now known as erikmwilson21:59
zarothat should come from jquery package21:59
mordredbut it shouldn't be symlink'd to jquery.js ...22:00
*** CheeseBurga has joined #openstack-infra22:01
zaroit looks like that's how it's setup on trusty22:02
morganfainbergdhellmann, so. what i was thinking was using some form of entry point to populate the version numbers based on loading in a file (like requirements.txt, but obviously just being like a .in file that gets converted)22:02
mordreddebian is lying22:02
mordredpleia2, zaro: I'm angry at ubuntu22:02
mordredzaro is right: lrwxrwxrwx 1 root root      9 Jul  5  2013 jquery.min.js -> jquery.js22:03
*** dboik has quit IRC22:03
mordredwhat the actual hell are they thinking?22:03
pleia2it wasn't like that in precise22:03
mordredI mean, they are just lying22:03
morganfainbergdhellmann, i know it feels weird, but maybe that solves things for us in a straight-forward-ish way. publish a g-r package if something is wrong. new g-r is branch specific.22:03
mordredthey have abdicated their position as people who correctly package software22:03
zaroohh wait, there's also jquery.lit.js and jquery.pack.js which point to jquery.min.js22:03
*** dboik has joined #openstack-infra22:03
morganfainbergmordred, wow.. .min to full non minified... wtf.22:03
zaroso maybe i should change that to jquery.min.js22:03
mordredmorganfainberg: right????22:04
* zaro will do now.22:04
mordredzaro: I'm going to suggest that we not use the distro package22:04
mordredjeblair: ^^ I'm fomenting sedition22:04
mordredjeblair: and you and fungi may need to jump in22:04
greghaynesuhhh, distro package for jquery?22:04
zaroohh wait, nope they all is sim linked to jquery.js22:05
morganfainbergdhellmann, i might stew on this some more and see if it really is wrong... but it would mean that no more proposal-bot reviews for requirements updates.22:05
pleia2zaro: yeah, that's how I'm reading the info in the commit message, but on precise an actual jquery.min.js does exist and we're using it22:05
jeblairmordred: if you're volunteering to be jquery maintainer for us, cool.  :)22:06
*** yfried|afk is now known as yfried_22:06
*** hashar has quit IRC22:06
mordredjeblair: well, I believe the current maintainer has abdicated their responsibility22:06
fungimordred: part of this is pain the javascript community brings on itself by insisting on using specific non-opensource minifiers and distributing the results, so the distros can't really ship minified versions in packages because they can't ensure that they can be rebuilt22:06
pleia2zaro: I don't think we want to switch from using jquery.min.js to jquery.js on our current production gerrit server22:06
mordredfungi: that's fine ... but I think that the debian pacakge should use an open source minifier22:06
mordredor not ship a file called "jquery.min.js"22:07
morganfainbergmordred, ++ oh yes OSS minifier *or* .. damn it you typed faster than i did.22:07
zaropleia2: hmm, do you have a suggestion?22:07
fungimordred: yep. i think the latter since last i heard the open source minifiers produce broken jquery22:07
greghaynesoh, theres an amazing github bug thread about that22:07
pleia2zaro: as fungi said, I think we need to make this change support both precise and trusty if it's going in before our migration22:07
mordredkrotscheck: ^^ you know way more about this subject than I22:07
zaroi'm guessing install a package with jquery.min.js?22:07
jeblairmordred, zaro, fungi: so i think that zuul depends on a newer jquery22:07
fungiideally they should help out the minifiers if they want to start relying on them22:08
*** asettle has quit IRC22:08
mordredfungi: ++22:08
jeblairmordred, zaro, fungi: we may already have the necessary puppet to deal with that22:08
mordredjeblair: yes. I would like to investigate using the puppet we have for minifying things to minify things22:08
mordredjeblair: and zuul is already using yul minified jquery, yeah?22:08
fungiyeah. also fair to point out that review-dev running trusty is serving the non-minified jquery as jquery.min.js so it at least works i think?22:08
*** mpavone has quit IRC22:09
pleia2fungi: yeah, I'm just concerned about current precise gerrit22:09
*** harlowja_ has quit IRC22:09
jeblairhrm... actually, it looks like it still uses packages22:09
zaropleia2: supporting both doesn't mean we need to use jquery.min.js22:09
jeblairi guess i remembered wrong22:09
mordredI really wish the distros woudl stop yelling at the javascript folks and instead work on figuring out how to provide valuable service22:10
*** LinuxJedi has quit IRC22:10
jeblairso we use yui-compressor on the jquery-visibility plugin22:10
jeblairfungi: but i'm guessing that would be broken for main jquery?22:10
pleia2zaro: my concern is that this patch switches from using an actual jquery.min.js to turning jquery.min.js into a symlink to a totally different file at jquery.js, on our production instance of gerrit22:10
pleia2maybe it'll be fine, but I don't know22:11
*** Somay has joined #openstack-infra22:11
*** rlandy has quit IRC22:11
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/devstack-gate: Add KEEP_WORKSPACE_CACHE variable to devstack-vm-gate-wrap  https://review.openstack.org/16523422:11
*** LinuxJedi has joined #openstack-infra22:11
*** harlowja has joined #openstack-infra22:11
*** nilasae has quit IRC22:12
fungijeblair: maybe. i honestly haven't tried and the last time it came up in discussions on debian-devel was roughly a year ago. so might have improved22:12
pleia2(I didn't -1, just left a comment with this info)22:12
zaropleia2: i think we've already decided that Gerrit 2.9 will _not_ be supported on precise so should this just go in after gerrit upgrade?22:12
jeblairzaro: i thought you said this had to go in before the upgrade?22:13
zarothat's pleia2 told me about the precies /trusty difference22:13
zarowith jquery22:13
*** LinuxJed_ has joined #openstack-infra22:13
zaro*that's/that's before22:14
*** sigmavirus24 is now known as sigmavirus24_awa22:14
*** angela-s has left #openstack-infra22:14
zaroso i guess this should go in after upgrade? which will fix the toogle ci button22:14
*** Rockyg has quit IRC22:15
pleia2zaro: also, can you mention in the commit message that it's fixing the toggle ci button? (I had no idea why this patch was going in)22:15
zaroohh actually trusty upgrade will break jquery i guess.  so this should go in after trusty upgrade22:15
jeblairzaro: the toggle ci button is broken?22:15
*** yfried_ is now known as yfried|afk22:16
mtreinishmordred: do you have any suggestions for ways to estimate how much longer the migration will take?22:16
mordredmtreinish: not good ones22:16
zaroit was broken on review-dev.o.o because toggle-ci needs jquery22:16
*** dangers is now known as dangers_away22:16
jeblairzaro: i thought fungi just said it was being served on review-dev22:16
*** LinuxJedi has quit IRC22:16
*** dangers_away is now known as dangers22:16
zaroi manually fixed it22:16
zaroturned off puppet for now22:16
jeblairzaro: what did you do?22:16
mtreinishmordred: heh, fair enough22:17
zarocopied jquery.js to jquery.min.js22:17
jeblairzaro: where?22:17
zaroin cp /usr/share/javascript/jquery/jquery.js /home/gerrit2/review_site/static/jquery.min.js22:18
*** hyakuhei has quit IRC22:18
jeblairzaro: okay.  i'm going to install yui-compressor and see if it produces a working jquery22:18
zaroturned off puppet to work onn changing the WIP icon as well22:18
jeblairzaro: what was there before you copied it?22:19
zaroa broken simlink22:19
mordredzaro: btw ... WIP icon ... AWESOME22:20
mordredzaro: best thing I've seen in days22:20
*** jgrimm is now known as zz_jgrimm22:21
jeblairzaro, fungi:  yui-compressor -o jquery.min.js /usr/share/javascript/jquery/jquery.js22:22
jeblairzaro, fungi: that seems to work on review-dev22:22
fungiyeah, if yui-compressor generates a working minified jquery then puppeting an exec triggered from updates to that file should be fairly straightforward22:22
jeblairfungi: yeah, that sounds workable to me22:23
*** nelsnelson has quit IRC22:23
fungiand we can stick them straight into the directory gerrit expects if so, rather than bothering with a symlink there22:23
jeblairzaro: that make sense?22:23
zaroyep, will do22:23
*** arosen has joined #openstack-infra22:23
mordredyay! I produced positive results by being negative and grumpy22:23
*** spzala has quit IRC22:24
mordredthis leads me to believe that I should be negative and grumpy more often22:24
*** peristeri has quit IRC22:24
*** arosen has quit IRC22:24
pleia2oh great22:24
*** harlowja has quit IRC22:24
greghaynesI believe statisticians refer to this as 'overfitting'22:24
*** harlowja_ has joined #openstack-infra22:24
* jeblair tries to distract mordred for the good of the team22:24
*** erikmwilson has quit IRC22:25
*** pfallenop has quit IRC22:25
*** sputnik13 has joined #openstack-infra22:28
*** crc32 has joined #openstack-infra22:29
*** timcline has quit IRC22:30
*** sputnik13 has quit IRC22:30
*** dangers is now known as dangers_away22:30
*** sputnik13 has joined #openstack-infra22:31
*** dangers_away is now known as dangers22:31
mtreinishanteaya: if you get a sec we've got some patches in tempest which need https://review.openstack.org/#/c/164451/ before we can land them22:31
*** adrian_otto has joined #openstack-infra22:33
*** unicell has quit IRC22:34
*** radez is now known as radez_g0n322:34
*** dustins has quit IRC22:34
*** unicell has joined #openstack-infra22:35
*** sambetts has quit IRC22:36
*** ddieterly has quit IRC22:36
*** e0ne has quit IRC22:36
greghaynesnibalizer: in site.pp, is it sane to glob out a node 'foo' definition? I want node '*.wheelmirrors.openstack.org', for example?22:37
*** whoops has quit IRC22:38
*** sambetts has joined #openstack-infra22:38
*** andreykurilin_ has quit IRC22:38
crinkleit uses regexes, not globbing22:38
nibalizercan also regex if you prefer22:38
greghaynesis that just node *.wheelmirrors.openstack.org { }22:38
*** tnovacik has quit IRC22:39
crinklegreghaynes: https://docs.puppetlabs.com/puppet/latest/reference/lang_node_definitions.html22:39
nibalizerclarkb: i thought it uses both22:39
greghaynesdocs, what are these22:39
nibalizerfor puppet? puppet doesn thave docs!22:40
*** marun has quit IRC22:40
*** dimtruck is now known as zz_dimtruck22:40
*** amitgandhinz has quit IRC22:42
*** erikmwilson_ is now known as erikmwilson22:43
openstackgerritgreghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror  https://review.openstack.org/16524022:44
greghaynesnibalizer: crinkle ^ time to tell me how bad my first puppet code his22:45
*** asettle has joined #openstack-infra22:45
stevemarare spontaneous merge conflicts a thing now?22:45
stevemarah the 'depends-on' commit was merged22:48
*** tim_o has quit IRC22:50
*** stevemar has quit IRC22:52
*** adrian_otto has quit IRC22:53
*** reed has quit IRC22:53
openstackgerritKhai Do proposed openstack-infra/system-config: Fix jquery setup on Gerrit server.  https://review.openstack.org/16514522:53
zaropleia2: ^22:53
*** tkelsey has joined #openstack-infra22:54
*** ajmiller__ has joined #openstack-infra22:55
greghaynesIs it a problem that its going to exec every time we puppet that box?22:56
greghaynessometimes those compression runs arent very quick22:56
*** sgordon has quit IRC22:56
zarogreghaynes: we don't puppet gerrit very often22:57
*** ajmiller_ has quit IRC22:58
*** sabeen has quit IRC22:58
*** mriedem is now known as mriedem_away22:59
*** tkelsey has quit IRC22:59
*** changbl has quit IRC22:59
*** rlucio has joined #openstack-infra23:02
*** ociuhandu has joined #openstack-infra23:04
SpamapSI've hit up arrow 3 times in a row with 'tox -epy27 -r' in the buffer instead of 'tox -epy27'23:04
* SpamapS taps fingers as stuff compiles.. again23:04
nibalizerzaro: puppet doesn't run on gerrit?23:06
nibalizeror it doesnt run every few mintues, its a manual thing?23:06
*** emagana has quit IRC23:08
openstackgerritKhai Do proposed openstack-infra/system-config: Fix jquery setup on Gerrit server.  https://review.openstack.org/16514523:08
*** dims has quit IRC23:09
mordredSpamapS: :)23:09
*** paul-- has quit IRC23:09
mtreinishSpamapS: heh, I know how you feel, my stupid migration has been running for hours now...23:09
zaronibalizer: yes it does, and i believe it updates at the same interval as the rest of the infra servers.23:10
crinklegreghaynes: it shouldn't run every time, because of the refreshonly23:10
crinkleyou could add an unless or onlyif parameter but i think the refreshonly takes care of it23:10
greghayneshuh, how does it know when to refresh since it doesnt know what the compression output would be until it compresses23:11
nibalizercrinkle: i think greg was refering to this, which doesn't have any of the that https://review.openstack.org/#/c/165145/3/modules/openstack_project/manifests/gerrit.pp23:11
nibalizerunless im blind and its there and i dont see it23:11
crinkleoh i'm looking at something else23:11
greghayneswell, I dont think you can, you have to get clever and refresh when the source file changes23:11
openstackgerritgreghaynes proposed openstack-infra/nodepool: Check image-list output in cmd tests  https://review.openstack.org/16435623:12
openstackgerritgreghaynes proposed openstack-infra/nodepool: Indicate error when deleting an invalid snapshot  https://review.openstack.org/16447223:12
openstackgerritgreghaynes proposed openstack-infra/nodepool: Test snapshot image deletion command  https://review.openstack.org/16447323:12
nibalizergreghaynes: so thats what puppet usually does, you add refreshonly => true, and subscribe => File[$target_file], and then it will only run the command if puppet also changed $target_file23:12
greghaynesah, well thar ya go23:13
nibalizerofc that breaks down if the file isn't managed directly by puppet, a file form a packge or a component of a git repo or something23:13
*** paul-- has joined #openstack-infra23:13
nibalizerbut in those cases you can just subscirbe => Package[foo], and sometimes you'll do work you didn' thave to do23:13
nibalizerhow long does it take to do tha tcompression?23:14
greghaynesyea yea, its just the on update hook23:14
nibalizerby default puppet timeouts at 300 seconds23:14
*** enikanorov has quit IRC23:14
*** enikanorov has joined #openstack-infra23:15
zaronibalizer: it's very fast23:15
zaroless than 5 secs23:16
zaronibalizer: more precisely 2 secs23:16
nibalizerah, no worries then :)23:16
*** dims has joined #openstack-infra23:20
*** ihrachyshka has quit IRC23:21
*** maulermark has quit IRC23:23
ianwgreghaynes: 164356 is looking better, i think there's room to tighten it.  personally i prefer the decorator to reduce a level of indenting, but whatever on that23:24
*** dmorita has quit IRC23:26
*** dangers is now known as dangers_away23:26
*** ChuckC has joined #openstack-infra23:26
*** dangers_away is now known as dangers23:26
*** mfink_ has quit IRC23:28
openstackgerritStephanie Miller proposed openstack-infra/puppet-zanata: Initial commit of Zanata-specific sections  https://review.openstack.org/14794723:28
greghaynesyea, thats a preference thing, unless its totally straightforward I try to avoid them because IMO its less straight forward than a context maanger and also with the default argument it gets funky23:29
greghaynesI am really +0 on that though so meh23:29
*** dmorita has joined #openstack-infra23:30
*** wenlock has quit IRC23:30
*** hdd has quit IRC23:31
*** dangers is now known as dangers_away23:34
harlowja_so qq, i'm trying to get the newer requirement for debtcollector into oslo-incubator for a commit, but i am thinking the 26 checks are borked23:34
*** dangers_away is now known as dangers23:34
harlowja_''Expected version spec in', 'debtcollector~=0.3.0', 'at', '~=0.3.0''23:35
harlowja_any ideas from folks?23:35
*** mfink_ has joined #openstack-infra23:35
openstackgerritgreghaynes proposed openstack-infra/nodepool: Check image-list output in cmd tests  https://review.openstack.org/16435623:35
*** Ryan_Lane has quit IRC23:35
harlowja_i tried debtcollector>=0.3.0 and that didn't work either, ha23:35
harlowja_that borks with http://logs.openstack.org/36/164836/7/check/gate-oslo-incubator-requirements/03016de/console.html23:35
harlowja_so maybe folks here have some idears23:35
*** nelsnelson has joined #openstack-infra23:36
dstufft~= is new in pip 6 setuptools 823:37
openstackgerritgreghaynes proposed openstack-infra/nodepool: Indicate error when deleting an invalid snapshot  https://review.openstack.org/16447223:38
harlowja_pew pew pew, will all 2.6 be dying soon?23:38
openstackgerritgreghaynes proposed openstack-infra/nodepool: Test snapshot image deletion command  https://review.openstack.org/16447323:38
dstufftpip 6 setuptools 8 supports Python 2.623:38
dstufftbut if your venv doesn't have it then you can't use ~=23:38
harlowja_hmmmm, something infra releated to make that happen?23:38
*** crc32 has quit IRC23:39
*** nanliu has quit IRC23:39
dstufft(I don't know what version you're using, just saying that error looks like your pip isn't new enough in that virtual environment)23:40
*** bhunter71 has quit IRC23:40
*** Qiming has joined #openstack-infra23:40
*** bdpayne has quit IRC23:41
harlowja_ya, it would appear so; maybe new CI stuff is needed (not sure either)23:41
*** Swami has quit IRC23:42
*** crc32 has joined #openstack-infra23:44
*** hdd has joined #openstack-infra23:45
*** banix has joined #openstack-infra23:46
*** ddieterly has joined #openstack-infra23:47
CheeseBurgaChanServ openstackstatus __afazekas _qb aarefiev abramley achanda adalbas adam_g adreznec Adri2000 ajmiller__ alaski alazarev alex_xu ameade andreaf anteaya anthonyper antonym apmelton Apsu armax arosen1 asadoughi asettle ashleighfarnham asselin banix baoli_ bcourt begal-sop_ bhuvan bkero bknudson blitzrage bnemec BobBall_AWOL BobH_ bodepd23:47
CheeseBurgabookwar bradm briancline briancurtin bswartz btran camunoz cburgess cebruns charz CheeseBurga chlong chmouel ChuckC cinerama clarkb Clint cloudnull cody-somerville comstud coolsvap|afk crc32 crinkle cschwede ctlaugh ctracey cyeoh d0ugal d34dh0r53 dalgaaf dangers dank_ dannywilson dansmith Darkwan darvon david-lyle davidlenwell Daviey dboik23:47
CheeseBurgaddieterly devananda dguerri dhellmann dims DinaBelova dkehn dkliban_afk dmellado dmitryme dmorita dmsimard dolphm dosaboy doug-fish dougwig dstanek dstufft dtantsur|afk dteselkin dtroyer DuncanT dvorak edwarnicke ekarlso elarson EmilienM enikanorov enikanorov__ erikmwilson erw esheffield__ esker esmute fabo fifieldt filler flaper87 freerunner freyes23:47
CheeseBurgafrickler frobware fungi funzo gaelL_ garyh GheRivero ghostpl ghostpl_ gibi gilliard gmann gnuoy gothicmindfood gpocentek grantbow greghaynes grue_pm gus gyee hamzy harlowja_ hdd Hefeweizen HenryG HeOS HeOS_ heyongli hockeynut hogepodge hrubi hughhalf hughsaunders Hunner ianw ianychoi igorbelikov ildikov ilyashakhat imcsk823:47
CheeseBurgaisq ivar-lazzaro j^2 jamespage jamielennox JayF jaypipes jbryce jcooley jd__ jeblair jesusaurus jgriffith jhesketh jklare jkt jlvillal jogo JoshNang jpeeler jraim jroll juggler jvrbanac jyuso jyuso1 kaisers kashyap katyafervent kevinbenton Kiall kiseok7_ kmartin koolhead17 kragniz krotscheck krtaylor kutija laurelm lbragstad23:47
CheeseBurgalifeless LinuxJed_ liusheng loquacities lxsli mancdaz marcusvrn1 markmcclain markvan markvoelker martingc masayukig mattoliverau maurosr mdenny med_ melwitt mescanef_ mestery mfink mfink_ mgagne michchap mika mikal Mithrandir mjturek1 mkj_wk mkoderer mmedvede Mmike mordred morgabra morganfainberg mrda mriedem_away mrodden msuriar mtanino mtreinish23:47
CheeseBurgamugsie mwagner_lap mwhudson Nakato naotokl nelsnelson neoice Ng nibalizer niedbalski nijaba nikhil_k NikitaKonovalov niska nithyag__ NobodyCam notmyname ociuhandu odyssey4me ondergetekende_ openstack openstackgerrit otherwiseguy pabelanger patrickeast paul-- PaulMurray pcrews pczesno persia peterstac phschwartz pleia2 prad praveenkumar primeministerp ptoohill purp Qiming rackertom23:47
CheeseBurgaradez_g0n3 raginbajin rainya rakhmerov rcarrillocruz redrobot rhe00 rkukura rlucio rm_work roaet_ rossella_s ruhe russellb rwsu ryanpetrello s0nea sabari salv-orl_ sambetts saper sbadia sc68cal schang sdague sdake sdake_ SergeyLukjanov SergK Shrews sigmavirus24_awa sileht simonmcc skraynev slagle SlickNik Slower Somay SotK SpamapS23:47
CheeseBurgaspiffxp spiffxp- spredzy|afk sputnik13 sreshetnyak stevebaker StevenK SumitNaiksatam Svedrin sweston taron tchaypo teran TheJulia therve timfreund timrc tlbr tnurlygayanov toabctl tobiash tonytan4ever tqtran trey tristanC tsg_ tsufiev_ tteggel ttx ubuntu2 ujuc unicell viglesia1 vipul vishy Vivek vponomaryov wayneeseguin waynr wendar23:47
CheeseBurgawillroberts_ wolsen woodster_ wznoinsk xianghui xnox yamahata yfried|afk yolanda zaro zhiyan zigo zigo_ zxiiro zz_avozza zz_dimtruck zz_ja zz_jgrimm zz_johnthetubagu ZZelle ZZelle_23:47
greghayneswell, id ping ops but apparently this spammer has done it for us23:48
ianwgreat, now i'm hungry23:48
dstufftI didn't get a ping23:48
JayFI just ping'd #freenode23:48
dstufftI feel left out23:48
*** CheeseBurga has quit IRC23:48
*** ghostpl_ has quit IRC23:49
*** tsg_ has quit IRC23:49
mattoliverauAnd here I thought someone wanted to talk to me :p23:49
mmedvedehi everybody :)23:49
taronmethinks it is time for me to get a separate highlight window again23:49
*** ajmiller__ has quit IRC23:50
*** david-lyle is now known as david-lyle_afk23:50
*** oomichi has joined #openstack-infra23:51
*** ddieterly has quit IRC23:51
*** ayoung has joined #openstack-infra23:53
*** yamamoto has joined #openstack-infra23:53
*** ZZelle_ has quit IRC23:54
*** SumitNaiksatam has quit IRC23:56
*** ddieterly has joined #openstack-infra23:57
*** sgordon has joined #openstack-infra23:57
*** Qiming has quit IRC23:58
*** dboik_ has joined #openstack-infra23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!