Monday, 2014-09-08

*** marun has quit IRC00:01
*** imeerovi has joined #openstack-infra00:08
*** gokrokve has joined #openstack-infra00:09
*** jpich has joined #openstack-infra00:12
*** weshay has quit IRC00:13
*** gokrokve has quit IRC00:14
mordredjogo: ok. I responded to your cycle goals thing00:15
mordredjogo: hopefully I won't get voted off the island00:15
annegentlemordred: heh00:15
clarkbmordred: the tribe has spoken00:15
annegentlemordred: I'm still formulating my response00:15
mordredtl;dr - end user experience in openstack SUCKS and we shoudl do nothing but fix it00:15
annegentleif only there was an openstack island. with beaches00:15
mordredannegentle: I like beaches00:15
annegentlemordred: how were the beaches last week?00:16
mordredannegentle: EXCELLENT00:16
annegentlemordred: and was burning man a mudpit?00:16
annegentledoes mud burn?00:16
mordredannegentle: it was for a few hours ... but it is a desert, so that did not last more than half a day00:16
mordredannegentle: anything can burn if you try hard enough00:17
annegentlegas mixes with dirt right?00:17
*** dkranz has quit IRC00:18
clarkbmordred: ++ (note 2 is a subset of 1 imo)00:19
anteayaI'm enjoying imagining the collective intake of breath from the vendors00:29
anteayathe air just all went out of the room00:29
*** yamamoto has joined #openstack-infra00:30
*** yamamoto has quit IRC00:34
*** oomichi has joined #openstack-infra00:35
*** jpich_ has joined #openstack-infra00:36
jogomordred: awesome thanks00:38
jogomordred: ++ to fixing user experience00:38
jogomordred: I like  "less features more win"00:38
*** jpich has quit IRC00:40
*** garyh has joined #openstack-infra00:40
*** mmaglana has joined #openstack-infra00:43
clarkbmordred: you taking shots on "OMAHA"s ?00:45
*** mmaglana_ has joined #openstack-infra00:45
*** dimtruck is now known as zz_dimtruck00:46
jogowow gate is empty ^_^00:47
*** mmaglana has quit IRC00:48
*** mmaglana_ has quit IRC00:50
bknudsonjogo: we keep rechecking but things keep failing pip installs00:50
*** mmaglana has joined #openstack-infra00:50
mordredclarkb: I do not know what you speak of?00:51
clarkbmordred: Manning screaming "Omaha" the quickest way to alcohol poisoning is to make it a drinking game00:51
clarkbbknudson: :/ I think we have somewhat proven that its internal cloud networking that is broken. Maybe we can get neutron folks to describe how we should be deploying onto it00:52
*** mmaglana has quit IRC00:55
bknudsonclarkb: seems pretty random but also happening too often for us to get a review to verify.00:55
clarkbits not random. it is 99% of the time happening in hpcloud00:55
*** mmaglana has joined #openstack-infra00:56
clarkbso last week we added per region pypi mirrors but taht hasn't fixed the problem. so not Internet being bad but cloud networking being bad00:56
bknudsonmordred: we had some of the discussion about "client library should contain exactly ZERO admin functions" early on in the openstack-sdk discussions00:57
mordredbknudson: I agree with them00:57
mordredbknudson: or, at the very least, there should be a clear and simple interface that lets 90% of the users do 90% of the things easily00:57
mordredand the hard/complex/special things can be somewhere else00:57
mordredbecause the people who just want a server do not care about them00:58
bknudsonmordred: I was hoping we'd end up with maybe a low-level API and a higher-level API built on it. I'm not sure what direction it went since I got too busy to help out with it.00:58
mordredbknudson: I believe that's what happening00:58
mordredright now there is only a low level API00:59
clarkbI do note that hpcloud b1, b3, and b5 appaer to have the most failures00:59
clarkbb2 and b4 are fine. I wonder if b1,3,5 are the ones we shifted load onto00:59
jogofungi: the ATC tool01:03
jogowhy not just use git logs?01:04
jamielennoxmordred, bknudson: there is a fairly low level, API version specific interface for a number of the services, the intention is then to build a cross-version simpler API on top of that01:04
jamielennoxan email went out very recently, part of that is asking for ideas of what the ideal higher level interface would look like, because i don't know/think it's the same as the current clients01:05
jamielennoxif you have any ideas they would be welcomed01:05
mordredjamielennox: yes - I'm working up a good response to that01:06
mordredjamielennox: I can tell you from current experience that it is NOT what the current libs look like :)01:06
*** zz_dimtruck is now known as dimtruck01:06
clarkbjamielennox: yes please less of this spaghetti kwargs everywhere code :)01:06
jamielennoxmordred: excellent, i'm hoping we can get together a few examples and make a decision at summit01:06
mordredjamielennox: I was telling dtroyer_zz the other day, I've been cooking up a lib recently that is an extraction of all of the code that infra uses as well as the code I've been writing for ansible recently01:07
mordredjamielennox: it currently uses the python-*client libs - but I think it might make a good straw man for the simple API - such that taking it and re-working the backend to use the low-level SDK code would be a fun exercise01:08
mordredjamielennox: so I'll also try to just send some ugly patches to talk about too01:08
jamielennoxmordred: cool - look forward to seeing it, i completely agree about the existing clients01:09
*** gokrokve has joined #openstack-infra01:09
jamielennoxit can be hard though having messed with them for so long to come up with completely new approaches, i would love to know what new people coming to OS expect to find01:11
jamielennoxbecause i'm pretty sure that boto is not the answer here either01:12
* mordred shudders01:12
clarkbjamielennox: I think the actual api matters less than having reasonable documentation so that I don't have to read nova code to figure out if this feature in python-novaclient is intended to work. And the high level api needs to cover common use cases. I want a node with a public ip. I want a node with a persistent volume01:13
clarkbjamielennox: doing those things should be easy and documented. What the actual client api looks like is less my concern01:13
*** mestery_ is now known as mestery01:14
*** gokrokve has quit IRC01:14
jamielennoxclarkb: definetly a good point01:15
jamielennoxclarkb: particularly that i'd like to care less about compute vs storage vs identity and just deal with operations on openstack01:15
clarkbjamielennox: yup exactly01:15
clarkbbecause as a user my intent is to have "tangible" things. not worry about what service provides which thing01:16
clarkbI shouldn't need to know that I talk to nova for a compute node, then I go talk to neutron for a floating ip, then I go talk to cinder for a volume and wire it all up before running config management on it01:16
clarkbI should just get network, cpu, and storage01:16
jamielennoxyep, i don't see the clients going anywhere - if you know enough to know which service you need they will be available, but that's not the 90% case01:17
*** harlowja_at_home has joined #openstack-infra01:22
lifelessjamielennox: my concern is that programmers writing to the API01:22
lifelessjamielennox: don't get any glue we put in the 'clients' - we need to sort our stuff out server side so that the restful API is actually clean01:22
lifelessjamielennox: whether thats a facade in front of the whole conglomeration or something else I dunno.01:23
clarkbanother thing that we need that is related to all this is "standardized" networking. We are currently strugglign with neutron because apparently we are doing it wrong01:23
clarkbwell what is the correct way to be setting this stuff up?01:23
morganfainbergmordred, jogo ++ on ux stuff01:29
morganfainbergjogo, i only responded to sdague's post because he covered most of the stuff that I would have phrased in there. though, mordred's response is awesome.01:29
*** yamamoto has joined #openstack-infra01:30
*** openstack_ has joined #openstack-infra01:31
jamielennoxlifeless: so i think the answer to that is that the clients are opinionated and start paying attention to what is published by various APIs and at what versions01:32
jamielennoxkeystone for example publishes at GET / the current version of it's API to minor level (3.3) and we should be able to know client side what that means and use it01:32
jamielennoxand just not support things we feel are stupid like the ability to throw random stuff at an API and have that serialized to a backend somewhere01:33
jamielennoxwhich is at least part of the reason we can never get out of using kwargs there01:33
*** JadnG has quit IRC01:35
jamielennoxclarkb: and 'i have no idea what to do about neutron' is my answer to anything like that01:36
mordredlifeless: I think nobody will write to the api01:36
lifelessjamielennox: I think that will be nicer but still keep the balkanisation intrenched01:36
*** yamahata has joined #openstack-infra01:36
mordredI mean, fix the rest api, sure, but that will take 5 years and never happen01:38
mordredwhereas we  can make a lib interface that dwim pretty easily by comparison01:39
lifelesssure, times N languages01:39
mordredexcept fog and jclouds01:40
lifelessthey already have a thing01:40
*** yamamoto has quit IRC01:43
*** erlon has joined #openstack-infra01:47
morganfainbergmordred, i think even with the SDK we'll be surprised how many people write to the APi. but... it'll definitely be a smaller-ish number01:47
*** nosnos has joined #openstack-infra01:51
*** bhuvan has joined #openstack-infra01:51
*** dims has joined #openstack-infra01:54
*** jpich_ is now known as jpich01:57
*** unicell has joined #openstack-infra02:00
mordredmorganfainberg: I would be surprised at that ... I never want to code to rest APIs ...02:01
*** HenryG_ is now known as HenryG02:01
morganfainbergmordred, totally agree on not wanting to code to rest apis :)02:02
morganfainbergmordred, i'm just guessing we will have orgs that want to do it "their" way not the SDK way. it's not a huge deal, but it'll be a bit more than anyone will truely expect if the SDK is solid.02:02
mordredyah. this is also the reason I want to focus on 80% stuff like what clarkb was referencing02:03
mordredif you wanna be fancy, ossum02:04
mordredbut if all you want is a node that works ... that should be like one function call02:04
harlowja_at_homemordred,  lifeless  u got a sec for a pbr question?02:04
mordredharlowja_at_home: on my phone, but I can try02:05
harlowja_at_homei've been trying to get the changelog included in the doc building process (to show like a history link with the changelog), it seems that i have to have the docs venv run python sdist to produce the changelog so that then the python build_sphinx command and associated stuff can include it, is that the normal way of doing this, or is there something easier?02:06
harlowja_at_home (was an attempt at this)02:06
morganfainbergmordred, def. cannot disagree on any front.02:06
harlowja_at_home seems to though be empty, ha02:06
mordredthere is not a better way, bit there should be02:07
mordredit's a function call in pbr, just not exposed except via sdist02:08
clarkbmordred: harlowja_at_home: this wouldn't be an issue if we went back to tox isntalling an sdist02:08
clarkbmordred: harlowja_at_home: which also covers the can we build an sdist question (this came up when I think it was heat that broke their sdist builds at some point)02:08
clarkbbut its slow02:08
mordredalso, sdists are mostly lame anyway02:09
mordredexcept when they aren't02:09
mordredeven still, there should be a setup02:09
harlowja_at_homeya, i just need it for the changelog (except that it didn't seem to get sucked in anyway, haha) - although trying this locally it does, which is odd02:09 target02:09
harlowja_at_homemordred, thx, guess that can be a future change, now i wonder why the changelog didn't get included; probably something off there i guess02:12
harlowja_at_homedoes the doc job just run the docs venv, or is that via some other mechanism (if it didn't run that venv, that would explain why its not there)02:14
openstackgerritDavid Shrewsbury proposed a change to stackforge/dox: Logging improvements
clarkbharlowja_at_home: it does not run the docs venv. docs venv is non standard and honestly I wish there was a bit more conversation about adding it so that we can do it right02:14
clarkbharlowja_at_home: docs jobs do `tox -evenv python build_sphinx`02:14
harlowja_at_homeah, hmmm02:15
harlowja_at_homewell that would explain that02:15
clarkbharlowja_at_home: see
clarkbthe docs target is ad hoc in some projects. other projects have 'doc'02:15
clarkbtl;dr don't count on it02:15
clarkbI am open to updating the project testing interface but that conversation hasn't come up02:15
lifelessharlowja_at_home: shoot02:16
harlowja_at_homeya, i thought docs job run docs venv, and all would be merry, ha02:16
harlowja_at_homethen my changelog inclusion plan would have worked out02:17
lifelessclarkb: neutron broke install02:17
*** yamamoto has joined #openstack-infra02:19
lifelessharlowja_at_home: we can add a command to do the changelog02:19
harlowja_at_homelifeless,  can it be connected into the build_sphinx ; so that people can include that changelog in the sphinx stuff02:19
harlowja_at_homeapparently its working (somehow) for oslo.concurrency @ (maybe cause they pip install themselves?)02:20
harlowja_at_homeor it could be a setup.cfg option for [build_sphinx] changelog = 1 or something02:20
SlickNikclarkb / mordred: seeing some failures in the check and gate jobs where the jobs aren't able to fetch from Known issue?02:22
anteayaSlickNik: possibly the network issue primarly on hpcloud nodes that clarkb mentioned in scrollback02:22
lifelessharlowja_at_home: possibly yeah02:22
anteayaSlickNik: have you a log to confirm?02:22
lifelessharlowja_at_home: would need to make sure we don't break building docs without git etc02:23
SlickNikanteaya: eg., and eg.
anteayawow that was a fast failure02:23
SlickNikerr, sorry copy paste error:
harlowja_at_homelifeless,  agreed02:24
anteayafatal: Could not read from remote repository02:24
*** tomoe_ has quit IRC02:25
anteayanothing odd jumping out on the farm:
*** otter768 has quit IRC02:27
anteayaI have forgotten the command to specify where i want to clone from02:27
anteayaa random clone completes without a hiccup for m02:28
SlickNikanteaya: yup, same here — works fine for me so it's possibly related to the hpcloud network connectivity issue that clarkb mentioned earlier.02:28
SlickNikwill keep an eye out to see if it's still happening.02:29
SlickNikBut hopefully, 'twas just something intermittent02:29
anteayawe can hope02:29
anteayathis works but this is from gerrit: git clone ssh://
anteayathis might just mean I hve the url incorrect02:34
anteayaor it might mean git05 is unhappy02:34
anteayaI must have the url incorrect, git01 - git05 give me the same response02:35
anteayaon closer inspection it looks like git03 had a hiccup but it appears to be back now:
*** daya_k has joined #openstack-infra02:41
anteayamy url was wrong, I had to add the port number02:43
anteayagit clone git://
anteayaworked from all 5 servers02:43
*** _nadya_ has joined #openstack-infra02:50
*** rushiagr_away is now known as rushiagr02:52
*** dims has quit IRC02:56
*** oomichi is now known as oomichi_away02:57
*** oomichi_away has quit IRC03:02
*** otter768 has joined #openstack-infra03:02
*** baohua has quit IRC03:03
*** rushiagr is now known as rushiagr_away03:03
*** rushiagr_away is now known as rushiagr03:05
*** gokrokve has joined #openstack-infra03:09
*** bdpayne has joined #openstack-infra03:13
*** craigbr has joined #openstack-infra03:13
*** otter768 has quit IRC03:13
*** gokrokve has quit IRC03:14
*** rushiagr is now known as rushiagr_away03:16
*** dimtruck is now known as zz_dimtruck03:21
*** tomoe_ has joined #openstack-infra03:23
*** sressot has quit IRC03:24
*** imcsk8 has quit IRC03:25
*** imcsk8 has joined #openstack-infra03:25
*** stevemar has quit IRC03:28
openstackgerritJoshua Harlow proposed a change to openstack-dev/pbr: Allow for inclusion of authors and changelog when building docs
harlowja_at_homelifeless, mordred let me know if ^ is ok with u03:29
harlowja_at_homenothing to complicated03:31
*** bdpayne has quit IRC03:33
*** hdd has joined #openstack-infra03:44
*** stevemar has joined #openstack-infra03:44
openstackgerritJulie Pichon proposed a change to openstack-infra/config: Add integration tests to the Horizon gate
*** harlowja_at_home has quit IRC03:54
*** erlon has quit IRC04:00
*** _nadya_ has quit IRC04:01
*** gokrokve has joined #openstack-infra04:09
*** garyh has quit IRC04:09
*** craigbr has quit IRC04:10
*** rushiagr_away is now known as rushiagr04:11
*** craigbr has joined #openstack-infra04:13
*** gokrokve has quit IRC04:14
*** torandu has joined #openstack-infra04:17
*** craigbr has quit IRC04:18
jogoanteaya: ping04:26
jogoanteaya: how do you currently track 3rd party CI  and if they are  running?04:27
*** koolhead17 has joined #openstack-infra04:28
*** openstack_ has quit IRC04:30
*** baohua has joined #openstack-infra04:40
*** yfried has quit IRC04:46
*** hdd has quit IRC04:48
*** arnaud has joined #openstack-infra04:48
*** baohua has quit IRC04:49
*** baohua has joined #openstack-infra04:49
*** otherwiseguy has joined #openstack-infra04:51
*** cipcosma has joined #openstack-infra05:05
*** koolhead17 has quit IRC05:09
*** frank_zhou has joined #openstack-infra05:13
*** QuinnyPig has quit IRC05:16
*** QuinnyPig has joined #openstack-infra05:16
*** otherwiseguy has quit IRC05:19
*** QuinnyPig has quit IRC05:21
*** QuinnyPig has joined #openstack-infra05:22
*** tsg has quit IRC05:27
*** arnaud has quit IRC05:32
*** amotoki has joined #openstack-infra05:34
*** ppai has joined #openstack-infra05:37
*** flaper87|afk is now known as flaper8705:39
*** isaacb has joined #openstack-infra05:43
openstackgerritSumit Naiksatam proposed a change to openstack-infra/config: Create Group Based Policy Stackforge project
openstackgerritPrashanth Pai proposed a change to openstack-infra/config: Add swiftonfile project to stackforge
*** isaacb has quit IRC05:52
*** loki184 has joined #openstack-infra05:56
*** nosnos has quit IRC05:57
*** nosnos_ has joined #openstack-infra06:00
*** imeerovi has quit IRC06:01
*** SumitNaiksatam has quit IRC06:02
*** SumitNaiksatam has joined #openstack-infra06:03
*** koolhead17 has joined #openstack-infra06:03
*** isaacb has joined #openstack-infra06:04
*** arnaud has joined #openstack-infra06:07
*** koolhead17 has quit IRC06:08
*** gokrokve has joined #openstack-infra06:09
*** gokrokve has quit IRC06:14
*** isaacb has quit IRC06:16
openstackgerritA change was merged to stackforge/dox: Logging improvements
*** JadnG has joined #openstack-infra06:26
openstackgerritSumit Naiksatam proposed a change to openstack-infra/config: Create Group Based Policy Stackforge project
SumitNaiksatamSergeyLukjanov: hi, thanks for your review, i have uploaded a new patch set:
*** isaacb has joined #openstack-infra06:30
*** yfried has joined #openstack-infra06:30
*** baohua has quit IRC06:36
yfriedBobBall_AWOL: looking for help with citrix gate06:36
*** loki184 has quit IRC06:39
*** HeOS has quit IRC06:45
*** pkoniszewski has joined #openstack-infra06:50
*** koolhead17 has joined #openstack-infra06:52
*** mpaolino has joined #openstack-infra06:59
*** che-arne has quit IRC06:59
*** ociuhandu has quit IRC07:02
*** mpaolino has quit IRC07:04
*** jcoufal has joined #openstack-infra07:05
*** yolanda has joined #openstack-infra07:09
*** stevemar has quit IRC07:13
*** jpich has quit IRC07:14
*** arxcruz has joined #openstack-infra07:14
*** lttrl has joined #openstack-infra07:15
*** Longgeek has joined #openstack-infra07:16
openstackgerritFlavio Percoco proposed a change to stackforge/dox: Use `:20` tag in f20's DockerFile
openstackgerritFlavio Percoco proposed a change to stackforge/dox: Add support for multiple images
openstackgerritFlavio Percoco proposed a change to stackforge/dox: Add support for multiple images
*** afazekas_ has joined #openstack-infra07:22
*** arnaud has quit IRC07:22
*** ihrachyshka has joined #openstack-infra07:22
*** e0ne has joined #openstack-infra07:22
*** jcoufal has quit IRC07:23
*** e0ne has quit IRC07:24
*** zul has quit IRC07:24
*** jcoufal has joined #openstack-infra07:25
*** arxcruz has quit IRC07:25
*** luqas has joined #openstack-infra07:26
*** yamamoto has quit IRC07:27
*** arxcruz has joined #openstack-infra07:29
*** ihrachyshka_ has joined #openstack-infra07:29
*** jistr has joined #openstack-infra07:29
*** ihrachyshka has quit IRC07:32
*** Longgeek has quit IRC07:33
*** ihrachyshka_ has quit IRC07:36
*** ihrachyshka_ has joined #openstack-infra07:36
*** jgallard has joined #openstack-infra07:37
*** MaxV has joined #openstack-infra07:38
*** MaxV has quit IRC07:40
*** pblaho has joined #openstack-infra07:41
*** pblaho has joined #openstack-infra07:41
*** Guest42286 has joined #openstack-infra07:45
ekarlso-phschwartz: ping when you're around if u will today :)07:46
openstackgerrityolanda.robla proposed a change to openstack-infra/jenkins-job-builder: Enable hipchat section into sample file
*** mpaolino has joined #openstack-infra07:50
*** HeOS has joined #openstack-infra07:55
*** jlibosva has joined #openstack-infra07:56
*** Steap has quit IRC07:56
*** shardy_afk is now known as shardy07:56
*** jlibosva has quit IRC07:58
*** andreykurilin_ has joined #openstack-infra07:58
*** berendt has joined #openstack-infra07:58
*** jlibosva has joined #openstack-infra08:00
*** Daviey has quit IRC08:01
*** marun has joined #openstack-infra08:02
*** arxcruz has quit IRC08:04
*** arxcruz has joined #openstack-infra08:07
*** Longgeek has joined #openstack-infra08:07
*** derekh has joined #openstack-infra08:07
*** gokrokve has joined #openstack-infra08:09
*** yamamoto has joined #openstack-infra08:10
*** gokrokve has quit IRC08:14
*** tomoe_ has quit IRC08:16
*** koolhead17 has quit IRC08:16
*** tomoe_ has joined #openstack-infra08:16
*** zz_johnthetubagu is now known as johnthetubaguy08:18
*** koolhead17 has joined #openstack-infra08:18
*** Steap has joined #openstack-infra08:18
*** ZZelle has quit IRC08:19
*** ZZelle has joined #openstack-infra08:20
*** che-arne has joined #openstack-infra08:20
*** e0ne has joined #openstack-infra08:20
*** Daviey has joined #openstack-infra08:21
*** _nadya_ has joined #openstack-infra08:21
*** jlibosva has quit IRC08:21
*** yamamoto has quit IRC08:21
*** jlibosva has joined #openstack-infra08:22
*** koolhead_ has joined #openstack-infra08:26
*** amuller has joined #openstack-infra08:29
*** koolhead17 has quit IRC08:29
*** andreaf_ is now known as andreaf08:29
*** bhuvan has quit IRC08:31
openstackgerritFlavio Percoco proposed a change to stackforge/dox: Add support for multiple images
*** marun has quit IRC08:34
*** KanagarajM has joined #openstack-infra08:43
*** loki184 has joined #openstack-infra08:44
*** frank_zhou has quit IRC08:49
*** luqas has quit IRC08:51
*** ildikov has joined #openstack-infra08:52
*** koolhead_ has quit IRC08:52
*** koolhead17 has joined #openstack-infra08:53
yfriedhi, can anyone help with citrix bot? it's failing to run on my patch08:53
*** yamamoto has joined #openstack-infra09:02
*** jp_at_hp has joined #openstack-infra09:03
*** yamamoto has quit IRC09:06
SergeyLukjanovjeblair, fungi, clarkb, if renaming will be this sept 13 - I'll be unable to drive it - it's my wedding day :)09:08
*** gokrokve has joined #openstack-infra09:09
*** dguerri`afk is now known as dguerri09:13
*** andreykurilin_ has quit IRC09:13
*** tomoe_ has quit IRC09:13
*** tomoe_ has joined #openstack-infra09:13
*** gokrokve has quit IRC09:13
*** tomoe__ has joined #openstack-infra09:14
*** tomoe_ has quit IRC09:18
*** Guest42286 has quit IRC09:20
*** yamamoto has joined #openstack-infra09:21
*** lttrl has quit IRC09:23
*** zul has joined #openstack-infra09:24
*** Hal has joined #openstack-infra09:25
*** Hal is now known as Guest1584909:25
*** luqas has joined #openstack-infra09:26
*** Guest15849 has quit IRC09:31
*** Guest15849 has joined #openstack-infra09:37
*** jcoufal has quit IRC09:42
*** isviridov_away is now known as isviridov09:45
*** Guest15849 has quit IRC09:46
*** rlandy has joined #openstack-infra09:47
*** yamamoto has quit IRC09:49
*** dtantsur is now known as dtantsur|lunch09:53
*** yamamoto has joined #openstack-infra09:55
openstackgerritChristian Berendt proposed a change to openstack-dev/cookiecutter: Remove unneeded definitions of Python Source Code Encoding
openstackgerritChristian Berendt proposed a change to openstack-dev/cookiecutter: Enable hacking check H104
*** pblaho is now known as pblaho|afk09:58
*** pblaho|afk is now known as pblaho09:59
*** yamamoto has quit IRC09:59
*** e0ne has quit IRC10:00
*** lttrl has joined #openstack-infra10:02
*** luqas has quit IRC10:04
kashyapI wonder if infra has plans for CLI-based interface for paste.o.o?  Fedora infra has `fpaste` which one can use on remote headless servers to pastebin snippets from CLI.10:08
*** yamahata has quit IRC10:09
*** gokrokve has joined #openstack-infra10:09
Kiallkashyap: `pastebinit` can be pointed at it...10:10
*** pcm_ has joined #openstack-infra10:10
KiallI'm sure other generic CLI's can too10:10
Kiall`echo "Test" | pastebinit -b`  ...
*** e0ne has joined #openstack-infra10:12
openstackgerritBoris Pavlovic proposed a change to openstack-infra/config: Set custom index.html to root of output of rally-dsvm jobs
kashyapKiall, Ah, thanks for the pointer. /me looks up in Fedora repos for what provides that tool.10:14
*** gokrokve has quit IRC10:14
*** jgallard has quit IRC10:16
*** koolhead17 has quit IRC10:16
*** MaxV has joined #openstack-infra10:20
*** yamamoto has joined #openstack-infra10:20
*** pblaho is now known as pblaho|afk10:21
*** yamamoto_ has joined #openstack-infra10:22
*** yamamot__ has joined #openstack-infra10:22
*** yamamoto_ has quit IRC10:22
*** yamamoto has quit IRC10:23
*** tomoe__ has quit IRC10:28
*** pelix has joined #openstack-infra10:30
*** jcoufal has joined #openstack-infra10:47
*** andreykurilin has joined #openstack-infra10:47
*** dizquierdo has joined #openstack-infra10:49
*** isviridov is now known as isviridov_away10:51
*** koolhead17 has joined #openstack-infra10:52
*** mpaolino has quit IRC10:54
*** daya_k has quit IRC10:59
*** dims has joined #openstack-infra11:00
*** pkoniszewski has quit IRC11:04
*** Guest15849 has joined #openstack-infra11:04
*** pkoniszewski has joined #openstack-infra11:04
*** yamamot__ has quit IRC11:06
*** ppai has quit IRC11:07
*** luqas has joined #openstack-infra11:08
*** gokrokve has joined #openstack-infra11:09
*** gokrokve has quit IRC11:14
*** Guest15849 has quit IRC11:15
*** yamamoto has joined #openstack-infra11:15
*** dims has quit IRC11:16
*** dims has joined #openstack-infra11:16
*** dims_ has joined #openstack-infra11:17
*** dmsimard_away is now known as dmsimard11:17
*** loki184 has quit IRC11:17
*** pblaho|afk is now known as pblaho11:19
*** dims has quit IRC11:21
*** e0ne has quit IRC11:22
*** ppai has joined #openstack-infra11:23
*** mwagner_lap has quit IRC11:23
*** rfolco has joined #openstack-infra11:24
*** yamamoto has quit IRC11:26
*** baoli has joined #openstack-infra11:27
*** baoli has joined #openstack-infra11:27
*** baoli has quit IRC11:28
*** baoli has joined #openstack-infra11:29
*** changbl has quit IRC11:34
*** luqas has quit IRC11:34
boris-42jhesketh hi11:38
*** mbacchi has joined #openstack-infra11:42
*** tomoe_ has joined #openstack-infra11:44
*** mmedvede has joined #openstack-infra11:45
*** daya_k has joined #openstack-infra11:46
*** ppai has quit IRC11:47
*** yaguang has joined #openstack-infra11:48
*** mmaglana has quit IRC11:50
*** mmaglana has joined #openstack-infra11:51
*** unicell has quit IRC11:55
*** mmaglana has quit IRC11:55
*** yamamoto has joined #openstack-infra11:56
*** MaxV has quit IRC11:57
*** MaxV has joined #openstack-infra11:58
*** yamamoto has quit IRC12:01
*** mwagner_lap has joined #openstack-infra12:02
*** MaxV has quit IRC12:03
*** thelexx is now known as lxsli12:03
*** kgiusti has joined #openstack-infra12:03
*** yfried has quit IRC12:06
*** pkoniszewski has quit IRC12:07
*** yamamoto has joined #openstack-infra12:08
openstackgerritFlavio Percoco proposed a change to stackforge/dox: Add support for multiple images
*** gokrokve has joined #openstack-infra12:09
*** nosnos_ has quit IRC12:13
*** gokrokve has quit IRC12:14
*** nosnos has joined #openstack-infra12:14
*** zns has joined #openstack-infra12:15
*** KanagarajM has quit IRC12:15
sdagueflaper87: on the dox hacking, how are you guys imagining multiple targets?12:15
*** dims_ has quit IRC12:16
*** dims has joined #openstack-infra12:16
flaper87sdague: I just threw this idea out there:
flaper87which is pretty much how tox does it12:16
*** weshay has joined #openstack-infra12:17
flaper87One problem is that `prep` is 1 for all images. Ideally, developers shouldn't need `prep` at all except for installing the test/project requirements12:17
flaper87This means we would probably have pre-built images per-project12:18
*** nosnos has quit IRC12:18
*** aysyd has joined #openstack-infra12:19
flaper87I still think we'll need a way to specify per-image `prep` sections but we'll take care of that later on12:19
sdagueflaper87: oh, so I was actually thinking the other direction12:19
sdaguelike zookeeper tests vs. mysql tests12:19
*** dkranz has joined #openstack-infra12:20
*** zns has quit IRC12:20
*** mkoderer_ is now known as mkoderer12:20
*** dims_ has joined #openstack-infra12:20
*** dims has quit IRC12:21
*** zns has joined #openstack-infra12:21
*** pblaho is now known as pblaho|afk12:21
*** adalbas has joined #openstack-infra12:24
*** mmaglana has joined #openstack-infra12:25
*** luqas has joined #openstack-infra12:27
openstackgerritMartin Mágr proposed a change to openstack-infra/config: Packstack changes
*** Mithrandir has quit IRC12:28
*** dtantsur|lunch is now known as dtantsur12:29
*** mmaglana has quit IRC12:30
*** ildikov_ has joined #openstack-infra12:31
*** ildikov has quit IRC12:34
*** dprince has joined #openstack-infra12:34
*** Mithrandir has joined #openstack-infra12:34
fungijogo: i'm missing context... were you talking about in regard to adding co-authored-by headers into the atc list generation? if so, sure that's the route i'd go. i can see about adding it unless someone's already working on a patch12:36
sdaguefungi: I really thought that already was happening12:38
*** gargola has joined #openstack-infra12:39
*** otherwiseguy has joined #openstack-infra12:40
*** bswartz has quit IRC12:40
fungisdague: i'll double-check teh script, but i don't believe that has ever been the case12:44
fungiseems like a reasonable addition though, i agree12:45
*** yamahata has joined #openstack-infra12:45
*** yfried has joined #openstack-infra12:45
*** marcusvrn has joined #openstack-infra12:46
*** pblaho|afk is now known as pblaho12:47
fungisdague: yeah, i don't see any commit message analysis happening at which is the only way it would have been happening (gerrit doesn't store any metadata for arbitrary commit message header lines)12:50
sdaguefungi: yep, fair12:50
*** otherwiseguy has quit IRC12:50
*** wenlock has joined #openstack-infra12:50
*** radez_g0n3 is now known as radez12:51
*** mmaglana has joined #openstack-infra12:52
fungisdague: in fact, it looks like it's based on the change owner account, not necessarily the git author or committer12:52
fungiso could probably use some improvement in that regard as well12:52
fungithough we also don't require that the git author matches to an account in gerrit, just the committer12:53
*** pkoniszewski has joined #openstack-infra12:53
*** dizquierdo has quit IRC12:53
fungiso change owner might be a more accurate proxy for the commit author, given that subsequent contributors may make adjustments to the original change (leaving them as the eventual committer on the approved patchset)12:54
*** isaacb has quit IRC12:55
*** zns has quit IRC12:55
fungiit's probably worth clarifying at a policy level what an "active technical contributor" is (keeping in mind what sorts of information we have to make that determination)12:56
*** mmaglana has quit IRC12:56
*** MaxV has joined #openstack-infra12:58
fungiright now it involves being a foundation member, which implies having an account in the foundation member database, which is only enforced based on the existence of a gerrit account (via cla linkage), which then means a change owner or committer (not necessarily author or co-author since those don't have to link back to accounts)12:58
*** lttrl has quit IRC12:59
*** lttrl has joined #openstack-infra12:59
fungithere's also the problem that committer is immutable in git but gerrit account contact info is mutable, so by the time we analyze the data in git some contributions may fall through the cracks because of drift in registered gerrit e-mail addresses13:00
*** zns has joined #openstack-infra13:01
*** pblaho is now known as pblaho|afk13:02
*** MaxV has quit IRC13:03
*** isviridov_away is now known as isviridov13:03
*** vhoward has joined #openstack-infra13:04
fungiin fact, given the level of potential inaccuracy as well as the low voter turn-out, we might ought to reconsider our current automatic conveyance of atc status and switch it to a voluntary opt-in process (if you want to vote, there's a web form where you can stuff in one or more commit shas which are analyzed for presence of author and co-author addresses matched against your foundation member13:04
*** lttrl has quit IRC13:04
fungithat solves the current problems with forcing contributors to be foundation members solely so that we can let some of them vote in technical elections13:05
dhellmannmordred: what's the story with sqlalchemy-migrate these days? we need for if that's possible13:05
*** bswartz has joined #openstack-infra13:06
*** Longgeek has quit IRC13:08
*** wenlock has quit IRC13:08
*** julim has joined #openstack-infra13:08
*** mpaolino has joined #openstack-infra13:08
*** julim has quit IRC13:08
*** julim has joined #openstack-infra13:09
*** e0ne has joined #openstack-infra13:09
*** otherwiseguy has joined #openstack-infra13:09
*** gokrokve has joined #openstack-infra13:09
*** dustins has joined #openstack-infra13:10
*** alaski_ is now known as alaski13:11
*** Longgeek has joined #openstack-infra13:11
*** pblaho|afk is now known as pblaho13:13
*** gokrokve has quit IRC13:13
fungimorning sdake!13:14
fungii find that my fingers always want to tab-complete the word "morning"13:14
*** zns has quit IRC13:17
*** zns has joined #openstack-infra13:19
*** sabeen has joined #openstack-infra13:20
*** MaxV has joined #openstack-infra13:20
*** mfer has joined #openstack-infra13:20
*** yolanda has quit IRC13:21
*** sandywalsh_ has joined #openstack-infra13:21
*** che-arne has quit IRC13:22
*** amuller_ has joined #openstack-infra13:23
*** jgallard has joined #openstack-infra13:23
*** vponomaryov has quit IRC13:24
*** amuller has quit IRC13:24
*** dkliban_afk is now known as dkliban13:24
*** amuller__ has joined #openstack-infra13:24
*** amuller__ is now known as amuller13:24
*** sandywalsh has quit IRC13:24
*** annegent_ has joined #openstack-infra13:24
*** bknudson has quit IRC13:25
*** MaxV has quit IRC13:26
*** mriedem has joined #openstack-infra13:27
*** wenlock has joined #openstack-infra13:27
*** amuller_ has quit IRC13:28
*** paulrad has joined #openstack-infra13:28
*** MaxV has joined #openstack-infra13:31
*** koolhead17 has quit IRC13:31
*** zns has quit IRC13:31
*** koolhead17 has joined #openstack-infra13:32
*** MaxV has quit IRC13:32
*** MaxV has joined #openstack-infra13:33
*** signed8bit has joined #openstack-infra13:35
*** mriedem1 has joined #openstack-infra13:35
*** MaxV_ has joined #openstack-infra13:36
*** mriedem has quit IRC13:36
*** kgiusti has quit IRC13:36
*** MaxV has quit IRC13:36
fungiprobably a sign that i'm not entirely awake until it's no longer morning13:36
*** nelsnelson has joined #openstack-infra13:37
*** koolhead17 has quit IRC13:37
*** kgiusti has joined #openstack-infra13:40
*** r-daneel has joined #openstack-infra13:45
*** Guest15849 has joined #openstack-infra13:45
*** bknudson has joined #openstack-infra13:46
*** yaguang has quit IRC13:46
*** miqui has joined #openstack-infra13:49
*** masayukig has quit IRC13:49
*** _nadya_ has quit IRC13:50
*** Guest15849 has quit IRC13:52
*** mmaglana has joined #openstack-infra13:52
*** doug-fish has joined #openstack-infra13:52
*** jaypipes has joined #openstack-infra13:55
*** masayukig has joined #openstack-infra13:55
openstackgerritA change was merged to openstack-infra/gitdm: Add committer details
openstackgerritA change was merged to openstack-infra/reviewstats: Make pdb debugging of openreviews possible.
openstackgerritA change was merged to openstack-infra/gitdm: Update James Li's affiliations
*** mmaglana has quit IRC13:56
*** homeless has joined #openstack-infra13:57
openstackgerritA change was merged to openstack-infra/reviewstats: Work toward Python 3.4 support and testing
*** vponomaryov has joined #openstack-infra13:58
*** salv-orlando has joined #openstack-infra13:58
*** paulrad has quit IRC13:58
*** paulrad has joined #openstack-infra13:58
*** gokrokve has joined #openstack-infra14:00
*** changbl has joined #openstack-infra14:01
*** markmcclain has joined #openstack-infra14:02
*** paulrad has quit IRC14:03
*** markmcclain has quit IRC14:03
*** otherwiseguy has quit IRC14:04
*** beekneemech is now known as bnemec14:04
*** pkoniszewski has quit IRC14:04
*** zns has joined #openstack-infra14:05
fungiwhoawaitwhat? the neutron server consumes the neutron client? that's the sort of ouroboros which can only lead to repeated compatibility problems14:05
*** markmcclain has joined #openstack-infra14:05
*** ihrachyshka_ has quit IRC14:06
*** baoli has quit IRC14:08
*** chuckC has quit IRC14:10
*** pblaho is now known as pblaho|afk14:10
*** Daviey has quit IRC14:10
*** baoli has joined #openstack-infra14:11
gilliard_+1 for "ouroboros" but I don't see it in that patch...14:11
openstackgerritDarragh Bailey proposed a change to openstack-infra/git-review: Enable color support based on tty and config
*** koolhead_ has joined #openstack-infra14:12
gilliard_oh now I do.14:12
*** zz_jgrimm is now known as jgrimm14:12
*** ilyashakhat has joined #openstack-infra14:12
*** Ajaeger has joined #openstack-infra14:13
*** Guest15849 has joined #openstack-infra14:13
*** pblaho|afk is now known as pblaho14:14
andreykurilinsdague, hi! do you have a bit free time to look at several my patches in requirements?:)14:15
fungigilliard_: yeah, a new neutronclient release broke havana neutron... not because it didn't work as a client for connecting to old neutron servers, but because neutron was itself using neutronclient internally14:15
*** esker has joined #openstack-infra14:15
gilliard_and havana neutron will use the latest-available client, not the havana one?14:16
fungiwell, there is no "havana" neutronclient14:16
fungithe clients aren't part of the integrated release, they just release linearly14:16
gilliard_oh, OK.14:17
gilliard_And the dependencies of a release aren't frozen.14:17
*** dkranz has quit IRC14:18
gilliard_I'm more used to the maven-style of dependency mgmt where everything is frozen all the time.14:18
fungigilliard_: right. at the moment we use the python clients as client interfaces and as internal communication libraries for the servers talking to each other, and expect them to be able to me comingled with client and communication library use cases on the same machine14:19
dhellmannfungi: apparently when oslo-messaging-ptl was renamed to oslo-messaging-release no members were added, can you put me in?,members14:19
*** tomoe_ has quit IRC14:20
fungidhellmann: that's... weird. renaming shouldn't ever remove members14:20
*** tomoe_ has joined #openstack-infra14:20
dhellmannfungi: I don't know what happened, then?14:20
fungidhellmann: i've set it to include the oslo-release group now14:20
fungidhellmann: did oslo.messaging at one point not have a separate ptl/release group and maybe recently grew one?14:21
*** _buzz has quit IRC14:21
dhellmannfungi: oh, perhaps that's it14:21
*** zz_dimtruck is now known as dimtruck14:21
openstackgerritA change was merged to stackforge/dox: Add `--fix-missing` to trusty's Dockerfile
openstackgerritA change was merged to stackforge/dox: Use `:20` tag in f20's DockerFile
fungiif so, the group starts out with no members and is automatically created when the acl change which introduces that new name is merged14:21
*** Guest15849 has quit IRC14:22
dhellmannyeah, that makes more sense14:22
*** tsg has joined #openstack-infra14:22
*** Daviey has joined #openstack-infra14:23
*** garyh has joined #openstack-infra14:24
*** atiwari has joined #openstack-infra14:25
boris-42fungi hi14:25
*** koolhead17 has joined #openstack-infra14:25
fungihi boris-42. have a question/request?14:25
boris-42fungi ya I have very simple patch that sets custom index.html (from our repo) on rally job14:26
*** eharney has joined #openstack-infra14:26
boris-42fungi it has +214:26
boris-42fungi could you please take a look
boris-42fungi just one more publisher added to job14:26
boris-42fungi *target*14:26
Ajaegerclarkb, fungi: Could you review , please?14:26
*** Guest15849 has joined #openstack-infra14:28
*** koolhead_ has quit IRC14:28
boris-42fungi thank you14:29
*** julim has quit IRC14:30
*** dkranz has joined #openstack-infra14:30
*** david-lyle has joined #openstack-infra14:30
*** koolhead_ has joined #openstack-infra14:31
dhellmannfungi: is there some sort of plan written down for these python 3.4 changes ( When is that cut-over going to happen, for example?14:31
dhellmannfungi: also, I'm not sure what to do about the stable branch versions of those changes14:31
*** _buzz has joined #openstack-infra14:32
fungidhellmann: i'm going to manually run the 3.4 equivalents of any currently voting python33 jobs on a trusty node to make sure they're sane, and as soon as that's done i'll propose the layout.yaml change to replace them all14:32
*** tonytan4ever has joined #openstack-infra14:33
fungidhellmann: as for the stable branches with python33 jobs currently voting, we either need to make sure they work with 3.4 or remove them from stable branches14:33
*** yamahata has quit IRC14:33
*** superdan is now known as dansmith14:34
fungidhellmann: i'm hoping this week, now that feature freeze is done with14:34
*** koolhea__ has joined #openstack-infra14:34
*** koolhead17 has quit IRC14:34
*** cdent has joined #openstack-infra14:34
fungidhellmann: it ends up needing to be a quick surgical strike, otherwise the situation might shift and/or the layout change will rapidly conflict with the state of the config repo14:35
*** paulrad has joined #openstack-infra14:36
fungidhellmann: similarly, i need to test any currently voting pypy jobs on trusty and then move them to run there14:36
fungidhellmann: once those two things happen, we should be able to dump our special py3k-precise nodes and get much more consistent response time on both py3k and pypy jobs14:37
*** koolhead_ has quit IRC14:37
*** esker has quit IRC14:37
*** zns has quit IRC14:38
openstackgerritA change was merged to openstack-infra/config: Set custom index.html to root of output of rally-dsvm jobs
dhellmannfungi: ok, so land the changes in master quickly and then sort out stable by deciding whether to test with 3.4 or drop the test?14:38
*** Guest15849 has quit IRC14:38
*** zns has joined #openstack-infra14:38
*** craigbr_ has joined #openstack-infra14:39
fungidhellmann: yeah, for most of the proposed master changes, those are just a convenience for devs to have the py34 env in their default tox run anywhere we previously also have py3314:40
fungihdfor a small percentage, i duplicated any special handling of py33 envs for py34 in tox.ini14:40
*** isviridov is now known as isviridov_away14:40
*** paulrad has quit IRC14:40
*** craigbr_ has quit IRC14:40
fungidhellmann: but for the most part that can happen asynchronously to the job changes14:40
Kiallmordred: about? Got a sec re the openstack/governance change for openstack/designate-dashboard?14:40
* Kiall has no clue what TZ mordred is in these days..14:41
openstackgerritBoris Pavlovic proposed a change to openstack/requirements: Rally needs bumping python-novaclient
boris-42dhellmann hi there14:41
boris-42dhellmann ^ could you take a look at this14:41
*** craigbr_ has joined #openstack-infra14:41
russellbjeblair: gertty may be my new favoritest thing ever.14:43
*** [HeOS] has joined #openstack-infra14:44
jeblairrussellb: aww yay!14:44
*** julim has joined #openstack-infra14:44
*** julim has quit IRC14:44
openstackgerritA change was merged to stackforge/gertty: Fix crash on prev/next change
*** julim has joined #openstack-infra14:45
openstackgerritA change was merged to stackforge/gertty: Fix another crash on prev/next change
*** julim has quit IRC14:45
*** HeOS has quit IRC14:46
*** Guest15849 has joined #openstack-infra14:46
*** MaxV_ has quit IRC14:46
*** MaxV has joined #openstack-infra14:46
*** julim has joined #openstack-infra14:46
fungirussellb: it is crazy useful14:46
*** julim has quit IRC14:46
dhellmannis anyone else having issues fetching from git? are the servers just really loaded or is work going on?14:47
dhellmannI've had a couple of times this morning where "git pull" just hung for a long time14:47
fungirussellb: i hate web browsers. i also have some machines i use on a regular basis where gerrit's webui is essentially unusable14:47
mtreinishdhellmann: yeah, I just noticed that myself14:47
*** julim has joined #openstack-infra14:47
fungidhellmann: what is your git remote url?14:47
russellbyeah, feels more efficient than the web UI so far, and the offline usage <3<3<314:47
*** julim has quit IRC14:48
dhellmannfungi: git://
fungidhellmann: just so i can narrow down investigation to specific propocols et cetera14:48
fungidhellmann: hrm, that should be lightweight... i'll look at cacti14:48
*** julim has joined #openstack-infra14:48
jeblairthere's a small gap in the cacti data there, which may mean either the load on git.o.o was so high that cacti could not poll it, or there was a network interruption near git.o.o14:49
*** Ajaeger has quit IRC14:49
*** julim has quit IRC14:49
jeblairthe second is more likely (because git.o.o is just a load balancer)14:49
*** julim has joined #openstack-infra14:50
*** julim has quit IRC14:50
*** yamamoto has quit IRC14:50
*** julim has joined #openstack-infra14:51
*** julim has quit IRC14:51
openstackgerritJulien Danjou proposed a change to openstack-infra/config: Use python3-jobs template where possible
*** MaxV has quit IRC14:51
*** julim has joined #openstack-infra14:52
*** lcheng_ has joined #openstack-infra14:52
*** julim has quit IRC14:52
fungijd__: thanks for ^ that! it's going to make my change to switch to python34 a lot simpler ;)14:52
*** julim has joined #openstack-infra14:53
anteayajogo: hi, I don't track it, if folks need to track that they do so and let me know if there is an issue, the issue I follow up is non-compliance with requirements which can get them disabled14:53
*** julim has quit IRC14:53
*** julim has joined #openstack-infra14:54
*** datsun180b has joined #openstack-infra14:54
*** julim has quit IRC14:54
anteayajogo: this is mikal's ci dashboard tool which I sucked into stackforge so people could expand it, if you want to add your last_commit tool that would be awesome:
*** esker has joined #openstack-infra14:54
*** koolhead17 has joined #openstack-infra14:55
jd__fungi: :-)14:55
*** julim has joined #openstack-infra14:55
*** julim has quit IRC14:55
*** esker has quit IRC14:55
*** jergerber has joined #openstack-infra14:55
*** esker has joined #openstack-infra14:55
anteayamorning should be tab complete, we just need a fake user in this channel with morning as a username14:55
dhellmannmordred, lifeless : are we ready for a pbr release today?14:55
*** esker has quit IRC14:55
fungijeblair: dhellmann: the git.o.o proxy doesn't currently seem to be under any load at all, so i agree it was probably a network blip14:55
*** esker has joined #openstack-infra14:56
*** julim has joined #openstack-infra14:56
fungii don't see any similar gaps on the individual servers in the farm except one many hours ago on git0314:56
*** julim has quit IRC14:56
anteayafungi: I'm open to the conversation of making participation in elections opt-in, if you want to launch the discussion, it won't affect this round of elections but it might affect the next round14:56
*** esker has quit IRC14:56
*** changbl has quit IRC14:56
*** jcoufal has quit IRC14:56
openstackgerritA change was merged to stackforge/gertty: Quote identifiers in migrations
*** julim has joined #openstack-infra14:57
fungianteaya: yeah, it's a broader discussion i think, from both dev and foundation sides14:57
*** esker has joined #openstack-infra14:57
*** julim has quit IRC14:57
*** changbl has joined #openstack-infra14:57
jeblairanteaya: context?14:58
openstackgerritA change was merged to stackforge/gertty: Add additional help text for openstack user/pass
dhellmannfungi: ok14:58
*** julim has joined #openstack-infra14:58
*** julim has quit IRC14:58
*** koolhea__ has quit IRC14:58
fungijeblair: scrollback, 12:36 utc14:58
*** chuckC has joined #openstack-infra14:59
*** julim has joined #openstack-infra14:59
*** maxbit has joined #openstack-infra14:59
*** julim has quit IRC14:59
jeblairfungi: yes, we sholud do the things you say.  :)14:59
*** lcheng_ has quit IRC15:00
*** julim has joined #openstack-infra15:00
jeblairwe should decouple membership from contribution since apparently they are not actually required and it is a barrier15:00
anteayayes, with whatever mechinism can facilitate the decoupling15:00
*** marcusvrn has quit IRC15:01
*** lcheng_ has joined #openstack-infra15:02
*** tomoe__ has joined #openstack-infra15:03
*** koolhead_ has joined #openstack-infra15:04
*** koolhead17 has quit IRC15:04
*** koolhead17 has joined #openstack-infra15:05
*** lcheng__ has joined #openstack-infra15:05
*** yamahata has joined #openstack-infra15:05
craigbr_Can someone from the core team look at All the chef builds are broken because of this issue15:05
anteayaanyone with a free hand able to re-enable 10119 varmourci vArmour CI openstack-ci-test@varmour.com15:05
*** paulrad has joined #openstack-infra15:06
*** tomoe_ has quit IRC15:07
*** lcheng_ has quit IRC15:07
fungianteaya: done15:07
*** koolhead_ has quit IRC15:08
*** enikanorov_ has joined #openstack-infra15:09
*** enikanorov_ has quit IRC15:09
*** unicell has joined #openstack-infra15:09
*** enikanorov_ has joined #openstack-infra15:10
*** paulrad has quit IRC15:10
*** daya_k has quit IRC15:11
*** jcoufal has joined #openstack-infra15:11
anteayafungi: thank you15:11
fungihrm, load average on our git servers looks like it's spiking (as much as 10 over a 5-minute sample...)
fungithe load balancer may have been unresponsive to snmp queries due to a hard rate limit? was up over 300mbps when the gap appeared...
*** salv-orlando has quit IRC15:14
*** MaxV has joined #openstack-infra15:15
*** stevemar has joined #openstack-infra15:15
jeblairfungi: the pricing page claims "1,600Mb/s" for that class of server15:15
fungiwell, we seem to spike to that same level consistently, which is slightly odd...
*** afazekas_ has quit IRC15:17
*** MaxV has quit IRC15:20
mordredcraigbr_: done15:20
*** hemna__ is now known as hemna15:20
craigbr_mordred: Thank you15:20
mordredcraigbr_: also, fungi's information that ruby1.9.1 --version returns 1.9.3 is one of my favorite pieces of new information15:21
fungimordred: oh, that's been a source of confusion in debian for years15:21
fungimordred: the backstory on why is amusing in and of itself15:22
fungimordred: or you can just tl;dr it as ERUBY and move on15:22
*** MaxV has joined #openstack-infra15:23
mordredfungi: oh, I was going to assume that the backstory at some point illustrated something where teh distro structure got in the way of helping15:23
*** andreykurilin has quit IRC15:23
mordredsuch as "we have a strict policy against chaning X in a Y type of release, so we're going to do something very strage which will confuse people for years rather than break the letter of that policy"15:23
*** isviridov_away is now known as isviridov15:23
* mordred is still caffinating15:24
*** _buzz has quit IRC15:24
*** koolhead17 has quit IRC15:24
*** markmcclain has quit IRC15:26
fungimordred: well, it stemmed from backward compatibility in ruby interpreter releases and software being written for different interpreter versions so needing to be able to install them side by side. apparently things written for 1.8 could run on the 1.9 release so the interpreter name remained ruby1.8, but the 1.9.1 release broke backward compatibility15:26
fungibut software written for 1.9.1 will run fine on 1.9.315:27
fungireally 1.9.1 *should* have been released as 1.10 or 2.0 or something15:27
*** MaxV_ has joined #openstack-infra15:28
*** MaxV has quit IRC15:28
*** Sincler has joined #openstack-infra15:29
openstackgerritA change was merged to openstack-infra/config: Fix chef gates, allow sudo for gem install and use ruby 1.9.1
*** gokrokve_ has joined #openstack-infra15:31
mordredfungi: wait - so you're telling me that debian shipped the first release of 1.9 as ruby1.8?15:32
mordredand then when there was a release of 1.9 that did not work with 1.8 code, rather than just making a 1.9 package like they should have in the first place, they made a package called 1.9.1?15:32
mordredfungi: and that anybody thought that either of those activities made sense?15:33
mordredfungi: (ignoring briefly the other thing, where I agree with you, that 1.9.1 upstream should have been called 1.1015:33
fungimordred: well, apparently there was lots of software written to invoke the interpreter as ruby1.8 and when the initial 1.9 release worked with it they didn't want to go through another huge ruby transition unnecessarily15:34
*** gokrokve has quit IRC15:34
fungiapparently the transition from 1.7 to 1.8 was a fairly massive undertaking15:35
jrollmordred: a ruby dev would tell you 'use rvm'15:35
jrollmordred: (because it's all a clusterfuck)15:35
*** paulrad has joined #openstack-infra15:36
fungimordred: the "good" news is that the ruby1.9.1 package has been removed from debian/testing so shouldn't be a source of confusion once nobody's using ubuntu trusty any longer ;)15:37
*** koolhead17 has joined #openstack-infra15:37
anteayaruby devs pointed tell newbies to stay away from ruby debian packaging15:38
*** bhuvan has joined #openstack-infra15:38
anteayait is one of the things that motivated the creation of rvm15:38
*** mmaglana has joined #openstack-infra15:38
mordreddistro packaging vs. language packaging is, in general, broken across the board15:38
anteayano argument here15:38
mordredsee also "don't use javascript tools and libraries like npm or bower because the distros don't really know how to package them yet"15:39
mordredand "go sucks because it statically compiles everything which is contrary to how the distros want to do things, so don't use it"15:40
fungioh, actually the ruby1.9.1 removal transition is still not complete in unstable, so it might end up in the debian/jessie release after all
*** mikedillion has joined #openstack-infra15:40
mordredfungi: spectacular15:40
Shrewsmordred: fungi: did you guys know ansible has a durham office? heh, neato15:40
mordredShrews: not just a durham office, they are, in fact, based completely in durham15:41
mordredShrews: at the old american tobacco campus15:41
fungioh, actually it was removed from testing in may, but is still in unstable because there are packages depending on it which are only in unstable15:41
Shrewsmordred: even more neato15:41
mordredShrews: the founder is a former redhat guy.15:41
*** ildikov_ has quit IRC15:41
fungiShrews: yeah, ansible dehaan was a bunch of the other rpath linux/conary folks15:42
Shrewssuch techhub15:42
fungier, was dehaan and a bunch of15:42
*** amotoki_ has joined #openstack-infra15:42
fungianyway, he spun it off when sas bought rpath, since he had been working on ansible in his own time and it wasn't owned by rpath15:43
*** pfallenop has quit IRC15:43
*** paulrad has quit IRC15:43
*** yamahata has quit IRC15:43
*** pfallenop has joined #openstack-infra15:43
*** mwagner_lap has quit IRC15:46
*** john-davidge has joined #openstack-infra15:47
*** salv-orlando has joined #openstack-infra15:48
fungithough from what i heard through the grapevine, he moved to the bay area to take a job with cisco after the buy-out, and founded ansibleworks out there as a result. only more recently did they open the office at american tobacco15:51
*** yamamoto has joined #openstack-infra15:51
*** esker has quit IRC15:53
clarkb119094 shouldnt be necessary on trusty. are we running the chef jobs on precise now?15:53
*** jgallard has quit IRC15:53
*** esker has joined #openstack-infra15:53
jeblairfungi, clarkb, mordred: does not look like per-region mirrors have reduced the incidence of 1334550 or 1270710 or 132681315:54
jeblairfungi, clarkb, mordred: however, 'successful' matches of some of those have increased, which is weird and warrants investigation15:54
jeblairjogo: ^15:54
clarkbjeblair: correct. hpcloud networking seems to continue its failures there15:55
*** yamamoto has quit IRC15:55
*** MaxV_ has quit IRC15:56
*** MaxV has joined #openstack-infra15:56
john-davidgeIs this bug with Citrix XenServer CI currently known?15:56
john-davidge[ERROR] /opt/stack/new/devstack/functions-common:565 Cloning not allowed in this configuration15:56
john-davidgejust started seeing it today15:56
clarkbjeblair: I think that confirms the trouble is near the nodes and not on the greater internets15:57
*** bdpayne has joined #openstack-infra15:57
*** esker has quit IRC15:58
jeblairthat's the first example i found15:58
*** stevemar has quit IRC15:58
jeblair2014-09-08 15:44:05.983 | error: [Errno 104] Connection reset by peer15:58
*** stevemar has joined #openstack-infra15:59
*** gyee has joined #openstack-infra15:59
*** markmcclain has joined #openstack-infra15:59
anteayajohn-davidge: might I get a link to a patch where this is happening?16:00
*** arxcruz has quit IRC16:00
fungijeblair: yeah, we were looking at those over the weekend. it does seem like maybe the one router per controller node suggestion might be the next thing to try16:00
*** MaxV has quit IRC16:01
mordredjeblair, fungi: we can also try the suggestion of getting the 9 new routers up and going ... since that is a suggestion we've gotten from them already16:01
fungimordred: yeah, that's what i meant16:02
*** MaxV has joined #openstack-infra16:02
anteayajohn-davidge: which log?16:02
jeblairmordred: we will have 72 images in hpcloud then :(16:02
mordredbut I'm going to send a follow up email to that guy and include you guys and let him know the problem we're currently seeing16:02
mordredjeblair: oh, that's a good point16:02
john-davidgeanteaya: devstack -
jeblairmordred: i had hoped to try that _after_ we switched to using dib16:02
mordredjeblair: well, I'll hack on the dib patches this week16:02
jeblairmordred: but if we think it's the only reliable way to get networking in hpcloud, then i guess we'll have to do it first16:02
*** zns has quit IRC16:03
clarkbcan we move the existing 5 first?16:03
mordredlet me ping rick and see what he thinks about intra-hp networking issues16:03
mordredclarkb: what do you mean?16:03
*** annegent_ has quit IRC16:03
fungimordred: though it sounded like the main benefit was going to be from pinning the routers to separate controllers, moreso than increasing the number of them16:03
clarkbwe have 5 routers today. can we make them optimally placed?16:03
fungiright, what clarkb said16:03
anteayaBobBall_AWOL: your system is broken and is about to be disabled16:03
*** mikedillion has quit IRC16:04
*** jlibosva has quit IRC16:04
*** Guest15849 has quit IRC16:04
clarkbthat is simple enough todo without downtime if we have a communication method16:04
*** arnaud has joined #openstack-infra16:04
jeblairclarkb, fungi: ^ that one failed several times!16:04
*** zns has joined #openstack-infra16:05
anteayawe need 10385 citrix_xenserver_ci Citrix XenServer CI disabled please16:05
* anteaya composes email to third-party-announce16:05
jeblairclarkb, fungi:
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Adding a final setup step to install_puppet
*** tomoe__ has quit IRC16:05
jeblairclarkb, fungi: that one succeeded16:05
john-davidgeanteaya: thanks16:06
*** pcrews has joined #openstack-infra16:06
jeblairclarkb, fungi: apparently devstack doesn't (always?) bail on pip install failures16:06
fungijeblair: the first cluster of failures i analyzed on saturday affected multiple downloads in several jobs all in one neutron network in a span of ~2 minutes16:06
jeblairfungi: ah, so you think they are topologically and temporally clustered16:06
anteayajohn-davidge: thank you!16:06
*** e0ne has quit IRC16:06
fungijeblair: yeah, i'm suspecting we may find similar patterns repeated16:07
*** Guest15849 has joined #openstack-infra16:07
*** ChuckC_ has joined #openstack-infra16:07
*** chuckC has quit IRC16:08
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Enable puppet3 master bootstrap
clarkbalso b1,3,5 seemed worst affected16:08
anteayafungi: when might you have a moment to disable 10385 citrix_xenserver_ci Citrix XenServer CI openstack@citrix.com16:09
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Formatting updates in hiera.yaml
*** andreaf is now known as andreaf_16:09
fungijeblair: around 21:4616:09
*** mikedillion has joined #openstack-infra16:09
*** MaxV has quit IRC16:09
fungianteaya: done16:09
*** MaxV has joined #openstack-infra16:10
* anteaya sends email16:10
jeblairfungi: oh, are they all in b3?16:11
*** bdpayne has quit IRC16:11
fungijeblair: in that particular incident they were, yes16:11
jeblairfungi: skimming logstash now, i see only b1 and b316:11
fungijeblair: at the time those were the only hits since we had completed the cutover16:11
*** bdpayne has joined #openstack-infra16:11
jeblairincidentally, b3 took the longest to rebuild images16:11
fungithat seems potentially related16:12
*** vponomaryov has left #openstack-infra16:13
*** isviridov is now known as isviridov_away16:13
*** _buzz has joined #openstack-infra16:13
*** rushiagr is now known as rushiagr_away16:14
*** MaxV has quit IRC16:14
*** vponomaryov has joined #openstack-infra16:16
dhellmannmordred: what's the story with sqlalchemy-migrate these days? we need for if that's possible16:16
dhellmannmaybe someone else knows ^^? I haven't been involved in managing that project at all, but now it's apparently a blocker for an oslo blueprint16:16
*** bdpayne has quit IRC16:16
*** pballand has quit IRC16:17
*** [HeOS] has quit IRC16:19
*** ociuhandu has joined #openstack-infra16:20
*** johnthetubaguy is now known as zz_johnthetubagu16:22
*** jistr has quit IRC16:22
*** pballand has joined #openstack-infra16:23
clarkb,members it has an independent core team16:23
*** hdd has joined #openstack-infra16:24
*** daya_k has joined #openstack-infra16:24
*** hdd has quit IRC16:27
*** gokrokve has joined #openstack-infra16:28
*** mikedillion has quit IRC16:28
*** yamamoto has joined #openstack-infra16:30
*** mikedillion has joined #openstack-infra16:30
*** gokrokve_ has quit IRC16:31
openstackgerritA change was merged to openstack/requirements: Bump glanceclient version to 0.14.0
*** gokrokve has quit IRC16:32
*** _nadya_ has joined #openstack-infra16:33
*** bhuvan has quit IRC16:33
*** changbl has quit IRC16:34
*** yamamoto has quit IRC16:35
fungijd__: see my comment on 110256... it's unclear to me why you're adding the python3-jobs template to stackforge/octavia16:35
jd__fungi: probably a rebase error16:35
jd__I've done so many of those16:35
jd__fungi: updated :)16:36
openstackgerritJulien Danjou proposed a change to openstack-infra/config: Use python3-jobs template where possible
jeblairfungi: !!!!! gnus honored the ansi color escape sequences in your email!16:36
fungijeblair: now i know how to annoy the crap out of you in e-mail ;)16:36
jeblairfungi: send html? ;)16:37
fungijeblair: hah16:37
*** yfried has quit IRC16:37
fungijeblair: i was disappointed, and then immediately relieved that mutt did not interpret the ansi escapes in my message16:37
fungijd__: approved since it already had prior +2 votes16:39
fungijd__: thanks again! this will be a huge help16:39
jd__fungi: while you're there, I'd need to push an old git tag to pylockfile without triggering any job release, could you do that?16:40
jeblairfungi: i'm sure i can (setq "disable-ansi-escape-sequences" '(eval (equal (gnus-article-from "fungi")))) or something :)16:40
jd__the git repo we migrated from github had no tag at all and I'd like to have this one to keep a track of what was the last release16:40
*** reed has joined #openstack-infra16:41
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Enable puppet3 master bootstrap
fungijd__: release jobs won't work for it yet anyway since its tox.ini isn't suitable to run them (there's no [testenv:venv] section in it)16:42
openstackgerritA change was merged to openstack-infra/config: Create Group Based Policy Stackforge project
jd__hum so I could push the tag if I had the right permission16:42
fungijd__: yeah, the tarball/wheel and pypi upload jobs will try and then break horribly, so should be fine to tag16:43
jd__fungi: I think i'm not in the release group for pylockfile on gerrit, can you add me?16:43
jd__I mean the tag group16:43
*** _nadya_ has quit IRC16:43
*** paulrad has joined #openstack-infra16:44
jd__I think it should be releated to oslo-core or something actually, not sure how it's done on other projects16:44
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Enable puppet3 master bootstrap
*** Sincler has quit IRC16:45
jd__dhellmann: if you're interested ^ :)16:45
fungijd__: i gather from that it's targeting inclusion in oslo... as such i've added oslo-release to pylockfile-release and would prefer to let dhellmann add you if that's his preference16:46
jd__thanks fungi!16:46
*** maxbit has quit IRC16:48
*** pblaho has quit IRC16:48
*** paulrad has quit IRC16:48
*** Sincler has joined #openstack-infra16:50
clarkbfungi maybe jeblairs irc client will do color too. clearly we need more color here (but we are +c)16:50
*** bhuvan has joined #openstack-infra16:50
funginah, he uses irssi which i seem to recall filtering embedded color by default back when i was running it. for that matter so did bitchx... the last client i remember using which allowed embedded color by default was ircii16:51
fungithat's been... a *very* long time16:52
fungi(i try not to think about how long, as it just makes me feel old)16:52
clarkbhrm irssi and weechat both did color for me until I fixed them16:53
stevemarinfra folks, i'm seeing a change get consistently -1'ed because it's failing py27 job - always with the same error message: sudo: unable to resolve host bare-trusty-<some_id> does this seem familiar at all?16:53
openstackgerritMehdi Abaakouk proposed a change to stackforge/dox: Checks that docker is installed
fungistevemar: that message is benign (you will see it on successful jobs too)16:53
*** gokrokve has joined #openstack-infra16:53
*** e0ne has joined #openstack-infra16:54
stevemarfungi, good to know16:54
fungistevemar: sudo simply likes to complain when it can't resolve the system's hostname to an ip address (usually because it's missing an appropriate mapping in /etc/hosts)16:54
fungistevemar: it still continues to work fine, you just can't use hostname-based rules in your sudoers files16:54
stevemarfungi, OK *that* makes sense,16:55
stevemarlet me see if i can find another reason why the patch is failing16:55
*** zns has quit IRC16:55
*** hdd has joined #openstack-infra16:56
fungiclarkb: oh, indeed you are correct! i have colors_receive=off set explicitly in my ~/.weechat/irc.conf16:57
fungimaybe i used to do that in irssi as well16:57
*** zns has joined #openstack-infra16:58
*** loki184 has joined #openstack-infra16:58
*** _nadya_ has joined #openstack-infra16:58
*** _nadya_ has quit IRC16:59
openstackgerritChmouel Boudjnah proposed a change to stackforge/dox: Checks that docker is installed
*** otherwiseguy has joined #openstack-infra16:59
*** atiwari has quit IRC17:00
*** marcoemorais has joined #openstack-infra17:00
*** derekh has quit IRC17:00
*** daya_k has quit IRC17:01
*** amuller has quit IRC17:01
JayFIs it possible to /commit/ something back to the repo in a post job?17:01
*** sarob has joined #openstack-infra17:02
JayFi.e. if Ironic wanted to do what Nova is doing (get rid of sample config, make it something generated)... could we add a post job that would commit back the updated ironic.conf.sample?17:02
clarkbJayF "yes" but we do it by proposing new changes/patchsets17:02
mordreddhellmann: have we not managed to move more things to alembic?17:03
*** _buzz has quit IRC17:03
mordreddhellmann: I thought us taking over sqlalchemy-migrate was intented to be a short term bridge to us finalizing the transition to alembic?17:03
JayFclarkb: so in this case, we'd likely just end up with a bonus patch that has to be merged with almost every other patch :( got it17:03
fungiJayF: basically a post job could regenerate the config, check to see if it differs from what's in the branch already, and then propose a patch if it differs17:03
*** loki184 has quit IRC17:03
*** annegent_ has joined #openstack-infra17:04
JayFSo the solution is somewhere in the middle then? Maybe a non-voting job to check if config matches, then something in post that would propose back anything to make the config consistent17:04
JayFso you'd lose the times you have to rebase just to fix the sample config, but in most cases the conf file would be updated in the normal patch17:04
*** maxbit has joined #openstack-infra17:04
clarkbso that is spscifically why nova works the way it does17:05
clarkbthat is tge way it was done17:05
clarkbit broke everything17:05
fungiJayF: probably just drop the check job about the config changing17:05
clarkbfungi no :) we did that in the past and it is terrible17:05
fungiJayF: since it will generally report failure until someone gets around to merging the update17:05
clarkbso rather that commit it at all it is just generated17:06
fungiclarkb: my preferred solution is that it's generated as part of the documentation instead, but lots of people seemed to be opposed to that suggestion17:06
*** rushiagr_away is now known as rushiagr17:06
jeblairtagged gertty 1.0.117:06
JayFWe just have a lot of folks in Ironic (myself included) who appreciate being able to see the conf sample file via a web browser (aka git) as well as seeing the config file diffs as valuable17:07
clarkbJayF thats fine but there will be tons of broken17:07
JayFYeah, there's tons of broken the way it is now too :)17:07
fungijeblair: awesome! upgrading17:07
clarkband most have decided that correct config at any point in time is better than broken17:07
JayFYeah, there's tons of broken the way it is now too :) ... for instance:
JayF(patch for Ironic to fix pep8 jobs... just a commit of the conf file)17:08
mordredjeblair, clarkb, fungi: you know - the region-b mirror is in a different tenant than the nodepool nodes, so I think that means that the network traffic is going to go through those virtual routers just like traffic to anything on the internet17:09
*** sarob has quit IRC17:09
*** yamamoto has joined #openstack-infra17:09
clarkbmordred probably17:09
fungimordred: right, i expected so17:09
mordredperhaps it really wants to be a node in the nodepool tenant and we want to have nodepool nodes talk to it over the 10. address to avoid virtual routers?17:09
clarkbJayF right stop testing that is the answer17:09
*** sarob has joined #openstack-infra17:10
clarkbJayF the underlying issue is that arbitrary deps can change your config17:10
jeblairmordred: the intent was to avoid internet breakage, not so much hpcloud breakage17:10
clarkbif we fix that these problems mostly go away17:10
mordredright. but if the breakage wasnt' actually the internet and it was the outbound traffic through the virtual routers that provide that for us17:10
jeblairmordred: traffic crossing a router is not something that one generally expects to be unreliable17:10
*** _nadya_ has joined #openstack-infra17:10
jeblairmordred: yeah, then i think we've identified a problem that we can help hpcloud fix17:11
jeblairmordred: but i don't actually think it's appropriate to design around that17:11
jeblairmordred: that's basically completely giving up on a working network at all17:11
fungimordred: i would consider that a trouble ticket worth opening with hpcloud... they can't say "oh, well we have no control over packet loss once it leaves our systems"17:11
mordredjeblair: why not? we design around deficiencies in clouds all the time17:11
jeblairmordred: this is not a deficiency in a piece of software, this is a _very_ broken network.17:12
*** gyee has quit IRC17:12
mordredI think we spend a lot of effort working under the assumption that clouds do not work, and we mitigate appropriately once we've learned what reality is like17:12
fungimordred: until now, we've been unable to say for sure it was a network problem within a specific provider, because we were seeing it with connections crossing the greater internet17:12
mordredjeblair: sure. but DNS also doesn't work so we started running our own17:12
openstackgerritp-draigbrady proposed a change to stackforge/gertty: Add help text for HTTP user/pass
jeblairmordred: i don't think we should have to do that either.17:12
mordredjeblair: I agree - but we do17:12
funginow we have something we can take back to hpcloud and say "this is broken"17:12
jeblairmordred: however, hpcloud refused to fix that problem17:12
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Update oslo.utils to our juno rc1 version
mordredthey may or may not fix this problem - but I am writing an email to the network guy and including you on it17:13
openstackgerritp-draigbrady proposed a change to stackforge/gertty: Add help text for HTTP user/pass
mordredas for me though, I'm going to assume that it's not going to get fixed at the cloud level, because my experience so far has taught me that it won't be17:14
mordredemail sent17:14
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Update oslo.utils to latest version
*** harlowja_away is now known as harlowja17:14
clarkba big difference here is without layer 3 networking how do we expect anything to work?17:14
jeblairclarkb: exactly17:14
clarkbthis is fundamental and must be fixed17:14
*** yamamoto has quit IRC17:14
mordredwe don't expect anything to work17:15
clarkbotherwise you are useless17:15
mordredwe go out of our way to never talk to the internet during tests17:15
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Update oslo.serialization to latest version
mordredwe've already assumed that nothing is ever going to work17:15
clarkbthis isnt the internet17:15
clarkbit is layer 3 routing within hpcloud17:15
mordredI believe that between tenants might need to be considered to be "the internet"17:16
clarkbif we want nodes to be able to talk to each other within a tenant across networks this must work17:16
*** Morgan_ has joined #openstack-infra17:16
*** tsg has quit IRC17:16
*** baoli has quit IRC17:17
*** _nadya_ has quit IRC17:17
*** atiwari has joined #openstack-infra17:19
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Update oslo.messaging to latest version
*** praneshp has joined #openstack-infra17:19
fungiand if it's a neutron reliability problem, getting fixes upstreamed so that we, say, don't run into them later when we start using neutron network in rackspace or other providers, would be wonderful17:19
Morgan_clarkb: have a question for you, if we (keystone) want to run tests against a live ldap, does it make sense to have a test just for keystone that does functional testing in a devstack? Or another full devstack-gate? Or something else? (Cc sdague )17:19
clarkbfunctional is probably a good place to start17:20
fungiMorgan_: is keystone's behavior expected to change outwardly (from the perspective of other services) when using ldap? fi not, then functional testing would be the way to go17:20
clarkbtempest shouldnt care what the backend is.17:20
*** MaxV has joined #openstack-infra17:20
clarkbfungi if it is that is hopefully a bug :)17:20
fungiclarkb: my thought as well17:20
Morgan_fungi: it shouldn't change much (a little but it's all deployer choices). Read only ldap vs read/write17:21
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Update oslo.i18n to latest version
Morgan_It really is an outlier in that regard. Eg not directly managing users vs directly managing users.17:21
fungion the other hand, if keystone and some other openstack component were designed to share an ldap database and coordinate things between one another within it, then that would be fodder for integration tests17:21
Morgan_Right don't think that is / will be the case for the foreseeable future17:22
*** maxbit has quit IRC17:22
*** changbl has joined #openstack-infra17:22
*** SumitNaiksatam has quit IRC17:22
fungior if keystone is intended to behave differently from the perspective of other services when used with an ldap backend17:22
fungibut yes this sounds like functional testing is what you want17:22
Morgan_Ok I'll look at setting up a functional gate job and get an install with ldap under experimental until we stabilize it.17:23
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Update oslo.config to latest rc1 version
*** bdpayne has joined #openstack-infra17:23
*** MarkAtwood has joined #openstack-infra17:24
Morgan_It's been on my "we really need this" list for a while17:24
*** markmcclain has quit IRC17:24
*** MaxV has quit IRC17:25
Morgan_Oh. I looked into pbr building tr sample config. Pbr can't hook into some setup commands well. It would be easier to make do sample config generation directly unless we really don't want that kind of logic in setup.py17:26
openstackgerritA change was merged to openstack-infra/config: Use python3-jobs template where possible
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Clean up bashate failures
Morgan_Setup does weird things like instatiate new commands and run them outside of where pbr wraps them.  I'd like to make sample config on-demand but it is a bit of a headache.17:27
*** zns has quit IRC17:28
jesusaurusnibalizer: have you done any work around splitting out an initial module yet?17:28
*** gyee has joined #openstack-infra17:29
*** Guest15849 has quit IRC17:29
*** baoli has joined #openstack-infra17:30
*** changbl has quit IRC17:32
*** yfried has joined #openstack-infra17:32
*** annegent_ has quit IRC17:34
*** IAX has quit IRC17:34
*** IAX7 has quit IRC17:34
*** markmcclain has joined #openstack-infra17:35
*** mwagner_lap has joined #openstack-infra17:36
*** aysyd has quit IRC17:36
*** SumitNaiksatam has joined #openstack-infra17:38
*** aysyd has joined #openstack-infra17:39
*** esker has joined #openstack-infra17:39
*** skelpter1 has joined #openstack-infra17:40
*** ildikov has joined #openstack-infra17:41
*** paulrad has joined #openstack-infra17:43
mordredMorgan_: we REALLY don't want logic in setup.py17:44
morganfainbergmordred, figured.17:44
clarkbjesusaurus: we split out storyboard, but that one was a bit different17:44
*** emagana has joined #openstack-infra17:45
morganfainbergmordred, unfortunately setuptools is uuuuuugly :( i'll see if i can figure out how to hook things into the dynamic commands but... it was not looking promising17:45
*** Ryan_Lane has joined #openstack-infra17:45
*** [HeOS] has joined #openstack-infra17:45
clarkbmorganfainberg: mordred: pbr does it with the test command, though I suppose that is already a thing in setuptools?17:45
*** gyee has quit IRC17:46
* mordred on the phone, let me read the scrollback in a sec17:46
*** IAX has joined #openstack-infra17:47
morganfainbergclarkb, the issue is in cases like develop, etc it creates the command 'build_py' or similar on the fly, and you can't do hooks around those. also it doesn't let you hook into develop, etc. and for the most part we need the package installed to create the sample config. we could probably do something totally different though to get sample configs built.17:47
*** paulrad has quit IRC17:47
dhellmannmordred: no, not yet. I'm hoping with oslo.db in a good state next cycle we'll be able to do that, but that's going to be another cross-project discussion17:48
*** alexpilotti has joined #openstack-infra17:48
*** maxbit has joined #openstack-infra17:49
*** annegent_ has joined #openstack-infra17:50
openstackgerritOscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Adding 'allow-https' parameter to projects config
*** bdpayne has quit IRC17:50
openstackgerritOscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Adding 'allow-https' parameter to projects config
krtayloranteaya, you are chair for today's meeting right?17:51
anteayaI am17:51
*** bdpayne has joined #openstack-infra17:51
krtayloranteaya, thanks, just checking :)17:51
krtayloranteaya,  I can take next week if you want, you have run a few in arow now17:51
anteayaokay thanks17:52
anteayait is yours17:52
krtaylorwe should get to every other week or something17:52
anteayalet's do clumps17:52
anteayaor whenever you want to take one17:52
krtayloranteaya, sure, that works too17:52
krtaylorI'll run a few starting next week17:52
anteayaI'm more of a clumps kind of person17:52
*** mpaolino has quit IRC17:53
*** _buzz has joined #openstack-infra17:54
*** cdent_ has joined #openstack-infra17:54
dhellmannhmm, does search not work in our wiki?17:54
anteayait doesn't work well17:54
dhellmann"An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later"17:54
*** zns has joined #openstack-infra17:54
anteayaI do terms in duckduckgo myself17:55
anteayahaven't seen that before17:55
dhellmannnot a high priority, I'll open a bug17:55
clarkbhrm ryan did do updates semi recently. I wonder if we need to udpate the es version17:55
adam_gclarkb, if you get a min i topic'ed the stuff we need to get the grenade sideways stuff rolling,n,z17:55
anteayaI wonder if that might be an artifact of a change Ryan_Lane made a week ago or so17:55
clarkbanteaya: ya I am guessing we need to update elasticsearch17:55
anteayadhellmann: might if I ask what you were searching for?17:55
anteayaclarkb: ah17:56
dhellmannanteaya: I tried "sqlalchemy", "sqlalchemy-migrate", and then "oslo"17:56
*** cdent has quit IRC17:56
*** cdent_ is now known as cdent17:56
anteayaerror on all three searches?17:56
Ryan_Laneoh, crap17:56
Ryan_LaneI think it's necessary to run some maintenance scripts17:56
Ryan_Lanesince I upgraded the version of the search extension17:57
anteayahey Ryan_Lane17:57
Ryan_LaneI always forget about that17:57
anteayathat would do it17:57
Ryan_Lanewe should document the upgrade steps in the docs17:57
anteayaisn't it nice Doug helped us to find that17:57
uvirtbotLaunchpad bug 1366915 in openstack-ci "wiki search returning error message" [Undecided,New]17:57
Ryan_LaneI can fix this when I get home, but I can also probably find docs right now for you17:57
openstackgerritJeremy Stanley proposed a change to stackforge/gertty: Display version in help dialog title
*** koolhead17 has quit IRC17:57
anteayaRyan_Lane: thanks17:58
Ryan_Laneyou run that from <current-slot>/extensions/CirrusSearch17:59
*** _nadya_ has joined #openstack-infra18:00
fungiRyan_Lane: anteaya: dhellmann: i can take care of it now18:00
*** mrmartin has joined #openstack-infra18:00
Ryan_Lanelet me know if you run into issues18:00
fungiRyan_Lane: which of those three options do you believe we need after your most recent update?18:01
mordredmorganfainberg: ok. I'm back ... you want to run tests with a config is the thing you're trying to solve?18:01
*** john-davidge has quit IRC18:01
*** _buzz_ has joined #openstack-infra18:01
*** _buzz has quit IRC18:02
clarkbadam_g: I don't understand what that has to do with cinder18:02
Ryan_Lanefungi: I'd guess 218:02
morganfainbergmordred, well what i was trying to solve was removing the sample.config from the tree but guarantee it was generated on setup, so setup develop would have the keystone config in tree, same for when someone does sdist or bdist, the sample config would be there for them to grab if they wanted (packaging). This solves the 'sample config is out of date' issue.18:03
fungiRyan_Lane: okay, thanks... giving that a shot now18:03
clarkbadam_g: it doesn't touch the sideways job at all18:03
*** dane_leblanc has joined #openstack-infra18:03
morganfainbergmordred, it's a very very minor low prio thing, but we handle sample configs differently in many projects and it would be nice to have them back to consistent without needing to gate on them being up-to-date.18:04
adam_gclarkb, the current check-tempest-dsvm-ironic can't run without cinder (or at least, it blew up when it was disabled last time)18:04
clarkbadam_g: right but that change doesn't do anything with cinder?18:04
adam_gclarkb, the future ironic tempest job, and the grenade job, will need cinder disabled.18:04
adam_gclarkb, we cant do that till
fungiRyan_Lane: is it safe to run those as root, or is there a specific user i need to run them as?18:05
adam_gclarkb, which will allow
morganfainbergmordred, my thought was "hey pbr might do this for us with some stuff in setup.cfg" - but setuptools is a bit wonky.18:05
mordredmorganfainberg: ok. so yeah, that sounds like a thing like how we generate AUTHORS and ChangeLog18:05
Ryan_Lanewww-data should be safe18:05
Ryan_LaneI think it's fine to run as root, though18:05
clarkbadam_g: and that is because we need to trim out the virtual jobs?18:05
mordredmorganfainberg: it's actually not terrible to implement if you know what it is that you want do make18:05
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Allow usage of customized venvs for docs building
*** ociuhandu_ has joined #openstack-infra18:06
*** sarob has quit IRC18:06
morganfainbergmordred, except authors doesn't require the code to be in the active python sys.path.18:06
*** ociuhandu_ has left #openstack-infra18:06
*** ericpeterson has joined #openstack-infra18:06
morganfainbergmordred, the entry points actually need to be fully realized in the venv/system to make it work (depending on project). it might not be worth the headache.18:06
mordredmorganfainberg: hrm.18:07
adam_gclarkb, sort of. we're just replacing the check-tempest-dsvm-ironic with the virtual job and renaming. that will still use the TEMPEST_REGEX to avoid the cinder tests while its still enabled in features.yaml18:07
mordredmorganfainberg: so, how important is the develop usecase?18:07
mordredmorganfainberg: if it does it for sdist - is that good enough?18:07
adam_gclarkb, let me jot down the order of operations in a pad somewhere.18:07
morganfainbergmordred, we use it in devstack. do we care if we have a sample config in devstack?18:08
*** TravT has joined #openstack-infra18:08
mordredmorganfainberg: ah - gotcha18:08
mordredmorganfainberg: so, we'd need to install keystone first, so that the entry points would be available for us to know what it is that we want to install18:08
clarkbadam_g: I am just trying to figure out if this is a hard requirement and it doesn't sound like it18:08
morganfainbergmordred, yeah i think i'll just table this till later on, we've survived until now, we can revisit in K18:09
mordredmorganfainberg: let me noodle on it in my head18:09
ericpetersonquestion on jjb, around the scm setup of remotes....  following    When I add more remotes, how do I give the remotes a name???18:09
*** yamamoto has joined #openstack-infra18:09
morganfainbergmordred, sounds good. no real rush. was just a "hey maybe we can fix this nicely" thought18:09
jogoanteaya: ahh thanks, I can make lastcomment take a list of names so you can check to see if a whole bunch of systems are running quickly18:09
morganfainbergand on the topic of "make things better for deployers/packagers/end users" :)18:10
mordredmorganfainberg: yah. seems like the type of thing pbr would want to be able to know how to do18:10
fungiRyan_Lane: thanks... so when i do 'sudo -u www-data php updateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now' it throws warnings and bails like
anteayajogo: that would be awesome18:10
*** bdpayne has quit IRC18:10
anteayajogo: can you tie it into radar or help me to?18:10
jogoanteaya: where is radar run currently?18:10
openstackgerritJay Faulkner proposed a change to openstack-infra/config: Fix typo causing failure in IPA post job
anteayajogo: stackforge18:11
*** bdpayne has joined #openstack-infra18:11
JayFI'm sorry but I had a typo caused breakage in the IPA post job -- fix is here -- should be a quick review as I'm just correcting the filename. Apologies again for the first patch being wrong...18:11
jogoanteaya: as in is radar being run activly somewhere? not where is the source hosted18:11
Ryan_Lanefungi: I'd try to go with option 1, then18:12
anteayajogo: mikal is running it18:12
Ryan_LaneI also wonder what version of ES is currently required....18:12
anteayajogo: that is the only instance I know of18:12
Ryan_Lane"1.3.2 and above are all fine."18:12
openstackgerritMehdi Abaakouk proposed a change to stackforge/dox: Checks that docker is installed
*** MarkAtwood has quit IRC18:13
mordredJayF: weird. you have a last name18:13
fungiRyan_Lane: "Looks like the index has more than one identifier. You should delete all18:13
fungibut the one of them currently active. Here is the list: openstack_wiki_content_1410199733,openstack_wiki_content_1410199743,openstack_wiki_content_1410199557,openstack_wiki_content_first"18:13
JayFmordred: I guess I'm used to it by now, I've always has a last name.18:14
*** yamamoto has quit IRC18:14
clarkbericpeterson: you may not be able to. do you need that name for something?18:14
clarkbericpeterson: if you list them it should do that right thing iirc18:14
JayFty mordred fungi18:14
*** jcoufal has quit IRC18:14
Ryan_Lanefungi: umm. I have no idea how to handle that18:15
*** tonytan4ever has quit IRC18:15
mordreddhellmann: ugh. ok18:15
Ryan_Laneone sec. I'll ask18:15
dhellmannmordred: we had some issues making both work together, so projects not already on alembic didn't want to adopt it. I think we have fixed that now, so next we have to convince everyone to actually make the switch.18:16
mordreddhellmann: the migration path involves having both work at the same time? that seems scary18:17
ericpetersonthanks clarkb18:17
dhellmannmordred: we run sqlalchemy-migrate, then alembic; then some day we delete all of the old sqlalchemy-migrate migrations and only need to run alembic18:17
*** david-lyle has quit IRC18:17
*** lcheng__ has quit IRC18:18
*** ^d has joined #openstack-infra18:19
^dRyan_Lane said somebody was looking for some help with Cirrus/Elastic for MW?18:19
*** david-lyle has joined #openstack-infra18:20
fungi^d: yeah18:20
fungi^d: i tried option 1A but am getting an error about having multiple index identifiers18:20
JayFfungi: mordred: So I'm just waiting for "Last reconfigured:" at the bottom of Zuul status to be AFTER that merge to test with a patch, correct?18:20
harlowjaclarkb does seem reasonable to u? (allowing those who want to use there own docs venv the ability to)18:21
fungi^d: just curious how to know for sure which one i want (the one with the latest epoch time) and how to remove the others18:22
clarkbharlowja: no, I think we update the standard. otherwise what is the standard for?18:22
*** garyh has quit IRC18:22
clarkbharlowja: then make everyone stick to it. that change only makes it more confusing18:22
harlowjaclarkb fair enough :)18:22
fungi^d: er, should have been a question... do i necessarily want to keep the one with the latest epoch time, or is there some other way of determining which one is supposed to be active?18:22
^dThe one with the latest epoch is definitely the one you're wanting.18:23
fungiokay, good18:23
fungi^d: so then just curious how to delete the others18:23
openstackgerritMatthew Treinish proposed a change to openstack-infra/devstack-gate: Add tempest-lib to the projects list
mordredharlowja: I agree with clarkb - and I think we should change the standard, and I think that the standard should be a venv named docs18:23
mordredand we should add one everywhere18:23
^dfungi: Could do it manually. Issue some curl -XDELETE's at the ES instance.18:23
harlowjamordred standard change it is then, wfm18:24
mordredmaybe this is my way to get my super-ATC status18:24
*** jp_at_hp has quit IRC18:25
harlowjamordred 50 + changes to all the projects are all yours ;)18:25
fungiJayF: contributor to projects in ~all programs18:25
JayFThat sounds... horrifying18:25
clarkbadam_g: one small thing in
fungiJayF: it's the freakish nightmare most of the horizontal effort contributors live with every day18:26
clarkbits like when you play don't starve and the shadows steal your fire18:26
clarkbyou are good for a while but if you don't pick flowers bad things happen18:27
JayFThere was a funny bit in ;login magazine recently with a good quote:18:27
adam_gclarkb, so there are two related threads of work. 1) migrate away from virtual ironic and TEMPEST_REGEX to a regular vanilla looking devstack job 2) sideways grenade, which requires a vanilla tempest run to pass.  2 depends on 1. 1 requires disabling cinder via features.yaml. doing so breaks ironics gate currently, 112134 is the migration to avoid that18:27
JayF"Systems programmers take care of the bear menace"18:27
*** _buzz_ has quit IRC18:27
JayFI think that applies to you guys18:27
clarkbadam_g: oh I didn't realize that dependency existed but I guess that makes sense since grenade doesn't let you turn all the knobs18:28
*** robemcallister has joined #openstack-infra18:28
adam_gclarkb, if we go with 112134, we are free to disable cinder since we'd be using the regex hack temporarily. ocne disabled, we can use the experimental job in 118700 to make sure the vanilla tempest stuff is reliable and stable, and then migrate off of the regex crap18:29
*** lcheng_ has joined #openstack-infra18:29
clarkbadam_g: will need to be rebased on 112134 to use the common builder?18:29
adam_gall the while grenade starts passing18:29
*** vhoward has left #openstack-infra18:30
*** zns has quit IRC18:30
fungi^d: thanks. that's probably getting too deep into es management for my current level of familiarity18:30
fungiclarkb: can you translate?18:30
adam_gclarkb, that depends. jroll has a patch dependent on the common builder to also test IPA agent deployment, which i think is better than copy/pasting boiler plate, IMO18:30
clarkbfungi: its pretty simple. we have a cron to do it for logstash to delete old data. you can look at that cron. I will get al ink18:30
^dfungi: `curl -XDELETE host:9200/index_name_you_want_to_drop`18:31
^dPretty straightforward.18:31
fungiclarkb: oh, no worries--i'll look in the crontab18:31
clarkbfungi: but basically you tell es to delete things via curl. if you want to gui it you can hook up es head to the wiki cron18:31
fungi^d: ahh, i'll give that a shot--thanks!18:31
clarkbfungi: note the crontab on es02 is what you want18:31
clarkber hook up es head to the wiki es18:31
*** ildikov_ has joined #openstack-infra18:31
JayFES Head plugin +++18:31
^dI tried Kopf recently. Like head only a little prettier.18:32
fungiclarkb: oh, neat. i didn't think about trying es head or bigdesk on the wiki server18:32
clarkbJayF: I actually don't run any of them as plugins because yay no auth. but its really easy to run locally and forward to server18:32
*** rushiagr is now known as rushiagr_away18:32
clarkbJayF: then you basically have ssh auth for control18:32
clarkbI am told auth is coming at some point in es18:32
JayFclarkb: I run them as plugins... but we don't open the port so you have to ssh tunnel it anyway. So pretty much same deal :)18:32
clarkbthat will be a great day18:33
^dThere's an authz/authn plugin for ES.18:33
^dI dunno anyone using it though.18:33
fungi^d: thanks for the api delete calls... that seems to have gotten it back on track18:34
^dyou're welcome!18:34
*** craigbr_ has quit IRC18:34
*** ildikov has quit IRC18:34
*** ericpeterson has left #openstack-infra18:34
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Consolidate and rename Ironic jobs
fungiRyan_Lane: anteaya: dhellmann: reindexing seems to be underway now. i'll test it out when it finishes and give you a heads up if it appears to be working thereafter18:35
^dJayF: The best plugin for ES is probably, but I'm biased :)18:35
openstackgerritJamie Finnigan proposed a change to openstack-infra/config: Adding Bandit project to StackForge
anteayafungi: thanks18:36
clarkb^d: the README doesn't tell me what it does :)18:36
clarkbdoes it store the indexes in swift?18:36
^dSnapshots in swift.18:36
^dThey already had a hdfs and gridfs plugin.18:36
clarkbhrm that may be a good alternative to deleting older indexes. however we already do all the terabytes18:37
clarkbmight not be very fast18:37
^dThey're incremental.18:37
^dSo might not be too bad after initial dump.18:37
clarkbswift supports incremental operations?18:37
*** tsg has joined #openstack-infra18:37
clarkboh the snapshots themselves are incremental18:37
*** annegent_ has quit IRC18:38
Ryan_Laneoh, wow, that's pretty cool18:38
^dI've run it on prod on testwiki and it works :)18:38
dhellmannfungi: thanks!18:39
clarkbI will definitely have to take a look at using it18:40
fungiRyan_Lane: "Indexed a total of 4564 pages at 29/second" but i'm still getting "An error has occurred while searching: We could not complete your search due to a temporary problem." when i try to search for things18:40
clarkbas it may make our rolling off old indexes less painful18:40
fungiRyan_Lane: is there anything else which needs doing/restarting?18:40
*** daya_k has joined #openstack-infra18:40
Ryan_Lane^d: ^^ ?18:40
Ryan_LaneI wouldn't think so18:40
fungiapache? php-fpm?18:42
*** MarkAtwood has joined #openstack-infra18:42
fungii'm trying things i don't think anyone has searched for recently, so hopefully isn't running into a cached negative response of some sort...18:42
fungirestarted both apache2 and php5-fpm just in case, but no change18:44
fungielasticsearch too18:44
clarkbadam_g: ok I am +2 on the cinder change and the config change it depends on. did you decide if that other config change needs to be rebased on the first one?18:44
*** esker has quit IRC18:45
^dGrrr, anything in apache error log?18:46
^dThat's a vague response and I hate it.18:46
^d(our own fault)18:46
jogoso oomkiller is back18:46
openstackgerritA change was merged to openstack-infra/config: Fix typo causing failure in IPA post job
*** dangers_away is now known as dangers18:48
*** daya_k has quit IRC18:48
fungi^d: Search backend error during full_text search for 'jenkins' after 124.  SearchPhaseExecutionException[Failed to execute phase [dfs], all shards failed] [Called from CirrusSearch\\ElasticsearchIntermediary::failure in /srv/mediawiki/slot0/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php at line 97] in /srv/mediawiki/slot0/includes/debug/MWDebug.php on line 30218:48
fungi^d: i'm guessing elasticsearch is unhappy with me18:48
^dEwww. I think we need a Nik.18:48
*** jerryz has joined #openstack-infra18:49
*** mikedillion has quit IRC18:49
clarkbjogo: looks like nova agent invoked it and mysqld was sacrificed18:49
jogoclarkb: heh yeah nova-agent is rax evilness18:50
*** manybubbles has joined #openstack-infra18:50
jogothis is the bug mtreinish found I think18:50
adam_gclarkb, it shouldn't. the second (118700) is required to set TEMPEST_CONCURRENCY, but adds that experimental job to prove the parallel tempest testing is stable before cutting over to it for the main jobs18:50
fungimanybubbles: it's a mediawiki party in here18:50
clarkbjogo: well yes, but at that point the next malloc would have caused it18:50
manybubblesits a mediawiki party everywhere ^d and I go18:50
clarkbjogo: so nova-agent is just noise really.18:50
clarkbadam_g: right but it doesn't use the standard builder18:50
*** openstackgerrit has quit IRC18:51
mtreinishclarkb, jogo: yeah mysqld always seems to be sacrificed18:51
jogomtreinish: so we are running out of memory in the gate18:51
jogoclarkb: query: Cannot set up guest memory 'pc.ram': Cannot allocate memory18:51
jogowrong bug18:51
adam_gclarkb, what is the standard builder?18:51
mtreinishjogo: I think it's because that run had the ceilo notification tests enabled18:51
jogomtreinish: interesting18:52
clarkbadam_g: the one you add in your other change18:52
jogomtreinish: so when 8GB of memory isn't enough18:52
mtreinishI do wish dstat showed the top memory consumers too18:52
fungimanybubbles: so... i'm getting errors in the apache log when i try to search, and they look like
clarkbmtreinish: oomkiller shows you18:52
fungimanybubbles: this is even after blowing away and regenerating the index from scratch18:53
*** robemcallister has quit IRC18:53
clarkbmtreinish: jogo: the rss column is the resident set size18:53
clarkbmtreinish: jogo: I think it is in kilobytes18:53
adam_gclarkb, no, it shouldn't be using that.18:53
clarkbadam_g: ok18:53
manybubblesfungi: k.  to which version have you upgraded?18:54
adam_gclarkb, that will go away once we get rid of the regex crap, or will just be reconfigured to look more like the more standard parallel one being added in 11870018:54
manybubblesand which version of elasticsearch are you using18:54
fungimanybubbles: to which version of mediawiki or which version of elasticsearch?18:54
*** jeblair-shout has joined #openstack-infra18:54
fungiahh, both. checking18:54
manybubblesfungi: all the versions!18:54
jogomtreinish: we can make it show that18:54
adam_gclarkb, sorry for the confusion18:54
manybubblesfungi: ok - pretty now18:55
mtreinishclarkb: It was more to show it over the whole run, not just when things blow up18:55
mtreinishbut it is interesting to look at when things fall apart18:55
mordredclarkb, harlowja:
manybubblesfungi: did you branch this from one of the wmf branches?18:55
fungimanybubbles: 1.24wmf19
mordredjeblair: ^^ you too18:55
fungimanybubbles: not sure... Ryan_Lane did the upgrade18:55
fungimanybubbles: but presumably18:56
harlowjamordred cool, i was just about to send a ML mail about that, just so people are aware18:56
manybubblesfungi: yeah - if its wmf19 then that is what is on wikipedia now18:56
mordredI have submitted 4 changes to governance, 2 to import the current project testing interface, one to update it to match reality, and one to add a docs env18:56
fungimanybubbles: looks like es 1.1.118:56
manybubblesfungi: yeah - you should upgrade to 1.3.218:56
fungimanybubbles: aha! okay, i'll dig into possibilities there18:56
fungiclarkb: sounds like maybe you were right ;)18:56
clarkbfungi: puppet can do it via a parameter18:56
jeblairjeblair-shout: hi18:56
clarkbfungi: if you want to make that change and have it do the upgrade for you18:56
manybubblesmodern versions of cirrus use groovy instead of mvel and groovy is only available in 1.3.X18:56
*** jeblair-shout has quit IRC18:57
mtreinishjogo, clarkb: like java is the #2 ram consumer18:57
clarkbmtreinish: ya18:57
manybubblesfungi: you can see the query its making:
clarkbmtreinish: actually no, its apache18:57
manybubblesyou could just copy that curl -XPOST that elasticsearch and mediawiki wouldn't be in the way at all18:57
clarkbmtreinish: then nova api18:58
manybubbleslike, if you thought mediawiki was eating an error or something (because, you know, things happen)18:58
jeblairmordred: are you sure we want to support something other than simply build_sphinx?18:58
clarkband the vms18:58
*** wendar_ is now known as wendar18:58
fungimanybubbles: that's an excellent troubleshooting tip. thanks18:58
mtreinishclarkb: oops, yeah looking at the wrong column18:58
manybubblesfungi: there are lots of nice debugging things in the version you have18:58
clarkbjava is up there too though18:58
jeblairmordred: (is there a current need for something other than that?  i thought most folks wanted a doc env just to make it easier, but didn't need to actually do any extra work)18:58
mordredjeblair: it seems folks may want to do proprocessing or the like18:59
manybubblesfungi: this dumps what Elasticsearch has in the index for a page:
clarkbin particular harlowja wanted to generated cahngelog to be included in docs18:59
mordredjeblair: and it's been useful to have an interface entrypoint like that for unittests18:59
jogowhat is hava?19:00
harlowjaya, same interface for all the things would be great19:00
mordredbasically, "make sure there is a tox -edocs and that it results in html docs going into a dir called docs/build"19:00
jogomtreinish: looks like the libvirt issue is related19:00
manybubblesfungi: it also proves you can connect to elasticsearch and there is stuff in it19:00
uvirtbotLaunchpad bug 1366931 in nova "libvirtError: internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory" [Undecided,Confirmed]19:00
harlowjamordred (feel free to add anything i guess)19:00
fungimanybubbles: yep, awesome19:02
fungiRyan_Lane: clarkb: upgrades elasticsearch on wiki.o.o19:02
fungialso openstackgerrit bot seems to have wandered off... checking in on it now19:03
jogomtreinish: lets move the coversation to -qa19:03
mtreinishclarkb: actually it looks like there are 2 procs named 'python' which are each eating more than an individual nova-api worker. (sort is useful...)19:03
clarkbfungi: we should double check that we are firewalled off too as that was an attack vector iirc (pretty sure we are firewalled there though so not a problem)19:03
clarkbmtreinish: those are your test processes19:03
fungiclarkb: looking now19:03
mtreinishclarkb: oh, right because we invoke them with python -m19:04
*** mikedillion has joined #openstack-infra19:04
fungiclarkb: ssh, http, https from everywhere, snmp from a couple specific sources. all other ipv4/ipv6 ingress rejected19:05
clarkbfungi: perfect19:05
fungiso sayeth ip{,6}tables -L19:05
clarkb9200-9400 is blocked so we are good19:05
*** ^d is now known as ^demon|lunch19:06
fungigerritbot restarted on review.o.o, should return to us momentarily19:07
jogomordred: did you know we run zookeeper in dsvm now19:07
jogomordred: the more you know19:07
*** openstackgerrit has joined #openstack-infra19:07
*** tonytan4ever has joined #openstack-infra19:07
clarkbjogo: we run it on all of our unittest slaves too19:08
jogoclarkb: in devstack it turns into a memory hunry java monster19:08
*** marcoemorais has quit IRC19:09
jogoclarkb: why in unittest slaves?19:09
*** yamamoto has joined #openstack-infra19:09
fungijogo: knowing is half the battle19:09
clarkbfor testing? iirc harlowja and jd__ wanted it to test tooz and things?19:09
*** marcoemorais has joined #openstack-infra19:10
harlowjaalthough i can fake most of zookeeper out with
jogofungi mordred: so for dsvm Sahara requires it19:10
*** yamamoto has quit IRC19:11
harlowjasaharas usage is probably different than [taskflow, tooz] usage i would guess19:11
*** yamamoto has joined #openstack-infra19:11
jogoharlowja: hadoop requires it19:11
harlowjaya, that would make sense :)19:11
fungiyeah, i expect it does19:11
fungiincestuous apache fun19:12
clarkbfungi: we have no footing to stand on while making fun of that :)19:12
openstackgerritEdward Raigosa (wenlock) proposed a change to openstack-infra/config: provide an option in plugin to pre-update a plugin with puppet
fungiclarkb: we are all about some of that, yessiree19:13
openstackgerritEdward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally
openstackgerritEdward Raigosa (wenlock) proposed a change to openstack-infra/config: lsb_release is missing for debian setup
*** mestery has quit IRC19:13
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1366931
uvirtbotLaunchpad bug 1366931 in nova "libvirtError: internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory" [Undecided,Confirmed]
*** mestery has joined #openstack-infra19:14
*** mestery has quit IRC19:15
*** Sukhdev has joined #openstack-infra19:15
*** yamamoto has quit IRC19:15
*** mestery has joined #openstack-infra19:15
*** emagana has quit IRC19:15
*** jcoufal has joined #openstack-infra19:16
*** garyh has joined #openstack-infra19:16
*** emagana has joined #openstack-infra19:16
*** terryw has joined #openstack-infra19:17
*** pblaho has joined #openstack-infra19:18
adam_gclarkb, any objection to making the experimental job i'm adding in 118700 a non-voting job on ironic instead?19:19
*** otherwiseguy has quit IRC19:19
*** SumitNaiksatam has left #openstack-infra19:19
clarkbadam_g: only if you expect this to take a long time. Do we expect it to just work?19:19
*** _buzz has joined #openstack-infra19:19
clarkbI assume you will work it and make it pass so no hard exception unless that assumption is terrible19:20
*** emagana has quit IRC19:20
*** SumitNaiksatam has joined #openstack-infra19:20
*** baoli has quit IRC19:21
adam_gclarkb, it should just work with that other stuff merged. once i manually hacked in the required changes:
*** emagana has joined #openstack-infra19:22
*** andreykurilin has joined #openstack-infra19:22
*** SumitNaiksatam has left #openstack-infra19:22
*** marcoemorais has quit IRC19:22
adam_gclarkb, ill mark 118700 as WIP and wait till that other stuff is in19:22
*** marcoemorais has joined #openstack-infra19:23
openstackgerritMark Vanderwiel proposed a change to openstack-infra/config: Set ruby and gem default to 1.9.1 for cookbook jobs on precise
*** zns has joined #openstack-infra19:23
*** zns has joined #openstack-infra19:23
*** SumitNaiksatam has joined #openstack-infra19:24
clarkbadam_g: ok19:24
clarkbadam_g: I am +2 on all of the other changes with that topic19:24
markvanfungi: ping,  still having issues with Ruby on precise, but getting closer.  I would like to confirm a couple assumptions I have made.19:25
clarkbmarkvan: fungi: is there a reason we are using precise instead of trusty?19:25
*** SumitNaiksatam has left #openstack-infra19:26
markvanclarkb: yeah, we are using the latest berkshelf 3.x and it's only support on precise at the moment because of ugly dependencies.  That's being worked, but would be ready for a while yet.19:26
adam_gclarkb, cool, thanks. the baguettes are on me in paris.19:26
fungiclarkb: i gather the jobs originally used trusty and have been switched to precise because puppet 2.7 isn't well suited to trusty versions of things19:26
*** penguinRaider has joined #openstack-infra19:26
fungimarkvan: ahh, so it was new berkshelf not working on trusty?19:26
*** SumitNaiksatam has joined #openstack-infra19:27
markvanfungi: yeah, that was the trigger that started this.  We need some of the features and stability in berkshelf 3.x for latest cookbooks.19:27
clarkbI see19:27
clarkbadam_g: no problem. now we need to bribe another core and split the baguettes19:28
markvanfungi: clarkb I pusked up another patch
*** yfried has quit IRC19:28
markvanfungi: clarkb: it's making the assumption that both ruby1.9.1 and update-alternatives is installed.   Can you comfirm that by running a "update-alternatives --list ruby" on a precise node?19:29
clarkbmarkvan: they should be installed19:29
clarkbwe used update alternatives on the slaves for other things19:29
markvanclarkb: ok, cool, then I should be on the right track here, at least for precise.19:30
clarkbturns out ant on precise depends on java6 and not java6|7 so we had to monkey patch that on precise19:30
clarkbit was good fun :)19:30
fungiupdate-alternatives is basically impossible to not find on any debian derivative distro19:30
*** doug-fish has left #openstack-infra19:31
clarkbwe do allow pushing via https now on our gerrit right?19:31
mordredwe do?19:31
clarkbzigo could in theory use that as an alternative to ssh over 29418?19:31
clarkbmordred: I think we do19:31
mordredclarkb: when did that start?19:31
clarkbmordred: when we upgraded gerrit19:32
mordredoh, weird19:32
* clarkb tries to figure out a definitive answer19:32
anteayaclarkb: would be easier than getting the chinese government to allow the ip19:32
clarkbanteaya: probably. I am going to give it a go with the sandbox repo19:32
zigoanteaya: I wouldn't bet on that !!!19:32
fungiclarkb: i *believe* the support for it is well baked in git-review now too... would be interesting to confirm whether it can be used entirely without ssh access now19:32
markvanfungi: clarkb: ok, thx.  My patch is then basically just trying to set the ruby and gem for 1.9.x.19:32
clarkbmarkvan: yup +219:33
markvanfungi: you once posted the packge list for the precise nodes, can you post that link again, would like to double check a few things.19:33
fungimarkvan: oh, i can probably dig it up out of my channel log19:33
fungijust a sec19:33
zigofungi: Any idea on what version?19:33
zigo(or how to do it...)19:34
clarkbzigo: latest version. you have to set your gerrit remote to be https19:34
clarkbzigo: I am sorting it out now and can give details shortly19:34
markvanfungi: clarkb:  ty.   the cookbook team is looking ahead to what it will take to get to trusty with ruby and berkshelf (or Chef SDK).19:34
fungizigo: 1.24 (maybe 1.23 had some support for it but i think it was incomplete)19:34
zigofungi: I have 1.24 (in fact, I maintain the package in Debian... :P)19:35
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1366931
uvirtbotLaunchpad bug 1366931 in nova "libvirtError: internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory" [Undecided,Confirmed]
*** craigbr has joined #openstack-infra19:36
fungizigo: yep, i remember. thanks again for keeping up that package19:37
* zigo goes to sleep, it's nearly 4am here... :(19:38
*** _nadya_ has quit IRC19:38
clarkbzigo: ok I will respond to your thread with what I have learned19:38
*** amcrn has joined #openstack-infra19:40
*** doug-fish has joined #openstack-infra19:40
*** paulrad has joined #openstack-infra19:40
*** salv-orlando has quit IRC19:40
fungiclarkb: i assume you have to authenticate with a gerrit http api password?19:42
*** baoli has joined #openstack-infra19:43
*** paulrad has quit IRC19:44
*** annegent_ has joined #openstack-infra19:45
clarkbfungi: zigo sent19:46
*** emagana has quit IRC19:47
*** baoli has quit IRC19:49
*** wenlock has quit IRC19:51
*** wenlock has joined #openstack-infra19:51
*** emagana_ has joined #openstack-infra19:51
*** wenlock has quit IRC19:51
*** wenlock has joined #openstack-infra19:51
*** baoli has joined #openstack-infra19:52
ekarlso-phschwartz:  < is he back yet ? :)19:52
phschwartzekarlso-: I am here. Haven't looked in #openstack-infra today. Been busy with a pressing change internal here.19:53
phschwartzekarlso-: whats up?19:53
phschwartzfungi: did you ping me the other day? did jroll help you or is there still an issue?19:53
*** jgcampbell300 has quit IRC19:54
fungiphschwartz: yeah, we figured it out... turns out someone accidentally launched some servers in the wrong tenant and didn't realize it19:54
openstackgerritA change was merged to openstack-infra/config: Upgrade wiki to use elasticsearch 1.3.2
phschwartzfungi: ah19:55
ekarlso-phschwartz: did you do some stuff with zuul + github?19:55
phschwartzfungi: how has the quota been going with the new version of code out in all regions?19:55
fungiphschwartz: the error+delete cruft is no longer impeding us, so kudos!19:55
phschwartzekarlso-: I have, currently using it in a test environment here to allow zuul to trigger from a pull request in github19:55
jrollphschwartz: wait, is the quota sync issue fixed with the new code?19:56
phschwartzfungi: :) and the neutron team here is working on the underlying issue so the redeletes arent needed (even though nodepool does it behind the scenes)19:56
phschwartzfungi: also looks like our api traffic for the infra tenant went down 20% so that is a plus :)19:56
fungiphschwartz: yep, very awesome19:57
*** ZZelle_ has joined #openstack-infra19:57
fungiphschwartz: that's presumably gained from all those retries we were doing which would never succeed19:57
phschwartzjroll: no, different issue, this is related to jogo's fix that johannes and I found that leaves instances in error+deleting with no way to get around it. The fix is in upstream, but was delayed into production for on_metal19:58
phschwartzfungi: correct.19:58
ekarlso-phschwartz: would you care to share that ? :)19:58
jrollphschwartz: ah, I see19:58
phschwartzekarlso-: I don't have the time at the moment to sanitize it from internal things that are hardcoded in the code at the moment. I will try to do it tonight so I can pass it along.19:58
jrollphschwartz: onmetal delayed the fix that already went out, or the quota fixes?19:58
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Some third-party.rst cleanup
jroll(regardless, sorry for delaying)19:59
openstackgerritA change was merged to stackforge/gertty: Add help text for HTTP user/pass
phschwartzjroll: The fix that already went out. Basically putting the decorator for task_state reset back into the api delete call.19:59
openstackgerritA change was merged to stackforge/gertty: Display version in help dialog title
ekarlso-phschwartz: is it really baked in or ?19:59
ekarlso-I wanna use it for the VyOS project19:59
jrollphschwartz: right, thanks20:00
phschwartzekarlso-: mine was a PoC so I didn't add the calls to pull things like api_key for github out of a config and hard coded it and user data in the triger. I have to clean that all out and set it to pull it from a config before I can put it out there.20:00
ekarlso-phschwartz: ping me when u got something somewhere : )20:01
ekarlso-would be awesome!20:01
phschwartzekarlso-: will do, it will most likely be in a patch in gerrit so I can point you to it when it is out there.20:01
*** dprince has quit IRC20:02
openstackgerritA change was merged to openstack-infra/infra-specs: Multiple Data Dirs proposal
*** gyee has joined #openstack-infra20:03
*** sarob has joined #openstack-infra20:04
*** dtantsur is now known as dtantsur|afk20:04
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Consolidate and rename Ironic jobs
adam_gclarkb, ^ gah20:06
*** zns has quit IRC20:06
clarkbsilly new projects20:06
clarkbok I am grabbing lunch then hopefully diving into proper code review20:07
*** markmcclain has quit IRC20:08
*** _nadya_ has joined #openstack-infra20:09
*** yamamoto has joined #openstack-infra20:09
*** e0ne has quit IRC20:09
mordredphschwartz: if you just xxxx out your keys and submit the patch, I'll be happy to work on the "pull from config" part20:09
*** berendt has quit IRC20:09
*** MarkAtwood has quit IRC20:09
*** mrmartin has quit IRC20:11
dims_mriedem1: jogo: can you please add couple of oslo sync that fix bugs reported against nova to your queue?
jeblairnibalizer: is correct?20:12
jeblairnibalizer: it claims that you own the copyright, rather than your employer.20:12
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Adds support for Ironic parallel Tempest testing
nibalizershould I re-do it to be (c) HP?20:13
*** yamamoto has quit IRC20:13
nibalizeri kinda just did that, let me go find a different HP Spec20:14
*** cipcosma has quit IRC20:14
jrollI don't understand why we make copyright lines anyway, when the foundation owns it :/20:14
jeblairnibalizer: unless you have a special employment contract, you're probably in the same boat as most of us, yeah, should probably just be hp20:14
jeblairjroll: what makes you think the foundation holds the copyright on the work we do?20:14
*** vladan has quit IRC20:15
mriedem1dims_: yeah20:15
jrolljeblair: I thought that's what the CLA does, I may be wrong20:15
*** spiffxp_ has quit IRC20:15
jeblairjroll: almost no one fully understands the CLA, including those of us who have spent quite a lot of time trying to.  but we're pretty sure that's not one of the things it does.  :)20:15
jrollhuh, interesting20:16
KiallI thought the CLA gave the OpenStack Foundation an unlimited / unrestricted right to use and redistribute, rather than gives them the copyright.. But.. As jeblair says.. Who knows...20:16
nibalizerjeblair: okay ill make a patch to fix it20:16
jrolljeblair: so, say, a file that rackspace made. HP adds a bunch of code to that. who owns the new code?20:16
*** spiffxp_ has joined #openstack-infra20:16
jeblairjroll: so welcome to the club of "developers who have agreed to something we don't fully understand".  we have _many_ members!  you are in good company! :)20:16
*** kgiusti has left #openstack-infra20:16
jrolljeblair: does rackspace own it, because the copyright line says rackspace?20:16
jrolljeblair: I thought that was all software devs :)20:16
jeblairKiall: my current understand is fairly close to what you have said, yes20:17
nibalizerits apache2 so 'ownership' is .... ehhh20:17
jrollright. hm20:17
*** Shrews has quit IRC20:17
*** Shrews has joined #openstack-infra20:17
jeblairjroll: the authors (in this case, generally companies since we are doing work-for-hire for them) own their respective parts of those files20:18
jrollright, ok20:18
jrollthanks :)20:18
clarkbother CLAs do come with copyright attribution but this one doesn't20:20
Kiall <-- That's the bit that makes be think it's a usage grant, rather than ownership transfer20:20
clarkbso beware that if you have signed other CLAs $org may actually get copyright20:20
mordredjroll: generally speaking, if you, as the author of a bunch of changes to a file that previously listed Rackspace in the copyright line, feel that you have made changes worthy of indicating copyright ownership - you can add a copyright line for yourself (or your employer as the case may be appropriate)20:20
mordredjroll: so in general, in the case you mention, HP should probably add an HP line20:20
jrollmordred: right on. more curious than anything20:21
mordredjroll: totally. it's an area full of grey :)20:21
jeblairKiall: yep, the foundation, as best as we understand, has rights that go far beyond what's in the apache license.  so while most of us receive the software under that license (from whom we receive it is not entirely clear); the foundation can do almost whatever they want with it.20:21
jrollmordred: as usual :)20:21
*** vladan has joined #openstack-infra20:22
mordredjeblair: to be fair, most of _us_ can also do almost whatever we want with it too, since it is Apache ...20:22
Kialljeblair: Yep, we both agree ;) Just pointing to the section that makes me think we don't transfer copyright :)20:22
mordredjeblair: but I agree with your assessment20:22
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Some third-party.rst cleanup
*** erlon has joined #openstack-infra20:26
anteayaKiall: I was just about to review your patches20:27
*** r-daneel has quit IRC20:27
anteayaKiall: where are we with the discussion with david-lyle about the way forward?20:27
anteayaKiall: or russellb's suggestion for a ml thread?20:27
*** r-daneel has joined #openstack-infra20:28
*** robemcallister has joined #openstack-infra20:28
openstackgerritJames E. Blair proposed a change to openstack-infra/infra-specs: Add config repo split spec
Kiallanteaya: Getting the discussion stared on the ML has falling off list today, I've asked ttx to take it off tomorrows TC agenda, so we can discuss on-list first.20:29
*** jergerber has quit IRC20:29
anteayaKiall: okay, I will hold off reviewing until we have the discussion20:29
jeblairfungi, clarkb: ^ that was a minor change to add 2 files that fungi suggested20:30
Kiallanteaya: no problem20:30
*** gpocentek has quit IRC20:30
*** doug-fish has left #openstack-infra20:30
openstackgerritA change was merged to openstack-infra/config: Set ruby and gem default to 1.9.1 for cookbook jobs on precise
openstackgerritA change was merged to openstack-infra/infra-specs: Add docs publishing spec
*** gpocentek has joined #openstack-infra20:31
annegent_woowoo on that merge! ^^  L(20:32
annegent_er that was meant to be a :) but capital L hijacked it20:32
*** sbadia has quit IRC20:32
*** ^demon|lunch is now known as ^d_20:33
*** bdpayne_ has joined #openstack-infra20:33
anteayacapital L20:33
anteayaalways getting in the way20:33
jeblairthis misunderstanding has been brought to you by the capital letter L20:34
anteayaone one letter L20:34
anteayaha ha ha ha20:34
anteaya*thunder and lightening*20:35
*** chuckC has joined #openstack-infra20:35
*** ChuckC_ has quit IRC20:35
*** bdpayne has quit IRC20:36
*** sbadia has joined #openstack-infra20:36
*** ^d_ is now known as ^d20:36
mordredjeblair: btw - out of all of the repos we have, 11 need to have their doc env renamed to docs, 9 need one added. the rest either already have a docs env, or don't have a docs dir20:38
mattoliverauLol, sesame st humour, is that what you all get up to while us Aussies sleep :p20:39
mordredjeblair: _all_ of the integrated projects have a docs20:39
*** gpocentek has quit IRC20:39
*** gpocentek has joined #openstack-infra20:40
anteayamattoliverau: :D20:40
*** virmitio has joined #openstack-infra20:40
jeblairmordred: ack20:40
anteayamattoliverau: that and so much more, and good morning20:40
mattoliverauLol, I knew if I slept I'd miss out on things! morning :)20:41
*** zns has joined #openstack-infra20:42
*** robemcallister has quit IRC20:42
*** robemcallister has joined #openstack-infra20:43
*** miqui has quit IRC20:43
*** nelsnelson has quit IRC20:43
fungijeblair: Kiall: jroll: mordred: also notable is that work-for-hire copyright ownership is apparently only the case in some jurisdictions. from what i gather quite a few european countries default to the employee owning the copyright and granting potential exclusive use to their employers20:44
*** robemcallister is now known as rmcall20:44
*** mrmartin has joined #openstack-infra20:44
*** annegent_ has quit IRC20:44
anteayamattoliverau: :D20:45
mattoliverauJust boarding a plane so phone is going off for a while, be back soon. :)20:45
jeblairfungi: yep!  it's so exciting!20:45
*** doude has quit IRC20:45
anteayamattoliverau: safe flight20:46
openstackgerritMonty Taylor proposed a change to openstack-infra/gear: Rename doc environment to docs
openstackgerritAlessandro Pilotti proposed a change to openstack-infra/config: Adds cloudbase-init to StackForge
openstackgerritMonty Taylor proposed a change to openstack-infra/jenkins-job-builder: Rename doc environment to docs
fungialso, *some* employers in the usa allow some employees to own copyright for their work during business hours (red hat apparently offers this as an option)20:47
mordredfungi: wait - are you saying something bout murica?20:47
openstackgerritMonty Taylor proposed a change to openstack-infra/zuul: Rename doc environment to docs
fungimordred: yes, land of the lawyers and home of corporations who are people20:48
*** HenryG has quit IRC20:48
*** MaxV has joined #openstack-infra20:48
Kiallfungi: Yep - Isn't that why we have the I-CLA and C-CLA? :)20:49
*** ajo has quit IRC20:49
*** rmcall has quit IRC20:50
*** markmcclain has joined #openstack-infra20:50
mordredKiall: nope. the reason we have those is that we had those when Rackspace owned things and it was perpetuated20:50
*** cnesa10 has joined #openstack-infra20:50
KiallOh.. Fair enough so20:50
clarkbalso somehow nasa owns copyright20:51
openstackgerritMonty Taylor proposed a change to stackforge/gertty: Rename doc environment to docs
mrmartinfungi: hi, what is the easiest way to add a release tag to the openstackid project? and who have the right to do it?20:51
jeblairclarkb: right, which they can't actually do20:51
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Publicize patches sent to openstack/governance
clarkbjeblair: right20:51
*** paulrad has joined #openstack-infra20:52
openstackgerritAlessandro Pilotti proposed a change to openstack-infra/config: Adds cloudbase-init to StackForge
anteayafungi: and women who don't exist20:52
*** terryw has quit IRC20:53
openstackgerritMonty Taylor proposed a change to stackforge/python-jenkins: Rename doc environment to docs
clarkbmrmartin:,members members of that group can push signed tags to gerrit20:53
*** rmcall has joined #openstack-infra20:53
clarkbfor that project20:54
fungimrmartin: according to,access signed tags can be pushed by openstackid-release group members, who are you and the infra-core team20:54
mrmartinso Sebastian can do it20:54
fungier, yeah, not you but smarcet20:54
*** iax7 has joined #openstack-infra20:55
*** zns has quit IRC20:55
*** lcheng_ has quit IRC20:55
mordredclarkb: sigh. the set of patches I just sent in get me exactly zero ATC status anywhere new :(20:55
*** lcheng_ has joined #openstack-infra20:55
*** lcheng_ has quit IRC20:56
clarkbmordred: oh no20:56
mordredclarkb: I know! how else am I going to game the system???20:56
clarkbmordred: whitespace cleanup patces20:56
mordredclarkb: GOLDEN!20:56
*** garyh has quit IRC20:57
clarkbfungi: did you see that git review bug from today? /me digs it up20:57
mordredclarkb: or, I suppose, changing testenv:pep8 to testenv:style everwhere ...20:57
*** yamamoto has joined #openstack-infra20:57
*** zns has joined #openstack-infra20:57
Kiallmordred: BTW the trick with pointless whitespace cleanup patches is to ensure it conflicts with EVERY OTHER change queued for review20:57
*** emagana_ has quit IRC20:57
Kialldon't forget ;)20:57
fungiclarkb: about unicode elipses?20:58
fungiskimmed, did not triage20:58
clarkbfungi: ya the windows thing20:58
nibalizerwe need that livegrep thing20:58
clarkbok I can't find it anymore... but glad you at least saw it20:58
nibalizerso we can search the codebase for unicode snowman20:58
clarkbnibalizer: did you seem my jjb change to add some?20:58
mordrednibalizer: oh, right. I was going to follow up on that livegrep patch a while back ...20:58
nibalizerclarkb: i didn't20:59
clarkblooks like it merged. anyways added snowmen to test that jjb supports unicopde20:59
nibalizerborrowing from rails, there kinda20:59
*** annegent_ has joined #openstack-infra21:00
*** HenryG has joined #openstack-infra21:00
anteayanibalizer: did you get your blkperl to submit a new patch for livegrep?21:00
nibalizeranteaya: i don't think so21:00
nibalizerhe's in mad interviews all day these days so may not do it21:01
nibalizeri should do it21:01
nibalizerbut other things21:01
*** tgohad has joined #openstack-infra21:01
nibalizerer :(21:01
anteayayeah :(21:01
*** emagana has joined #openstack-infra21:01
anteayabut yay for blkperl and his mad interviews21:01
anteayamay he get what he wants21:01
mordredclarkb: you know what fails the snowman test? the solaris filesystem21:02
nibalizermordred: oh i have a good one on that21:02
clarkbmordred: I wouldn't be surprised if a lot of file systems fail that test21:02
nibalizerso nfs to linux clients from solaris21:02
clarkbwe should try fat21:02
nibalizerlinux clinet is very confused about ntp, thinks its the future, writes a file that is older than 203821:02
*** tsg has quit IRC21:02
mrmartinfungi: and what is the proper way to add myself as a group member to openstackid with access to tag creation?21:02
nibalizersolairs can't stat the file on its own filesystem21:02
mordreddhellmann: we could make the next release of pbr be 1.021:03
fungimrmartin: generally you would just need confirmation from smarcet, preferably along with him asking to be removed if he's not going to do it himself21:03
dhellmannmordred: that would make me happy, but it looks like we still have a lot in flight right now21:03
mordreddhellmann: it hasn't had much churn other than lifeless version cleanup - and it's certainly in wide use21:03
dhellmannmordred: the semver stuff lifeless is working on, in particular21:03
*** stevemar has quit IRC21:03
clarkbmordred: did the version thing throwing an exception get sorted out for you?21:04
*** mriedem1 is now known as mriedem21:04
mrmartinfungi: ok, I got it, but does it require some infra config change?21:04
dhellmannI sort of doubt that's going to land this cycle, so maybe that's a post 1.0 feature21:04
mordreddhellmann: oh, I thought that had all landed - let me go look at that stack21:04
mordredclarkb: lifeless tells me he and ttx have a plan21:04
*** stevemar has joined #openstack-infra21:04
*** baoli has quit IRC21:05
dhellmannmordred: tbh, I haven't been focusing on pbr very much this cycle and some of that automation feels interesting but unecessary21:05
*** melwitt has joined #openstack-infra21:05
*** mmedvede has quit IRC21:05
dhellmannmordred: perhaps there's background I just don't know about21:05
fungimrmartin: just requires one of the gerrit admins (myself, mordred, clarkb, jeblair, SergeyLukjanov) to add you to that group. no config change needed21:05
lifelesswhats the about ?21:05
dhellmannmordred: I think the *fixes* landed, but not all of the "make me a new version number" stuff21:05
lifelesswhats *this* about21:05
*** amcrn has quit IRC21:05
fungimrmartin: or preferably, smarcet can add you to that group himself21:05
mrmartinfungi, thanks, I'll drop a mail to smarcet about this21:05
dhellmannlifeless: can pbr be version 1.0 and what's the status of the semver stuff21:06
lifelessthere is a crucial issue we need to finish fixing before doing 1.0.021:06
mordredlifeless: current state of pbr semver21:06
lifelessthe semver stuff has a long tail and as you say much of the tail isn't needed21:06
*** pelix has quit IRC21:06
lifelessthe patch to ignore *bad tags* is needed21:06
lifelessthats two up in the stack last I checked21:06
mordredlifeless: what's the critical?21:06
*** zns has quit IRC21:06
dhellmannlifeless: this one?
lifelesswith the interesting and fun python 3.3 issue I haven't tracked down yet21:07
mrmartinfungi: we have a single master branch there, I found this guide:, is the "Manually push new tag on that branch" required to add / push the new release tag?21:08
*** mwagner_lap has quit IRC21:08
*** tsg has joined #openstack-infra21:08
lifelessmordred: the current state is that if someone does a git tag of 'fred' we blow up rather than ignoring 'fred'. Current released pbr creates a version called 'fred' which is 100% bogus, but doesn't blow up. We shouldn't blow up there.21:08
fungimrmartin: the more general documentation is
*** zns has joined #openstack-infra21:08
lifelessmordred: and I mean literally 'fred', not 'some arbitrary version' - strings that are not versions21:09
*** tgohad has quit IRC21:10
*** ociuhandu has quit IRC21:10
*** arborism has joined #openstack-infra21:10
*** arborism is now known as amcrn21:11
mordredlifeless: yah. grok21:11
mordredlifeless: and I agree- we shoudl not blow up21:11
*** bswartz has quit IRC21:12
anteayafungi: any reason for the first two lines to be there in this acl file?
*** tonytan4ever has quit IRC21:13
anteayafungi: I don't think they need to be there but I am doubting myself21:13
jeblairmordred, clarkb: i'm woring on a reply about the hpcloud router thing21:14
mordredjeblair: cool21:14
*** dkranz has quit IRC21:14
jeblairmordred, clarkb: what's involved on our side if he moves them?  should we spin down our use of the pseudo-providers associated with the routers he's moving?21:15
jeblairmordred, clarkb: because, do we expect a brief, full network outage as it happens?21:15
*** Sukhdev has quit IRC21:15
clarkbjeblair: yes there will be network outage. so I think process our side would be to set quota to zerp for a pseudo provider, let that fall to zero, then move it and reenable21:16
clarkbrinse and repeat21:16
jeblairclarkb: also, just double checking, our router names == our pseudo-provider names, right?  so hpcloud-b5 is openstackjenkins2-router5?21:16
jeblairclarkb: and hpcloud-b1 is openstackjenkins2-router[nothing]?21:16
clarkbor as an alternative, spin up 5 new routers and networks, have them plce them properly, then change the psuedo providers all at once21:16
fungianteaya: that allows their core team to create new branches. changes are they'd be better off restricting that to their release team21:16
clarkbjeblair: ya, we got the digitless network by default21:16
clarkbjeblair: so I didn't bother adding a digit21:16
*** marcoemorais has quit IRC21:17
jeblairclarkb: i actually kind of want to spin down some of hp to reduce the error incidence anyway, so maybe let's do that.21:18
*** TravT has quit IRC21:18
clarkbjeblair: sounds good to me21:19
anteayafungi: I will make that suggestion21:19
*** weshay has quit IRC21:19
lifelessmordred: I'm fine with blowing up when we've got incompatible requirements from the user, but versions like 'fred.dev1.g123123' were never intended ;)21:20
jeblairclarkb: how many ips do we have in a network?  256ish?  think we can bump each other pseudo-provider by some amount?  66 would keep us at capacity.21:22
jeblairthat's "+66".21:22
*** mrmartin has quit IRC21:22
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Reduce hpcloud-b1 and b2 usage to 0
jeblairclarkb, fungi, mordred: ^ strawman21:23
jeblairclarkb, fungi, mordred: also, errors trailed off by 1500 utc today.21:24
openstackgerritRamy Asselin proposed a change to openstack-infra/config: Log jenkins job builder errors on failure.
anteayafungi: are we encouraging folks to use py34 instead of py33 for stackforge yet?
*** david-lyle has quit IRC21:28
*** cnesa10 has quit IRC21:29
*** mfer has quit IRC21:29
jeblairanteaya: we don't encourage anything for stackforge21:31
nibalizeris it fair to say we encourage them to use trusty or be trusty compatible?21:31
EmilienMmordred: I wonder if dox could be used for Automated integration testing in puppet modules21:31
anteayajeblair: very good21:32
fungianteaya: i'll be submitting a mass changeover as soon as i'm done confirming voting jobs for official projects, but note that merged so hopefully projects are using the python3-jobs template in layout.yaml now21:32
clarkbjeblair: I think the dhpc range is a bit smaller than 256 due to ips reservered for other things but not a ton smaller21:33
anteayafungi: thanks, I will keep an eye out for that21:34
jeblairnibalizer, anteaya: stackforge is a self-service system that lets people use the openstack development infrastructure; projects are welcome to do whatever they want.  they probably _should_ do what the rest of openstack is doing, because we add and drop testing platforms according to what it needs.21:34
jeblairnibalizer, anteaya: but the intent is very explicitly that the infra team is not responsible for what happens there.  it is very explicitly out of scope for things that we sink our time into.  :)21:35
*** SpamapS_ is now known as SpamapS21:35
*** SpamapS has quit IRC21:35
*** SpamapS has joined #openstack-infra21:35
jeblairhaving said that, we're nice folks and like to talk about this stuff and fungi is even nicer to propose all those changes.  ;)21:35
*** yamahata has joined #openstack-infra21:35
*** jcoufal has quit IRC21:35
*** bdpayne_ has quit IRC21:35
*** yamahata has quit IRC21:36
*** annegent_ has quit IRC21:36
*** yamahata has joined #openstack-infra21:36
jeblairso depending on exactly what anteaya was asking, the answer is probably "infra has no opinion; addressing that issue is not required for a complete review of the change in question" or "yeah, there's in general a big switch to py34 underway and it might be worth mentioning it to them so they know about it".21:37
*** bdpayne has joined #openstack-infra21:37
*** pblaho has quit IRC21:37
openstackgerritClark Boylan proposed a change to openstack-infra/config: Trim jenkins::slave package installations further
clarkbSergeyLukjanov: jogo sdague ^21:38
clarkbthat will install zk only on the nodes that need it21:38
jeblairanteaya, nibalizer: anyway, this is all overwhelming enough as it is, i just want to make sure folks know that we're only responsible for 112 projects, not 328.  :)21:38
*** flaper87 is now known as flaper87|afk21:38
jeblairclarkb: that being the case, +66 nodes is probably okay, yeah?21:39
clarkbjeblair: yup should be fine21:39
anteayajeblair: message recieved21:39
mordredEmilienM: I'm sure that it could!21:39
jeblairclarkb, fungi, mordred: speedy review of please?  then i'll ask that b1 and b2 be moved21:39
anteayaand fungi is a nice guy to tidy things up21:39
clarkbjeblair: reviewing now21:39
mordredjeblair: on it21:39
mordredthat is21:40
EmilienMmordred: so I'm definitely interested to work on it. If you have 2 min, the context:
clarkbjeblair: you should be able to double check the dhcp ranges if you are worried about it21:40
clarkbof course the tricky part if figuring out hwo to do that.21:40
clarkbgive me a minute and I will see if I can figure it out21:40
mordredEmilienM: we've spun up a channel ... #dox ... that you might want to join21:40
SergeyLukjanovjogo, fungi, mordred, harlowja, fwiw Sahara doesn't require any additional software on the host21:41
*** paulrad has quit IRC21:41
jogoSergeyLukjanov: ack that was a red herring sorry for alarmin you21:42
* mordred throws cat at SergeyLukjanov just for good measure21:42
*** paulrad has joined #openstack-infra21:42
anteayaSergeyLukjanov: hi21:42
anteayaso your wedding is the 13th yeah?21:42
anteayaSergeyLukjanov: so yay for the 13th21:42
SergeyLukjanovjogo, yeah, just making sure that sahara is still in the gate ;)21:42
anteayaand I'll try to remember to embarass you on irc on that date too21:42
anteayajust for the logs21:42
mtreinishSergeyLukjanov: heh, I wouldn't let anyone pull it out without telling you first :)21:43
*** tgohad has joined #openstack-infra21:43
clarkbjeblair: looks like .2 to .254 are in the pool so we are fine21:43
clarkbjeblair: neutron subnet-list21:43
*** MaxV has quit IRC21:44
SergeyLukjanovanteaya, yup, it's sept 13 ;)21:44
*** mriedem1 has joined #openstack-infra21:44
SergeyLukjanovmtreinish, thx :)21:44
anteayaSergeyLukjanov: yay21:44
anteayaDinaBelova: yay21:45
SergeyLukjanovanteaya, but I'm 100% sure that I'll not be in irc this day21:45
pleia2shouldn't be on irc on your wedding day :)21:45
SergeyLukjanovotherwise DinaBelova will kill me21:45
*** mriedem has quit IRC21:45
anteayaSergeyLukjanov: yes she will21:45
anteayabut one for the logs, so you can read scrollback21:45
*** tsg has quit IRC21:46
*** mikedillion has quit IRC21:46
anteayamriedem1: so one of the ci systems is blocked in china21:46
mordredSergeyLukjanov: I duno - I kinda feel like you guys should get married in an IRC meeting ...21:46
anteayaclarkb: I dont' think third party ci accounts can use http21:46
jeblairclarkb, mordred, fungi: cool then we're all set.  when 119933 merges ("Two ones, ha ha ha.  Two nines, ha, ha, ha.  Two threes, ha ha ha."), i'll let rick know that he can move openstackjenkins2-router and openstackjenkins2-router2 an hour after that.21:46
anteayaha ha ha21:47
SergeyLukjanovmordred :)21:47
fungithanks, count21:47
* anteaya pictures the count wearing a fedora, vest, and bowtie21:47
*** paulrad has quit IRC21:47
mriedem1anteaya: yeah21:47
anteayaand looking snappy21:47
mriedem1db2 ci is in china21:47
clarkbjeblair: reading email are those the two we want to move?21:47
anteayamriedem1: hmmmm21:47
mriedem1anteaya: i forwarded the ML thread to the maintainers21:47
mriedem1since right now i have to head out for awhile :)21:47
anteayamriedem1: so I don't think third party ci systems can use http21:47
jeblairclarkb: please double check, but that's the conclusion i came to.  :)21:47
anteayamriedem1: kk, thanks for confirming21:47
clarkbjeblair: oh yup  Imisread the jenkins2 username21:48
anteayamriedem1: I didnt' know this was the case21:48
mriedem1anteaya: no idea, their alternative was moving stuff to the US21:48
jeblairclarkb: i'm assuming that he wants to move any two routers that are on the same host as another router21:48
fungianteaya: yeah, afaik the gerrit event stream is only accessible via ssh21:48
clarkbjeblair: yup tat was my assumption21:48
clarkbjeblair: and 1 and 2 are in that boat21:48
DinaBelovamordred :D lol21:48
anteayamriedem1: hmmm, well hopefully one of their maintainers will at least show up in infra, so we can talk and see if there is another way21:48
*** dims_ has quit IRC21:48
clarkbmordred: jeblair: the answer of use a mirror per router is a bit crazy though21:49
*** dims has joined #openstack-infra21:49
clarkbwe can cross that bridge when we get there21:49
clarkbI think that may have been a network pun21:50
*** sarob has quit IRC21:50
* mordred throws wet cat at clarkb21:50
mordredDinaBelova: all of your family knows how to join an IRC meeting of course, right? :)21:50
jeblairclarkb, mordred, fungi: it sounds like the major issue is that the virtual routers are overloaded.  mostly because of a bad tenant sharing the host that two of our routers are on...21:50
*** alkari has quit IRC21:50
*** alkari has joined #openstack-infra21:51
mordredjeblair: yah21:51
jeblairclarkb, mordred, fungi: and somewhat less so because _we_ are sharing a host (by having 2 of our routers on it)21:51
anteayamordred: will you be officiating? at teh DinaBelova SergeyLukjanov irc wedding?21:51
*** alkari has quit IRC21:51
*** pcm_ has quit IRC21:51
fungijeblair: unless... we... are the "bad tenant"?21:51
*** ^d has left #openstack-infra21:51
mordredfungi: nope, there is amazingly someone apparently worse than us21:52
jeblairclarkb, mordred, fungi: so fixing the bad tenant is one thing (i'm guessing that may have happened earlier, based on lowered error counts), but us moving may help too21:52
* fungi is amazed21:52
*** emagana has quit IRC21:52
nibalizerjust an fyi, we're about to move to, github claims that this won't break anything21:52
fungijeblair: yeah, though doesn't prevent another from taking its place, as history has taught us repeatedly21:52
jeblairat the _very_ least, it makes us more diversified in the bad neighbor case in the future21:52
jeblairand moving to 9 routers will make us even more diversified21:53
mordredjeblair: maybe we shoudl move to 18 routers21:53
mordredjeblair: or or or - router per host!!!21:53
fungiyou beat me to it21:53
* fungi was about to propose that as a nodepool optimization21:53
*** sarob has joined #openstack-infra21:53
clarkbI can just imagine that quota request. we need 600 routers!21:53
mordredwe could do 2 routers per host, and then attach a floating ip from each network to the host and multi-home them21:53
*** dims has quit IRC21:53
anteayanibalizer: let's see what really happens21:53
clarkboh also we should double ceck that 9 routers won't need a quota bump (it shouldn't iirc bceause i asked for 10 and got them)21:54
fungiwe promise we'll never use more routers than we have instances21:54
anteayafungi: ha ha ha21:54
clarkbjeblair: mordred: fungi: another thing to consider is shifting load to west21:54
jeblairexcept it's not really the virtual router that's overloaded, it's the host upon which the virtual router runs that's overloaded21:54
clarkbso add another 5 routers there21:54
sdaguemordred: dox question, how do you envision target setting for different things like zookeeper vs. mysql21:54
clarkbjeblair: mordred fungi: then halve the resources on our existing 5 networks21:54
clarkbmordred: any idea if that is feasible?21:54
mordredsdague: similar to tox - -epy27-ish, I think21:55
jeblairmordred: can we attach to multiple public networks?21:55
mordredjeblair: I do not know - worth checking?21:55
*** dane_leblanc has quit IRC21:55
clarkbjeblair: no I think there is only one public network in theat cloud21:55
clarkbmoving back into west is something we should be doing regardless right?21:55
mordredclarkb: each of our subnets/routers presents a public network21:55
jeblairmordred: because using multiple virtual routers (which we know will be on different physical hosts) actually would help with this problem.21:55
clarkbmordred: but its just Ext-Net they hook into21:55
clarkbwhich is some arbitrary thing we have no insight into aiui21:56
mordredjeblair: yeah- it would be an interesting thing to try - it would CERTAINLY mean adding neutron awareness to nodepool21:56
jeblairmordred: and we don't actually care that we would NAT-out through different ips21:56
jeblairmordred: (until we use afs, then we might care; we'd at least have to think harder about it)21:56
mordredjeblair: but AFS will also solve the problem21:56
*** dims has joined #openstack-infra21:56
jeblairmordred: udp ftw? :)21:56
clarkbI do like the idea of splitting into west as part of this rather than doubling our network/router count in east21:56
clarkbbecause we get more redundancy that way21:56
jeblairclarkb: yes; are they ready for that?21:57
clarkbjeblair: I don't know21:57
sdaguemordred: ok, it seemed like there was only 1 prep section in the yaml21:57
mordredsdague: I do want to make sure we can sanely a) do daily simple things easily and b) express slightly more complex things like "I want a zookeeper env, I want a mysql env, I want to run py27 workload on each"21:57
mordredsdague: I think flaper87|afk and Shrews are poking at that at the moment perhaps21:57
sdaguemordred: sure21:57
sdagueI'm going to play with this a little bit tomorrow to figure out if this is the right seed element for nova functional tests21:58
sdagueI have a couple of good use cases for functional tests we should have, and am going to figure out what infrastructure we need to actually tackle them21:58
*** emagana has joined #openstack-infra21:59
clarkbmordred: who should I/we ask about moving into west?21:59
jeblairsdague: i'm very interested and excited about the dox work, but if we can avoid making functional testing depend on it, that would be swell21:59
clarkbmordred: is that ulf?21:59
*** maxbit has quit IRC21:59
*** tgohad has quit IRC22:00
jeblairsdague: how do you think it might help nova func testing?22:00
mordredclarkb: ulf is certainly a good person to ask22:00
*** datsun180b has quit IRC22:00
sdaguejeblair: mostly, it means you could run real services on a laptop, instead of requiring a vm, and not muck everything else up22:01
*** homeless has quit IRC22:01
clarkbsdague: I don't know if yo ucaught the discussion yesterday but I will continue to use VMs22:01
*** eharney has quit IRC22:01
clarkbsdague: docker isn't a good sandbox env imo22:01
clarkbit solves other problems22:01
sdagueclarkb: that's fair22:01
*** homeless has joined #openstack-infra22:02
sdagueI just want to play with a couple of different approaches here to figure out what's actually going to be helpful for people to run and debug22:02
*** dims has quit IRC22:03
sdagueanyway, there will be some exploration here for a bit to come up with a workable model22:03
mordredthat's my POV at the moment too22:03
*** dims has joined #openstack-infra22:03
*** marcoemorais has joined #openstack-infra22:03
*** wenlock has quit IRC22:03
mordredfolks in the world have done things that are worth poking at - they may solve problems, they may not22:03
*** emagana has quit IRC22:04
jeblairmordred: what's the docker in docker situation like?22:04
*** sarob has quit IRC22:04
*** Sukhdev has joined #openstack-infra22:05
*** ZZelle_ has quit IRC22:05
*** jpich has joined #openstack-infra22:05
mordredjeblair: I do not know - I'd imagine it's on par with whatever lxc in lxc is like?22:06
jeblairbecause i also worry about the test infrastructure affecting test results, so i'm not certain we'd want to run docker in ci22:06
clarkbjeblair: that was actually how the stuff I asked about yesterday came up in my head22:07
jeblairsdague: oh so this is for zookeeper?22:07
clarkbI was thinking about how our sudo checks would potentially change22:07
jeblaircause i mean, we added sudo support to deal with this22:07
clarkband went down the rabbit hole22:07
jeblairwhy don't we just install zookeeper when running the nova unit tests?22:07
*** sarob has joined #openstack-infra22:08
clarkbjeblair: we already install it everywhere22:08
mordredjeblair: we can totally do that ... the question is, how does that translate to something a dev can easily reproduce locally22:08
clarkb(literally everywhere, addresses that)22:08
*** dims has quit IRC22:08
mordredjeblair: one of the nice things about tox is that the thing we run in CI can also be the thing the dev runs locally22:08
jeblairsdague: wait, then what was the problem you mentioned on the thread?22:09
clarkbjeblair: I think it was client bindings22:09
clarkbwhich aren't necessary aiui22:09
clarkbharlowja: ^22:09
jeblairmordred: except that i don't know that we want to use containers in CI22:10
mordredjeblair: I don't know that we want to do that either22:10
jeblairmordred: i think people have have gotten that impression from that thread :(22:11
mordredjeblair: I don't know that we _don't_ ... I feel there is a ton to learn before making a concrete suggestion in either direction22:11
mordredjeblair: nod. I'll follow up22:11
mordredmy main focus with dox is honestly dev workstation22:11
jeblair22:08 < mordred> jeblair: one of the nice things about tox is that the thing we run in CI can also be the thing the dev runs locally22:11
jeblairmordred: i'm having trouble reconciling those two :(22:11
mordredI would love it if we can do a thing that would get us closer to having the same stuff run in both places22:12
mordredbut if we're sudo installing various things on unittest nodes before tests, which makes sense22:12
*** stevemar has quit IRC22:12
mordredthen we're a step away from tox being the encapsulation of what's running on the dev laptop already22:12
mordredbecause there are things that are expressed in the puppet or the jenkins jobs taht are not expressed in teh tox.ini22:13
sdagueclarkb: the client bindings are still required22:13
*** miqui has joined #openstack-infra22:13
sdaguethere is an alternate pure python set of bindings that maybe you could port the code to22:13
mordred_maybe_ an answer to that is just to also express those things ina  docker snippet so that there is an easy way for a dev to run close to the same thing locally22:13
sdaguebut no one is signed up for that22:13
jeblairsdague: what needs to happen for the zookeeper tests to run?22:13
clarkbsdague: I thought they had ported that I guess I was mistaken22:13
sdagueclarkb: nope, the interface is different22:14
anteayazaro where are we with these changes?,n,z22:14
sdaguejeblair: we need to install the python-zookeeper module22:14
sdaguewhich is all C code22:14
jeblairclarkb: and that's not the thing you say is installed everywhere already?22:15
clarkbjeblair: no zookeeper the server is installed already22:15
clarkbjeblair: and that was installed to test the tooz stuff which is the python native lib22:15
jeblairsdague: no-site-packages will prevent us from using python-zookeeper if it's installed anyway, right?22:16
clarkbjeblair: correct.22:16
clarkbI think what sdague really needs is the -dev package needed to link against when pip installs22:16
anteayawho would know how close hpcloud is to the tip of master?22:16
jeblairclarkb: yeah, that was my next question22:16
jeblairclarkb: because pip should download and compile it, right?22:16
clarkbjeblair: yup22:17
jeblairclarkb: but the -dev package is not installed22:17
clarkbya I think that is the issue22:17
mordredoh,well, installing -dev packages on hosts is pretty easy22:17
jeblairclarkb, sdague: so we can add the -dev package to the slave manifest22:17
clarkblibzookeeper-mt-dev is proably what we want22:18
jeblairyeah, i'm not trying to rain on anyone's parade here; i'm just trying to minimize the dependency chain because we have quite long enough chains as it is :)22:18
*** jgrimm is now known as zz_jgrimm22:18
mordredjeblair: but chains are sexy22:19
jeblairso i'd rather not see "we need docker in order to test zookeeper" when it's clearly not the case (regardless of whether docker would make testing zookeeper easier in some situations)22:19
jeblairmordred: i got that from the thread too22:19
mordredjeblair: yah. I do not think we need docker to test anything - we have a cloud22:20
*** jmeridth has joined #openstack-infra22:20
*** dimtruck is now known as zz_dimtruck22:21
jeblairmordred: right, and we decided a while ago not to re-use nodes for both reproducibility and security purposes22:22
jeblairmordred: and docker (ie, linux containers) aren't secure sandboxes, so i'd expect us to continue to spin up/destroy nodes at least as long as that was the case (things are apparently improving there)22:23
jeblair(regardless of whether we _also_ ran docker on them)22:23
*** emagana has joined #openstack-infra22:23
*** ociuhandu has joined #openstack-infra22:23
mordredI completely agree22:24
jeblairmordred: it seems like it's the _specification_ that's important here...22:24
fungii thought the argument was that (some) devs didn't want python-zookeeper in test-requirements.txt because it would require they install zookeeper so that pip would be able to compile the bindings22:24
mordredI think that if we use docker in our gate at any point in time, we will still want single use nodes22:24
mordredI think fungi's thing is the cruz22:24
mordredit's not about zookeeper in the gate22:24
jeblairfungi: thank you :)22:24
mordredit's about zookeeper on dev laptops22:24
*** mwagner_lap has joined #openstack-infra22:24
mordredand dealing with that22:24
mordredespecially for folks who do things like run osx22:24
clarkbmordred: docker doesn't solve osx iether... I mean we can pretend I guess22:25
jerryzclarkb: is http backend option for zuul merger available now? i am worried that china unicom blocking ssh port would affect zuul in china.22:25
fungiso ideally there would be an "optional" version of test-requirements.txt for things which are opportunistically tested but which some devs may not want to install the system prerequisites22:25
mordredjeblair: one of the (admitedly minor) wins we could get, is that with docker image layers, we can, like we do with devstack and git, pre-cache all or almost all of it in the local cache during the prep step22:26
clarkbjerryz: I don't think that change has merged22:26
clarkbjerryz: it needs additional testing and i haven't had time to write it because it will require new features in the test framework22:26
fungijerryz: also, did that solve the lack of gerrit stream-events via http?22:26
mordredjeblair: and then the devs can use the _same_ image locally, also with caching ... but this is about 100 steps in the future with a ton of viability questions22:26
jerryzfungi: yeah.22:26
clarkbmy change doesn't22:27
anteayajerryz: you aren't part of an ibm ci group are you?22:27
clarkbyou still need ssh zuul to gerrit22:27
jerryzanteaya: no22:27
openstackgerritEmilien Macchi proposed a change to stackforge/dox: First implementation of Puppet unit testing
*** emagana has quit IRC22:27
anteayajerryz: okay thanks22:27
ianwcrinkle: you mentioned the release of was due last week, did that not happen?22:28
jeblairmordred: what are you thoughts about clarkb's point about osx?22:28
nibalizerianw: i believe that master is broken so they don't feel comfortable releasing right now22:28
clarkbjeblair: there is apparently a thing in osx that will run docker in a linux vm for you22:28
nibalizerbut she can fill in the details22:28
openstackgerritA change was merged to openstack-infra/config: Reduce hpcloud-b1 and b2 usage to 0
clarkbjeblair: which sort of ends up solving the problem in the most round about way :)22:29
nibalizerclarkb: lots of people use the boot2docker vm in various hypervisors22:29
fungithat seems like a crazy situation22:29
*** emagana has joined #openstack-infra22:30
fungipeople using containers to avoid hypervisor overhead, causing people to use a particular container manager, causing people who run platforms with no container support to run said containers in a vm under a hypervisor...22:30
jerryzclarkb: ok. we will figure it out. We have been forced to be experts on by passing the GFW since long ago.22:30
*** amcrn_ has joined #openstack-infra22:30
jeblairmordred: so there's also the uncertainty around how containers might impact the tests themselves, which i admit is totally fud now, except that it might have an impact on cinder, and it might have an impact if we start testing docker itself.22:31
jeblairmordred: that's just something to keep in mind as we explore22:31
mordredjeblair: yes indeed.22:31
nibalizerfungi: but its all okay right because vmware is fasssssst22:32
mordredjeblair: so, I think the osx thing is not an issue - osx people seem very comfortable with docker in general via whatever that thing is22:32
jeblairmordred: wow :)22:32
mordredjeblair: and the other thing - I don't really think this has any place in integration/devstack22:32
*** amcrn has quit IRC22:32
jeblairmordred: but it's being considered for functional testing22:32
mordredbut yeah - I mean, basically there is a really nice potential workflow and a TON of questions22:32
*** cdent has quit IRC22:33
jeblairmordred: which is like devstack but with less stack.22:33
*** paulrad has joined #openstack-infra22:33
sdaguethe right answer might be to refactor devstack to do dev env setup and not just service setup, as much of the logic is already captured there22:34
sdagueand the run isolation could be determined external22:34
openstackgerritClark Boylan proposed a change to openstack-infra/config: Trim jenkins::slave package installations further
ianwnibalizer: you mean the ci failures?22:34
jeblairsdague: yeah, i think that gets us to functional testing most quickly (is a good idea in general anyway) and doesn't prevent any future work in this area22:35
nibalizerianw: yea, plus beaker failures too i think22:35
nibalizerHunner: maybe knows22:35
sdagueanyway, I need to go play with some power tools for a bit, and do non virtual hacking before dinner. Talk to folks tomorrow.22:35
mordredlocal on-laptop repeatability and the way that the caching works is the nicest part22:35
mordredsdague: enjoy!22:35
*** paulrad has quit IRC22:36
*** bhuvan has quit IRC22:36
*** paulrad has joined #openstack-infra22:36
jeblairmordred: yeah, it's mapping that to CI that i'm concerned about; i'm not sure we're going to get anything from docker and so am not excited about running it in ci; it seems to make the environment more complex than needed22:36
clarkbjeblair: except clearly this is the future so we must use it >_>22:37
clarkbthat must be my cue to go sort out apartment situation. back later22:37
*** tsg has joined #openstack-infra22:37
*** bhuvan has joined #openstack-infra22:38
*** tgohad has joined #openstack-infra22:39
Hunnerianw: Yeah, beaker tests had some sticky bits when changing the port from one value to another on debian platforms, and we couldn't figure it out by last week :(22:39
HunnerErr, the tests were fine. It's just the module and how the debian init scripts work that was the issue22:39
*** tgohad__ has joined #openstack-infra22:39
ianwHunner: is the failure in fixed?  that seems to have been breaking since "strict_variables" in
*** paulrad has quit IRC22:40
*** tsg has quit IRC22:41
*** tgohad has quit IRC22:42
*** sarob has quit IRC22:43
*** sarob has joined #openstack-infra22:44
jogosdague: I think I have an os-loganalyze bug22:44
jogosdague: where do I File it22:44
*** wenlock has joined #openstack-infra22:45
ianwHunner: "rake spec" -> "170 examples, 115 failures"  ... so clearly it needs more setup than clone, run :(22:45
*** skelpter1 has quit IRC22:46
*** gokrokve has quit IRC22:48
*** bknudson has quit IRC22:49
*** tomoe_ has joined #openstack-infra22:49
*** dmsimard is now known as dmsimard_away22:49
Hunnerianw: so I'll see if that fixes the unit tests22:49
*** asettle has quit IRC22:51
uvirtbotLaunchpad bug 1367021 in openstack-ci "os-loganalyze incorrectly rendering logs" [Undecided,New]22:52
*** sarob has quit IRC22:53
anteayamorning jhesketh22:55
*** marcoemorais has quit IRC22:55
*** marcoemorais has joined #openstack-infra22:55
anteayaI again feel we should have a fake user morning in the channel so tab complete will work22:55
anteayabut they we would get morning: jhesketh22:55
*** paulrad has joined #openstack-infra22:56
*** marcoemorais has quit IRC22:56
*** marcoemorais has joined #openstack-infra22:56
crinkleianw: postgresql did not happen, we had trouble with jenkins, hoping for this week22:56
crinkleianw: sorry :(22:56
openstackgerritElizabeth K. Joseph proposed a change to openstack-infra/publications: Update slides for Fossetcon
harlowjaclarkb seems ok, how does one move to using a thick slave?22:58
clarkbharlowja you are already using them22:59
*** dustins has quit IRC22:59
harlowjathx :)22:59
*** skelpter has joined #openstack-infra22:59
*** tomoe_ has quit IRC23:01
*** amcrn_ is now known as amcrn23:01
*** tomoe_ has joined #openstack-infra23:01
*** emagana has quit IRC23:02
*** emagana has joined #openstack-infra23:02
*** alexpilotti has quit IRC23:02
*** pcrews has quit IRC23:03
*** skelpter has quit IRC23:03
*** marcoemorais1 has joined #openstack-infra23:04
*** _buzz has quit IRC23:04
*** atiwari has quit IRC23:05
*** pcrews has joined #openstack-infra23:05
*** marcoemorais1 has quit IRC23:06
*** _buzz has joined #openstack-infra23:06
*** marcoemorais1 has joined #openstack-infra23:06
*** emagana has quit IRC23:06
*** virmitio has quit IRC23:07
*** marcoemorais has quit IRC23:08
*** flaviof_zzz has joined #openstack-infra23:08
*** dmsimard_away is now known as dmsimard23:09
*** emagana has joined #openstack-infra23:09
*** wenlock has quit IRC23:09
*** signed8bit has quit IRC23:12
fungijeblair: regarding your comment on 110730,2 about layout-dev.yaml being a fit, i based that on the fact that you included review-dev.projects.yaml. in what ways do their inclusion/exclusion differ? maybe i'm still misunderstanding the purpose of the split after all...23:12
*** Longgeek has quit IRC23:14
*** markmcclain has quit IRC23:14
jeblairfungi: huh, well that should probably be consistent :)23:15
ianwHunner: thanks :)23:15
*** bswartz has joined #openstack-infra23:15
*** Sincler has quit IRC23:16
fungiclarkb: the elasticsearch upgrade change seems not to have actually done anything on wiki.o.o (mediawiki is still reporting 1.1.1 present even though i restarted the service for good measure... digging deeper now)23:17
*** penguinRaider has quit IRC23:17
clarkbfungi: hrm, it should check if the deb exists and if not download, verify sha1, then install23:17
fungiohhh... defunct puppet processes23:17
fungisince may23:18
clarkband it uses the apt provider to determine if it needs to do any of those steps23:18
*** bswartz has quit IRC23:18
*** bswartz has joined #openstack-infra23:18
fungisomeone ran 'puppet agent disable' (instead of --disable) and it was still hanging around in the process list23:19
fungihrm... though seems it was still actually applying updates. no outstanding delta23:20
fungidpkg -l claims elasticsearch 1.3.2 is installed23:21
*** yamamoto has quit IRC23:21
fungi disagrees23:21
*** yamamoto has joined #openstack-infra23:21
clarkbfungi: has it been restarted?23:21
clarkbthe package maybe doesn't do that anymore23:22
fungiyeah, i manually restarted it23:22
fungialso restarted php5-fpm and apache2 again just for good measure23:22
*** paulrad has quit IRC23:23
*** paulrad has joined #openstack-infra23:23
*** paulrad_ has joined #openstack-infra23:24
*** paulrad has quit IRC23:24
fungiaha, though according to my terminal history when i first logged in ps showed 1.1.1 running and when i restarted the daemon ps showed it as 1.3.2 thereafter23:25
*** yamamoto has quit IRC23:25
clarkboh good23:25
*** yamamoto has joined #openstack-infra23:25
clarkbbut the wiki search is still broken? maybe we need to reindex again?23:25
fungiso yeah, neither puppet nor deb mainscripts restarted it automagically23:25
*** craigbr has quit IRC23:25
fungii can reindex again, sure23:25
*** sarob has joined #openstack-infra23:26
*** hemna is now known as hemna_23:26
clarkbya I think  Iremember that the es debs were not good debs in that they do not do the debian thing of restarting services now23:26
fungirebuilding now23:26
clarkbthey did in the past but "fixed" that behavior23:26
*** gokrokve has joined #openstack-infra23:28
clarkbbtw  Ifixed the issue with so it passes jenkins now23:28
fungiclarkb: bingo! es upgrade followed by reindex seems to have been the golden ticket23:30
*** tsg has joined #openstack-infra23:30
*** tgohad__ has quit IRC23:30
fungithough strangely the mw version page still reports an older es23:30
clarkbfungi: odd23:31
fungiwell, at this point the primary objective is achieved, so i'll close the bug23:31
*** asettle has joined #openstack-infra23:32
*** tgohad has joined #openstack-infra23:32
*** tsg has quit IRC23:35
*** yamahata has quit IRC23:37
*** emagana has quit IRC23:39
*** luqas has quit IRC23:40
*** itzikb has joined #openstack-infra23:43
*** bdpayne has quit IRC23:44
*** arnaud has quit IRC23:44
*** amotoki_ has quit IRC23:44
*** luqas has joined #openstack-infra23:44
*** yaguang has joined #openstack-infra23:45
*** bdpayne has joined #openstack-infra23:45
*** jaypipes has quit IRC23:46
*** stevemar has joined #openstack-infra23:48
*** aysyd has quit IRC23:48
*** bhuvan has quit IRC23:48
*** weshay has joined #openstack-infra23:49
*** tomoe__ has joined #openstack-infra23:50
*** amotoki_ has joined #openstack-infra23:50
*** amotoki_ has quit IRC23:51
*** sarob_ has joined #openstack-infra23:52
*** gokrokve has quit IRC23:53
*** tomoe_ has quit IRC23:53
*** chuckC has quit IRC23:56
*** rmcall has quit IRC23:56
*** rfolco has quit IRC23:57
*** chuckC has joined #openstack-infra23:57
*** rmcall has joined #openstack-infra23:57
*** unicell has quit IRC23:59
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add Python 3.4 and PyPy to Ubuntu Trusty slaves
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add non-voting gate-git-review-python34 job
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Update StackForge documentation to recommend py34
*** chuckC has quit IRC23:59
*** paulrad_ has quit IRC23:59
*** dims has joined #openstack-infra23:59
*** dangers is now known as dangers_away23:59
*** paulrad has joined #openstack-infra23:59
*** _buzz_ has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!