Thursday, 2015-05-14

fungigrey area, i could go either way on the project creators guide00:00
*** hdd has joined #openstack-infra00:00
fungiit's just nice to have great docs. i'm so glad the problem is not "why don't we have documentation" now, it's "where do we put it?"00:00
jogojeblair: thanks! This is part of larger effort to make the useable00:01
jeblairfungi: haha!  indeed00:01
*** mahito has joined #openstack-infra00:02
fungifirst world developer problems00:02
jeblairjogo: cool, thanks :)00:02
jeblairi need to shift my position in space; bye for now00:02
fungiif you shift it particularly, watch out for time dilation00:03
fungier, particularly quickly00:03
*** hdd has quit IRC00:04
openstackgerritMerged openstack/requirements: Use httpretty < 0.8.7 to avoid broken unittests
fungi(and also lorentz contraction i suppose)00:04
*** banix has joined #openstack-infra00:05
openstackgerritClark Boylan proposed openstack-infra/puppet-elasticsearch: Cleanup old ES logs
clarkbpleia2: jesusaurus (and others) ^ that should keep ES logs reasonable00:06
*** teran_ has quit IRC00:06
fungigrr. we still don't have any jobs registered for bare-precise workers00:06
clarkbfungi: are they just failing to boot?00:06
harlowjaqq, i'm seeing00:07
fungithere's 1 in use for more than an hour, and one stuck in building for almost as long00:07
harlowja'NOT_REGISTERED' on for 2.6; trying a recheck, did something else get bused that might have removed that?00:07
*** teran has joined #openstack-infra00:08
fungii was able to delete the building one, so hopefully it will be replaced shorting by one that successfully boots00:08
fungiharlowja: any python 3.6 jobs should be fune to recheck now. those have been successfully registered00:09
fungiwe're just left dealing with the bare-precise jobs now00:09
harlowjaokie dokie00:09
fungiokay, there's a new one building. here's crossed fingers00:10
fungithere's something not right about the test nodes graph00:11
fungilooks like it flatlined00:12
harlowjaCPR CPR00:12
fungii think it's misreporting00:12
clarkbfungi: ya I wonder if it stopped sending statsd for some reason00:12
*** sputnik13 has quit IRC00:12
fungiokay, that bare-precise worker went ready, so we should have jobs in a sec00:12
clarkbfungi: iirc you can just tcpdump -A the udp port but I Have to remember what port that is00:13
fungilooks like they've registered now00:13
clarkblol did you guys like the clarkb not in sudoers from the git-fe* nodes?00:13
clarkbthat happened because i didn't regroup after changing the sudoer group00:14
fungiboris-42: you should be safe to recheck for not_registered jobs which should have run on bare-precise workers00:14
pleia2clarkb: I saw, figured it was something like that :)00:14
*** gyee has quit IRC00:14
fungiclarkb: yeah, i ignored it. i knew you were on there and doing stuff00:14
clarkbfungi: `sudo tcpdump -i any -A port 8125` doesn't show anything on nodepool.o.o00:15
clarkbI think that is the correct port00:15
fungiclarkb: now that we've got stuff built and registered i can restart nodepoold00:15
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Improve error message on auth_plugin failure
SpamapSgreghaynes: ^ trying to get more insight into what is failing00:16
fungithe graphite server seems healthy00:16
clarkbfungi: ya I think it is nodepool side based on my tcpdump00:16
clarkb/etc/default/nodepool sets up STATSD00:17
fungiclarkb: when did nodepool get restarted for the shade patch?00:17
clarkblet me check irc00:18
clarkbprobably a couple minutes before that00:18
clarkbfungi: looking at the environ file for that process in /proc shows that the stuff was passed in00:19
clarkbfungi: doesn't appear related to when I restarted nodepool for shade if I am reading this graph correctly00:20
fungiyeah, the zuul job queue says there are ~300 jobs running, while the test nodes graph says maybe 75 used00:20
clarkbgraphite.o.o was not rebooted00:21
*** haypo has joined #openstack-infra00:21
fungilooks like your nodepool restart was a good couple hours before the graph went weird, so more likely this is from our reboots00:21
clarkbI wonder if when doing a hard reboot networking is such that we couldn't resolve that node or hit it so statsd just turns off00:21
fungii'm not seeing anything in the logs to explain it00:22
*** ashleighfarnham has quit IRC00:23
haypojenkins voted -1 on a change, but all tests pass: there is "gate-oslo.db-python26 NOT_REGISTERED" which looks suspicious. is there something wrong with this gate?00:23
*** ashleighfarnham has joined #openstack-infra00:23
bknudsonkeystoneclient functional tests are failing...
bknudsonlooks like for some reason the env vars aren't making it to the tox job00:25
clarkbfungi: it definitely appears to only do the getaddrinfo when it starts00:25
*** pabelanger has quit IRC00:26
fungibknudson: if it's complaining about envbindir then new tox is not your friend and you need to do something like neutron did to work around it (or you are not going to space today)00:26
*** mahito has quit IRC00:26
fungihaypo: recheck. when we restarted zuul some jobs ran before any new bare-centos6 workers had registered their ability to run python 2.6 jobs00:26
clarkbbknudson: new tox filters out env vars, you can either whitelist them or explicitly setenv in your tox.ini00:26
bknudsonnew tox?00:26
haypofungi, hum, ok :)00:27
bknudsonclarkb: fungi: thanks, I'll look into it.00:27
fungiclarkb: yep, definitely no packets. i agree that nodepoold probably started before the vm had good network00:27
*** ashleighfarnham has quit IRC00:28
fungirestarting nodepoold now unless there's anything else you want to analyze first00:28
fungishould it have thrown an error about that initially?00:28
*** dangers is now known as dangers_away00:28
fungii couldn't find one if so00:28
clarkbfungi: reading further into statsd code I am convinced the issue was in name resolution00:29
clarkbfungi: I don't think I need to read anything else00:29
clarkbfungi: no statsd masks the error for you00:29
zarolinuxmaniac: finally got around to looking at
fungii expected as much00:29
clarkbfungi: and of course we can't manipulate sysv init order when using upstart and puppet00:30
*** mahito has joined #openstack-infra00:30
fungiyep, i agree00:30
fungiokay, restarting the daemon now00:30
*** zz_dimtruck is now known as dimtruck00:30
clarkbI am checking zuul.o.o now00:30
*** haypo has quit IRC00:31
clarkbI see packets on zuul.o.o00:31
*** otter768 has joined #openstack-infra00:31
fungiclarkb: also when you have a sec, should hopefully fix puppet on the zuul servers00:31
clarkbfungi: looking00:32
clarkbapproved, thats a simple change00:33
fungiit was a simple bug ;)00:33
clarkbI am going to go visit family now, will check in on es cluster later00:33
fungihave a good evening00:33
bknudsoneasy fix --
fungii'm winding down finally as well00:33
fungibknudson: oh, that's a different bug then. i take it also new for tox 2.000:34
fungibknudson: if passenv is more recent than tox 1.6 you probably also want to make sure to bump minversion at the top of the file (i've not seen it used but could predate 1.6 for all i know)00:35
bknudsonfungi: I can try it with an older tox to see if tox barfs on it.00:35
*** bhuvan has quit IRC00:35
*** otter768 has quit IRC00:36
*** Alexandra_ has joined #openstack-infra00:37
bknudsonseems like tox should ignore directives it doesn't understand00:37
fungiseems like a lot of things tox should do... i don't have much faith in assumptions about its behaviors any longer00:37
fungiwell, the test nodes graph is once again nonlinear. that seems to have brought its stats reporting back to reality again00:39
*** ZZelle has joined #openstack-infra00:39
*** claudiub has quit IRC00:41
*** rbradfor has quit IRC00:42
*** mahito has quit IRC00:42
*** signed8bit is now known as signed8bit_ZZZzz00:43
*** mahito has joined #openstack-infra00:44
*** teran_ has joined #openstack-infra00:45
openstackgerritMerged openstack-infra/puppet-zuul: Make sure /var/lib/zuul/www/lib exists
*** teran has quit IRC00:45
*** mtanino has joined #openstack-infra00:46
*** tiswanso has joined #openstack-infra00:49
bknudsontox docs still say {envbindir} --
fungiyeah, the manual claims it exists. i meant to open a bug report about that and then got sidetracked by so so many vm reboots00:51
fungiif you beat me to it. i won't complain00:51
openstackgerritMerged openstack-infra/system-config: Move to HTTPS
openstackgerritMerged openstack-infra/system-config: Correct ci.o.o redirects so slashes are optional
*** alexsyip has quit IRC01:04
*** _1_Abed has joined #openstack-infra01:07
*** ZZelle has quit IRC01:07
*** _1_Abed has left #openstack-infra01:08
openstackgerritDebo~ Dutta proposed openstack-infra/project-config: WIP: Cognitive - Machine Learning as a Service
*** mattfarina has joined #openstack-infra01:13
*** hdd has joined #openstack-infra01:14
*** mattfarina has quit IRC01:16
*** tiswanso has quit IRC01:18
*** sigmavirus24 is now known as sigmavirus24_awa01:18
*** tiswanso has joined #openstack-infra01:18
*** puranamr has quit IRC01:19
*** zhiwei has joined #openstack-infra01:20
fungibknudson: awesome--thanks!!!01:25
bknudsonno problem.01:26
*** ivar-lazzaro has quit IRC01:26
bknudsonlooks like it's only some nodes that have tox 201:26
openstackgerritDebo~ Dutta proposed openstack-infra/project-config: WIP: Cognitive - Machine Learning as a Service
fungiit'll depend on what images were updated and what weren't01:27
fungiwe probably have some stale images for some labels in some providers01:27
fungiokay, i've confirmed that security.o.o is properly redirecting to https and the ssl configuration looks good (chain is intact and whatnot)01:36
fungialso i've switched the dns for to our redirector now, so everyone should be ending up at when they try to follow old links or bookmarks to it01:37
fungilooks like it's propagated and working as expected01:37
openstackgerritDebo~ Dutta proposed openstack-infra/project-config: WIP: Cognitive - Machine Learning as a Service
*** Longgeek has quit IRC01:38
fungialso that puppet-zuul fix seems to have cleared up the puppetboard failures list now01:39
*** woodster_ has quit IRC01:40
*** dims has quit IRC01:42
*** Krinkle is now known as Krinkle|detached01:44
*** mahito has quit IRC01:44
*** Rockyg has quit IRC01:47
*** banix has quit IRC01:48
*** teran has joined #openstack-infra01:56
*** teran_ has quit IRC01:58
*** banix has joined #openstack-infra02:08
*** sdake has quit IRC02:09
*** mahito has joined #openstack-infra02:09
*** Alexandra_ has quit IRC02:10
*** packet has quit IRC02:12
*** sdake has joined #openstack-infra02:12
fungibrb. time to reboot my own servers!02:14
*** fungi has quit IRC02:14
*** spzala has joined #openstack-infra02:16
*** yamahata has quit IRC02:16
*** packet has joined #openstack-infra02:19
*** sarob has quit IRC02:22
openstackgerritMerged openstack-infra/devstack-gate: Remove sahara from the default configuration
openstackgerritMerged openstack/requirements: Removing unused irdragdrop from requirements
*** enikanorov2 has quit IRC02:27
*** unicell has quit IRC02:27
*** fungi has joined #openstack-infra02:27
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras and env markers
*** Longgeek has joined #openstack-infra02:28
tchayposo that time, git review took02:31
fungithat's a lotta waitin'02:32
tchaypo1m53s of that happened *after* the bot announced here that I’d proposed something02:32
*** otter768 has joined #openstack-infra02:32
*** yamamoto has joined #openstack-infra02:35
*** patrickeast has quit IRC02:35
*** jeblair has quit IRC02:36
*** jeblair has joined #openstack-infra02:36
*** otter768 has quit IRC02:37
*** asadoughi has quit IRC02:40
*** Longgeek has quit IRC02:41
*** Longgeek_ has joined #openstack-infra02:41
*** asadoughi has joined #openstack-infra02:44
*** baoli has joined #openstack-infra02:48
*** bhunter71 has quit IRC02:53
*** enikanorov2 has joined #openstack-infra02:57
*** Longgeek_ has quit IRC03:04
*** ivar-lazzaro has joined #openstack-infra03:05
*** alex_xu has quit IRC03:05
*** Longgeek has joined #openstack-infra03:06
*** blahdeblah has joined #openstack-infra03:07
*** david-lyle_ has quit IRC03:11
*** david-lyle_ has joined #openstack-infra03:12
*** panda has quit IRC03:13
*** panda has joined #openstack-infra03:14
*** hdd has quit IRC03:15
*** david-lyle_ has quit IRC03:17
*** markvoelker has joined #openstack-infra03:18
*** achanda has quit IRC03:23
*** banix has quit IRC03:24
SpamapSgreghaynes: looks like it was something that got fixed elsewhere... tests are passing now03:26
*** mahito has quit IRC03:27
greghaynesI have a devstack I stood up during that time period, I might see if I can repro03:27
*** Alexandra_ has joined #openstack-infra03:27
*** Alexandra_ has quit IRC03:28
*** signed8bit_ZZZzz is now known as signed8bit03:28
*** Alexandra_ has joined #openstack-infra03:28
*** signed8bit is now known as signed8bit_ZZZzz03:28
SpamapS2015-05-14 00:35:33.506 | 2015-05-14 00:35:33.480 | shade.exc.OpenStackCloudException: Error getting compute endpoint: Error constructing auth plugin: password __init__() takes at least 2 arguments (1 given)03:29
SpamapSI wonder if keystoneclient had some.. issues?03:29
greghaynesoh, yea03:29
jamielennoxi don't think keystoneclient changed anything03:30
SpamapSgreghaynes: this one succeeded03:30
*** dims has joined #openstack-infra03:31
SpamapScould also still be os-client-config issues03:31
*** packet has quit IRC03:31
*** dimtruck is now known as zz_dimtruck03:31
* SpamapS goes back to family time03:32
*** mahito has joined #openstack-infra03:32
*** tiswanso has quit IRC03:32
*** tnovacik has joined #openstack-infra03:32
jamielennoxgreghaynes, SpamapS: there is a tox issues with v203:32
greghaynesyea, I dont think theres a whole lot we can do with that amount of info :(03:32
greghaynesjamielennox: hrm03:33
jamielennoxwith tox 2 you need to whitelist ENV vars that are passed through03:33
jamielennoxi assume shade is testing that way as most other functional tests do03:33
greghaynesjamielennox: for the auth vars?03:33
*** btully has quit IRC03:34
jamielennoxgreghaynes: for everything03:34
greghayneshuh, thats definitely plausible03:34
jamielennoxif it's doing functional tests by sourcing the rc file that comes from devstack it might be a problem03:34
jamielennox is the ksc fix03:34
greghaynesjamielennox: were testing a thing that loads the config - os-client-config03:34
greghaynesbut it would be succeptible to the same issue03:34
jamielennoxbut it's only in tox 2 so not all the CI tests are picking it up03:34
clarkbare the vars laoded before or after running tox?03:36
clarkbif before then yes likely the issue03:36
greghaynesI actually have no idea for that specific test03:36
greghaynesos-c-c supports both03:36
clarkbProbably makes sense to write a config fike instead03:36
*** dims has quit IRC03:36
*** baoli has quit IRC03:37
jamielennoxthere's a few things we could fix up from the accrc files that devstack writes03:38
*** BharatK has joined #openstack-infra03:39
greghaynesim working on some stuff right now to get devstack writing out an os-cloud-config config file03:40
greghaynesso, we could probably just use that03:40
greghayneser, os-client-config03:40
*** packet has joined #openstack-infra03:41
*** coolsvap|afk is now known as coolsvap03:42
*** yamahata has joined #openstack-infra03:46
*** banix has joined #openstack-infra03:46
*** tnovacik has quit IRC03:47
*** banix has quit IRC03:47
*** yamamoto has quit IRC03:48
*** yamamoto_ has joined #openstack-infra03:48
*** Longgeek has quit IRC03:50
*** mahito has quit IRC03:53
*** Longgeek has joined #openstack-infra03:55
*** mahito has joined #openstack-infra03:58
timrcclarkb, jeblair, mordred, fungi I've hit a bug using apache::vhost::proxy  provided by the 0.0.4 apache puppet module on Trusty with default apache 2.4.7-1ubuntu4.4.  Basically because the site file is not written out to sites-enabled with a .conf terminating it, the vhost goes unrecognized.  The fix was introduced in 0.3.0 --
nibalizertimrc: propose a fix to openstackinfra-httpd ?04:00
clarkbir just use vhost04:00
StevenKtimrc: The apache2 configuration was changed between Lucid and Trusty to stop doing '*' and start doing '*.conf'04:00
clarkbthats why we sticj to the old version04:00
clarkbvhost just works04:00
*** Longgeek has quit IRC04:00
clarkbgerrit, nodepool, and zuul are all trusty at thus point with apache2.4 too04:02
timrcclarkb, Cool.  Surprisingly simple solution to my problem :)04:03
timrcclarkb, Thanks, man!04:03
* timrc just works around the bugs :)04:04
*** jamespage has quit IRC04:05
*** gnuoy has quit IRC04:05
*** jamespage has joined #openstack-infra04:06
*** gnuoy has joined #openstack-infra04:06
*** yamamoto_ has quit IRC04:06
*** amotoki_ has joined #openstack-infra04:07
*** spzala has quit IRC04:10
*** mrmartin has joined #openstack-infra04:12
openstackgerritWayne Warren proposed openstack-infra/infra-specs: Jenkins Job Builder 2.0.0 API Changes/Rewrite
*** dannywilson has quit IRC04:14
*** unicell has joined #openstack-infra04:16
*** packet has quit IRC04:18
*** achanda has joined #openstack-infra04:20
*** unicell has quit IRC04:21
*** unicell has joined #openstack-infra04:22
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Change defcore / refstack Chair
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Update Chef cookbooks meeting
*** JoshNang_ has joined #openstack-infra04:26
*** JoshNang_ has quit IRC04:27
*** JoshNang_ has joined #openstack-infra04:27
waynris anyone aware of JJB config repositories other than openstack-infra/project-configs and wikimedia's?04:27
*** Sukhdev has joined #openstack-infra04:29
*** amotoki_ has quit IRC04:31
nibalizerwaynr: I have a private one ...04:31
nibalizerso if you have questions about it I can answer them04:31
*** JoshNang_ has quit IRC04:31
nibalizernot super useful I know04:31
*** JoshNang_ has joined #openstack-infra04:31
waynri was mostly curious if there were other open source repos, but thanks for the offer :)04:32
*** hdd has joined #openstack-infra04:32
waynri am curious to see what kind of JJB features are widely used04:32
waynrthe 500+ jobs that I've been managing for example don't use macros at all, but I see they are fairly widely used in openstack-infra/project-configs04:33
*** otter768 has joined #openstack-infra04:33
openstackgerritJeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla
openstackgerritJeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla
*** btully has joined #openstack-infra04:37
*** yamamoto has joined #openstack-infra04:37
*** otter768 has quit IRC04:38
*** BharatK has quit IRC04:40
*** btully has quit IRC04:41
*** mrmartin has quit IRC04:41
*** woodster_ has joined #openstack-infra04:47
*** ppai has joined #openstack-infra04:47
*** markvoelker has quit IRC04:47
*** JoshNang_ has quit IRC04:49
*** markvoelker has joined #openstack-infra04:49
*** teran has quit IRC04:50
*** sks has joined #openstack-infra04:52
*** sarob has joined #openstack-infra04:52
*** BharatK has joined #openstack-infra04:53
*** deepakcs has joined #openstack-infra04:54
*** Longgeek has joined #openstack-infra04:57
*** Swami_ has quit IRC04:57
*** yfried has quit IRC04:59
*** abregman has joined #openstack-infra05:00
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron
*** nfedotov has joined #openstack-infra05:01
ppaiHi guys, I'm not able to push signed tags to a stackforge project in which I'm part of the release group. Is there something that I'm doing wrong ?05:01
ppai*unsigned annotated tags05:02
clarkbthey must be signed05:02
*** alex_xu has joined #openstack-infra05:03
ppaiclarkb, thanks!05:03
*** btully has joined #openstack-infra05:04
*** turul_ has quit IRC05:07
*** Longgeek has quit IRC05:08
*** sarob has quit IRC05:14
*** pal has joined #openstack-infra05:17
*** mahito has quit IRC05:20
*** toabctl has quit IRC05:21
*** sdake_ has joined #openstack-infra05:22
*** jlvillal has quit IRC05:26
*** sdake has quit IRC05:26
*** BharatK has quit IRC05:30
*** naggappan has joined #openstack-infra05:36
*** sdake_ is now known as sdake05:38
*** marun_ has joined #openstack-infra05:38
*** marun has quit IRC05:38
*** ildikov has quit IRC05:41
*** hdd has quit IRC05:41
*** JoshNang_ has joined #openstack-infra05:41
*** BharatK has joined #openstack-infra05:42
*** JoshNang_ has quit IRC05:45
*** JoshNang_ has joined #openstack-infra05:45
*** smccully has quit IRC05:45
*** smccully has joined #openstack-infra05:46
*** claudiub has joined #openstack-infra05:47
*** pal has quit IRC05:47
*** baoli has joined #openstack-infra05:48
*** nfedotov has quit IRC05:49
naggappanhi i installed the gerrit and its UI is different from the one which is used in "". In we can point out each job name and its relult and when they click it opens a URL . Is a custom version of gerrit used in "". If so how can i downlaod and install05:49
*** baoli has quit IRC05:53
openstackgerritMerged openstack-infra/project-config: Neutron: enable a voting py34 gate on master
openstackgerritMerged openstack-infra/project-config: Switch on requirements check for Magnum
*** JoshNang has quit IRC05:57
*** JoshNang_ is now known as JoshNang05:57
*** armax has quit IRC05:59
*** dims has joined #openstack-infra06:00
morganfainbergpurp: *waves* just remembered you were on IRC. Annd now back to your regularly scheduled random IRC noise.06:02
*** mahito has joined #openstack-infra06:03
*** teran has joined #openstack-infra06:03
*** Alexandra_ has quit IRC06:04
*** Longgeek has joined #openstack-infra06:05
*** dims has quit IRC06:06
*** mahito has quit IRC06:07
*** mahito has joined #openstack-infra06:08
*** zhiwei has quit IRC06:09
*** camunoz has quit IRC06:14
*** yfried has joined #openstack-infra06:14
openstackgerritMerged openstack-infra/tripleo-ci: Add a ceph job
*** ildikov has joined #openstack-infra06:17
*** mtanino has quit IRC06:17
*** claudiub has quit IRC06:18
*** Longgeek has quit IRC06:22
*** jcoufal has joined #openstack-infra06:24
*** maishsk_afk has joined #openstack-infra06:24
*** afazekas_ has joined #openstack-infra06:25
*** afazekas has joined #openstack-infra06:25
*** GheRiver1 is now known as GheRivero06:25
*** Sukhdev has quit IRC06:29
*** shardy has joined #openstack-infra06:29
*** sdake has quit IRC06:31
*** otter768 has joined #openstack-infra06:34
*** kevinbenton has quit IRC06:36
*** otter768 has quit IRC06:38
*** woodster_ has quit IRC06:50
*** gema_ is now known as gema06:54
*** mpaolino has joined #openstack-infra07:01
*** HeOS has quit IRC07:02
tchayponaggappan: take a look at
tchayponaggappan: there are a few gerrit-related repos - and in particular are probably interesting to you07:06
*** sergsh has joined #openstack-infra07:06
*** ZZelle_ has quit IRC07:08
*** ZZelle_ has joined #openstack-infra07:08
*** nfedotov has joined #openstack-infra07:09
GheRiveromorning all07:09
*** devvesa has joined #openstack-infra07:10
*** e0ne has joined #openstack-infra07:16
*** abregman_ has joined #openstack-infra07:22
*** abregman has quit IRC07:24
*** afazekas_ has quit IRC07:25
*** afazekas has quit IRC07:25
*** mpaolino has quit IRC07:25
*** yamahata has quit IRC07:26
*** afazekas has joined #openstack-infra07:27
*** arxcruz has joined #openstack-infra07:28
*** hichihara has quit IRC07:28
*** jlvillal has joined #openstack-infra07:29
openstackgerritDmitry Tantsur proposed openstack/requirements: Add ironic-discoverd to projects.txt
*** kashyap` is now known as kashyap07:31
*** jistr has joined #openstack-infra07:34
openstackgerritDavide Guerri proposed openstack-infra/shade: Add Keystone service resource methods
openstackgerritDavide Guerri proposed openstack-infra/shade: Add keystone endpoint resource methods
openstackgerritlifeless proposed openstack-dev/pbr: Bump integration test timeouts.
*** zhiwei has joined #openstack-infra07:37
*** achanda has quit IRC07:38
*** mahito_ has joined #openstack-infra07:41
Kennanhi did we upgrade tox version ?07:42
Kennanit seems the latest tox can cause some unit failure07:42
*** ajo has joined #openstack-infra07:42
*** jlanoux has joined #openstack-infra07:42
Kennanthe new version tox not same as old07:42
*** achanda_ has joined #openstack-infra07:42
Kennandid we decide to use the new tox ?07:42
*** ivar-laz_ has joined #openstack-infra07:43
*** mahito has quit IRC07:44
*** jtomasek has joined #openstack-infra07:45
*** btully has quit IRC07:45
*** ivar-lazzaro has quit IRC07:46
*** geguileo has quit IRC07:48
*** Longgeek has joined #openstack-infra07:49
*** k4n0 has joined #openstack-infra07:51
*** dizquierdo has joined #openstack-infra07:57
*** tnovacik has joined #openstack-infra07:57
lifelessKennan: 2.0 is causing issues? The environment isolations tuff perhaps - do you have a link to a failed log or better yet a bug filed?07:58
lifelessKennan: I don't think an explicit choice was made, we just had an open ended requirement/07:58
SpamapSyeah I was just looking at where we install it07:59
SpamapShard to find07:59
Kennanlifeless: for exmaple08:00
Kennanfor pep8 it use tox 2.0.108:00
*** Longgeek has quit IRC08:00
Kennanbut in
*** freerunner has joined #openstack-infra08:00
Kennanit use tox 1.9.208:01
KennanI don't know how our jenkins env operated08:01
Kennanour local env tox 1.6, it works without issue08:01
SpamapSprobably just new images failing, with old images working08:01
Kennanfor 1.9.2 the logs in
Kennanit failued for unit08:01
*** links has joined #openstack-infra08:02
Kennanlifeless do you know how our jenkins control tox version ?08:02
lifelesswe don't have tox in our global-requirements. An oversight to correct.08:03
lifeless[yes, I know its meta, I don't care :)]08:03
lifelessSpamapS: dvsm-gate probable08:03
SpamapSlifeless: it sits in an odd spot, since tox is used to run allthethings08:04
lifelessSpamapS: sure, but we can document the version we want in the same spot as the rest08:04
lifelesssince it participates in the set f things installed that need coinstallability08:04
SpamapSWouldn't hurt.08:04
lifelessok, notin dv-gate08:04
lifelessin devstack08:05
*** freerunner has quit IRC08:05
*** derekh has joined #openstack-infra08:05
*** freerunner has joined #openstack-infra08:05
lifelesscommit 33dc8699e6b774cc39b39e0dba97be41bbb47abe08:06
lifelessAuthor: Clark Boylan <>08:06
lifelessDate:   Tue Aug 19 22:37:10 2014 -070008:06
lifelessthats when we went from <1.708:06
lifelessto open ended08:06
SpamapSlifeless: it would be worthwhile to have that grep global reqs and just use whatever it finds.08:06
lifelessSpamapS: thats what I'm saying08:06
SpamapScause yeah, open ended just doesn't work out for us08:07
lifelesssure it does08:07
lifelessfor 3-4 days at a time08:07
SpamapSand on holidays08:07
SpamapSwhen nobody is watching or releasing08:07
SpamapSok snark quota reached, time for bed08:08
lifelessKennan: SpamapS: what version works?08:08
lifeless2.0 just came out08:08
SpamapSlifeless: <2.008:08
lifelessI presume 2.0 is wat broke?08:08
SpamapSYeah, I see 1.9 working08:08
Kennanmy local use 1.6 worked, not sure newer. (jenkines worked before, just failured today)08:09
KennanI noticed lifeless08:09
lifelessKennan: failure doesn't look like its related to tox08:09
lifelesswsme.exc.InvalidInput: Invalid input for field/attribute dns_nameserver. Value: ''. Value should be IPv4 format08:09
lifelessSpamapS: do you have a log of a failure that looks tox related?08:09
lifelessoh and now pypi is down.08:10
SpamapSlifeless: it's not super obvious it is tox's fault.. and we need to work on those errors, but OS_* not passing through is the cause of those failures.08:12
SpamapSand other projects do similar things08:12
lifelessSpamapS: ahha08:12
lifelessSpamapS: ok, so the environment thing. Right08:12
SpamapSand I can reproduce on my test vm08:12
KennanOS**__ env sometimes can not get, I also noticed such failure in our env08:12
Kennanit is very strange08:12
lifelessKennan: its in the changelog for tox.08:13
SpamapSso yeah, <2.0 seems prudent08:13
lifelessdoing so now08:13
SpamapSand then we probably need to raise the alarm and get onto 2.0 soonish08:14
lifelessI'll send a mail08:14
* SpamapS stumbles off to bed08:16
lifelessmail sent.08:16
*** teran has quit IRC08:16
lifeless will fix it one hopes08:17
*** teran has joined #openstack-infra08:18
*** HeOS has joined #openstack-infra08:19
*** notnownikki has joined #openstack-infra08:21
megmHi all.  Can someone answer a fairly simple (I hope) git question?08:21
*** freerunner has quit IRC08:22
*** spredzy is now known as spredzy|afk08:23
megmgit config --list includes core.editor=vim08:23
megmbut when I do git commit, I get error: cannot run vim: No such file or directory08:23
megmerror: unable to start editor 'vim'08:23
StevenKmegm: What does 'which vim' print out?08:25
megmStevenK: it just returns -- no output.  I'm running Ubuntu 14.04, BTW08:26
StevenKmegm: Then you don't have vim installed, you should install it by using 'apt-get install vim' as root08:27
StevenKOr Software Centre, etc08:27
megmstevek: really?  That's easy enough -- but I run vim all the time so I'm baffled.08:28
xnoxonly 'vi' is installed by default i believe.08:29
xnoxaka vim-minimal, not the full one....08:29
*** btully has joined #openstack-infra08:29
megmHow interesting!  But thanks so much -- apt-get worked and git commit now puts me into a vim buffer!08:30
*** teran has quit IRC08:31
*** kevinbenton has joined #openstack-infra08:32
xnoxlifeless: troll =)08:32
lifelessxnox: Ahh, you know me08:33
xnoxlifeless: being an ex-canonical myself, you are a legend ;-) and have i been told stories about you ;-)08:34
*** btully has quit IRC08:34
xnoxlifeless: launchpad team is back picking up people by the way. micro-services & git support are developed
*** teran has joined #openstack-infra08:35
*** otter768 has joined #openstack-infra08:35
lifelessxnox: yeah, I'm v happy to see that08:35
lifelessI was advocating for microservices before I joined HP08:36
lifelessand git support was a discussion point back then too08:36
lifelessits nice to see it happening08:36
xnoxyeah. cjwatson is doing it with wgrant.... and you know shallow clone support available in git =) and with insteadof git setting, one can do08:38
xnox$ git clone lp:project08:38
*** btully has joined #openstack-infra08:39
*** otter768 has quit IRC08:40
*** e0ne has quit IRC08:40
*** achanda_ has quit IRC08:40
*** matrohon has joined #openstack-infra08:42
*** btully has quit IRC08:44
*** matrohon has quit IRC08:46
*** SergK_ is now known as SergK08:48
rcarrillocruzhmm, is down?08:48
electricalrcarrillocruz: same for me.08:50 is working for me08:50
electricalvery slow now for me08:51
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Enable Heat services for functional tests
rcarrillocruzyes, it's crawling...08:54
rcarrillocruzi could open some ironic page from, but the front page won't least it's been trying for 3-4 minutes08:54
*** achanda has joined #openstack-infra08:54
*** maishsk_afk has quit IRC08:57
*** dtantsur is now known as dtantsur|bbl09:04
*** mkerrin has joined #openstack-infra09:05
*** ssam2 has joined #openstack-infra09:05
*** jhesketh has quit IRC09:06
*** jhesketh has joined #openstack-infra09:07
*** rlandy has joined #openstack-infra09:07
*** jhesketh has quit IRC09:08
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
*** jhesketh has joined #openstack-infra09:09
johnthetubaguymordred: sorry, I was sleeping when you pinged me, support is the best option at that point.09:11
*** mikal has quit IRC09:11
*** loquacities has quit IRC09:12
*** bnemec has quit IRC09:13
*** bnemec has joined #openstack-infra09:13
*** slagle has quit IRC09:14
*** ivar-laz_ has quit IRC09:14
*** ivar-lazzaro has joined #openstack-infra09:15
*** achanda has quit IRC09:15
*** pelix has joined #openstack-infra09:17
*** ihrachyshka has joined #openstack-infra09:20
*** slagle has joined #openstack-infra09:21
*** achanda has joined #openstack-infra09:22
*** freerunner has joined #openstack-infra09:26
*** ivar-lazzaro has quit IRC09:27
*** ajo has quit IRC09:30
*** mahito_ has quit IRC09:30
*** achanda has quit IRC09:31
openstackgerritRichard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x
*** afazekas has quit IRC09:32
*** freerunner has quit IRC09:33
electricalzaro: as promised:
*** freerunner has joined #openstack-infra09:34
*** dims has joined #openstack-infra09:36
*** Longgeek has joined #openstack-infra09:36
*** dims has quit IRC09:40
*** e0ne has joined #openstack-infra09:44
*** e0ne is now known as e0ne_09:44
*** Longgeek has quit IRC09:44
*** teran has quit IRC09:46
*** mattymo_ is now known as mattymo09:49
*** e0ne_ is now known as e0ne09:52
*** derekh has quit IRC09:55
openstackgerritRichard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x
*** afazekas has joined #openstack-infra09:58
*** ivar-lazzaro has joined #openstack-infra09:58
*** ivar-lazzaro has quit IRC10:03
openstackgerritLuis Cañas-Díaz proposed openstack-infra/activity-board: Update JS library and company panel to add top people by company
*** dims has joined #openstack-infra10:07
*** weshay has quit IRC10:07
*** cdent has joined #openstack-infra10:08
*** weshay has joined #openstack-infra10:09
*** amotoki has quit IRC10:10
*** dguerri is now known as _dguerri10:11
*** enikanorov2 has quit IRC10:15
*** _dguerri is now known as dguerri10:15
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Switch on docs and publish jobs for magnum
*** ajo has joined #openstack-infra10:16
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Switch on docs and publish jobs for magnum
*** kevinbenton has quit IRC10:25
*** kevinbenton has joined #openstack-infra10:26
*** sdake has joined #openstack-infra10:34
*** otter768 has joined #openstack-infra10:36
*** teran has joined #openstack-infra10:37
*** ppai has quit IRC10:39
*** teran_ has joined #openstack-infra10:40
*** otter768 has quit IRC10:40
*** yfried is now known as yfried|afk10:42
*** zhiwei has quit IRC10:43
*** teran has quit IRC10:43
*** ihrachyshka_ has joined #openstack-infra10:46
*** dguerri is now known as _dguerri10:47
*** ihrachyshka has quit IRC10:49
electricalfungi: pelix yolanda not sure if any of you can have a look at  test is failing of a part i didn't touch so it must have been failing before ?10:49
*** abregman_ has quit IRC10:50
pelixelectrical: you need to remove the use of decode from the tests10:50
*** yamamoto has quit IRC10:51
pelixIn fixing the code you've broken the tests, but then the tests were wrong anyway, in that they shouldn't have been using decode10:51
pelixthat should have been done by the code they were testing10:51
electricalAh okay. i only see 1 event of using decode in the tests10:52
sdaguelifeless: there isn't much meat on your email or the review and lots of stuff is passing, so "gate wedged" needs to be more specific as to what jobs seem to be impacted10:52
*** ppai has joined #openstack-infra10:52
electricalpelix: will remove it and re-submit10:53
electricalthanks for the quick check :-)10:53
pelixelectrical: np10:56
openstackgerritRichard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x
electricalpelix: ^^ that should solve it then i hope :-)10:57
*** jamielennox is now known as jamielennox|away10:58
*** ppai has quit IRC10:58
*** dizquierdo has quit IRC10:58
*** ivar-lazzaro has joined #openstack-infra10:59
sambettsHey, is there anyway to clean up the CI comments on a patch? A couple of weeks ago, A10 networks CI went baserk and left hundreds of comments on a couple of my patchs which makes loading the review page for those patches really slow, I think because of the JS processing the comments into that nice box, any ideas how to fix this or is it just going to have to be like that until that patch merges?10:59
*** sdake_ has joined #openstack-infra10:59
*** abregman_ has joined #openstack-infra11:02
*** sdake has quit IRC11:02
*** dtantsur|bbl is now known as dtantsur11:03
*** ivar-lazzaro has quit IRC11:04
pelixelectrical: think it might have been my mistake, it's the encode calls that need to be looked at11:04
*** mwagner_lap has joined #openstack-infra11:05
*** weshay has quit IRC11:06
pelixelectrical: ah, no I spotted the problem, there were additional decode calls being made in the python jenkins, in decoding all responses before returning them, needed to remove extraneous decodes from elsewhere.11:06
electricalpelix: ahhh okay. makes sense since we have now a single place doing it11:06
electricalpelix: removing it from the test was still a good move or do i need to place it back?11:07
sdaguesambetts: no, deleting comments isn't really a thing11:08
pelixI think that one needs to be put back11:08
electricalpelix: okay11:08
pelixsince it's inspecting the attribute directly rather than the response11:09
electricalpelix: i'm having quite some problems running the tests my self. using pyenv to select the python version. running tox fails continuously for me locally11:09
electricaleven running python testr --slowest doesn't work11:10
pelixelectrical: I'm taking a look at it now as I have 3.4 and 2.7 installed via my system ;)11:10
sambettssdague: If you want to see what I mean check out this patch toggle CI if you have to, to see all the comments and then scroll down, its insane and shame I can't get it cleaned up11:10
electricalpelix: ah okay thanks :-)11:11
sdaguesambetts: right, I'm aware what happens, but deleting comments isn't really a thing as far as I know11:11
*** derekh has joined #openstack-infra11:11
*** ppai has joined #openstack-infra11:11
sambettssdague: :( ah well11:12
*** e0ne is now known as e0ne_11:12
pelixthe places that need to be changed are the removal of encode of the json.dumps(data) on the test_jenkins_open test, add back in the decode on the j.auth, and removal of the decode in the maybe_add_crumb11:12
*** pelix has quit IRC11:13
*** panda has quit IRC11:13
*** pelix has joined #openstack-infra11:14
*** deepakcs has quit IRC11:14
*** panda has joined #openstack-infra11:14
*** loquacities has joined #openstack-infra11:15
*** e0ne_ is now known as e0ne11:19
*** ihrachyshka_ has quit IRC11:20
openstackgerritRichard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x
electricalpelix: ^^ that should be the final fix i hope :-)11:23
*** yfried|afk is now known as yfried11:24
*** banix has joined #openstack-infra11:25
*** woodster_ has joined #openstack-infra11:26
*** yamamoto has joined #openstack-infra11:26
*** banix has quit IRC11:27
*** banix has joined #openstack-infra11:27
electricalpelix: an other error now :-(11:28
*** doug-fish has joined #openstack-infra11:28
*** Longgeek has joined #openstack-infra11:29
*** ihrachyshka_ has joined #openstack-infra11:32
electricalpelix: managed to get test runs to work. fixed the issue. pushing new patch11:33
openstackgerritRichard Pijnenburg proposed stackforge/python-jenkins: Add decode to utf8 for python 3.x
*** ldnunes has joined #openstack-infra11:35
*** banix has quit IRC11:35
electricalpelix: finally a +1 from jenkins :D11:38
*** lil_cain has joined #openstack-infra11:39
*** fhubik has joined #openstack-infra11:39
lil_cainHey - I have some jenkins failures - is there an easy way to tell what platform/pypy version jenkins is running the tests on?11:41
*** weshay has joined #openstack-infra11:44
*** chlong has joined #openstack-infra11:48
*** soren_ has joined #openstack-infra11:49
*** baoli has joined #openstack-infra11:50
*** freerunner has quit IRC11:52
*** luqas__ has quit IRC11:52
*** pc_m has joined #openstack-infra11:52
*** ssam2 has left #openstack-infra11:55
*** _dguerri is now known as dguerri11:58
*** mwagner_lap has quit IRC11:58
*** devvesa has quit IRC11:59
*** ivar-lazzaro has joined #openstack-infra11:59
*** freerunner has joined #openstack-infra12:00
openstackgerritVictor Seva proposed openstack-infra/jenkins-job-builder: Allow use of aliases defined previously inside included files
*** Longgeek has quit IRC12:01
openstackgerritRuslan Kamaldinov proposed openstack-infra/project-config: Mark apps-catalog py34 job as non-voting
*** dizquierdo has joined #openstack-infra12:01
*** ivar-lazzaro has quit IRC12:04
linuxmaniaczaro: Thanks for the review. I've reduced the xml produced as you asked.12:05
*** e0ne is now known as e0ne_12:05
*** e0ne_ is now known as e0ne12:06
*** zhiwei has joined #openstack-infra12:06
*** ihrachyshka_ has quit IRC12:08
*** gordc has joined #openstack-infra12:10
*** devvesa has joined #openstack-infra12:11
*** claudiub has joined #openstack-infra12:13
*** claudiub|2 has joined #openstack-infra12:14
*** dprince has joined #openstack-infra12:16
*** claudiub|3 has joined #openstack-infra12:17
*** claudiub_ has joined #openstack-infra12:18
*** claudiub has quit IRC12:18
*** claudiub has joined #openstack-infra12:18
*** rfolco has joined #openstack-infra12:18
*** claudiub|2 has quit IRC12:19
*** claudiub|2 has joined #openstack-infra12:20
*** claudiub|3 has quit IRC12:21
*** claudiub|3 has joined #openstack-infra12:22
*** yamamoto has quit IRC12:22
*** coolsvap is now known as coolsvap|afk12:22
*** claudiub|4 has joined #openstack-infra12:23
*** claudiub has quit IRC12:23
*** claudiub has joined #openstack-infra12:24
*** ihrachyshka_ has joined #openstack-infra12:24
*** kgiusti has joined #openstack-infra12:24
*** claudiub|2 has quit IRC12:25
*** BharatK has quit IRC12:26
*** claudiub|3 has quit IRC12:26
*** claudiub|4 has quit IRC12:27
*** bhunter71 has joined #openstack-infra12:29
*** yamamoto has joined #openstack-infra12:31
*** markvoelker has quit IRC12:31
*** markvoelker has joined #openstack-infra12:31
*** claudiub|2 has joined #openstack-infra12:32
*** zz_dimtruck is now known as dimtruck12:33
*** claudiub_ has quit IRC12:34
*** kadm__ has joined #openstack-infra12:34
*** claudiub has quit IRC12:34
*** BharatK has joined #openstack-infra12:35
*** claudiub|2 has quit IRC12:36
*** kadm__ has quit IRC12:36
*** kadm__ has joined #openstack-infra12:37
*** otter768 has joined #openstack-infra12:37
electricalpelix: could i pick your brains sometime on how i should template the jenkins yaml files for jjb in the best way? I have allot of jobs for the same 'product' ( puppet module ) and each product has the same set of jobs. there are also 3 jobs that are dependent on each other ( 1 master job with 2 jobs that get triggered of it12:38
*** erlon has joined #openstack-infra12:38
*** stevemar has joined #openstack-infra12:38
*** yfried is now known as yfried|afk12:40
*** yfried|afk is now known as yfried12:41
*** otter768 has quit IRC12:41
*** bknudson has quit IRC12:42
*** sdake_ has quit IRC12:43
*** kadm__ has quit IRC12:44
*** sdake has joined #openstack-infra12:44
*** ihrachyshka_ has quit IRC12:46
*** mwagner_lap has joined #openstack-infra12:46
*** bswartz has quit IRC12:46
*** aysyanne has joined #openstack-infra12:48
*** e0ne is now known as e0ne_12:51
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Add glusterfs/centos7 experimental job
*** abregman_ has quit IRC12:52
*** e0ne_ is now known as e0ne12:53
*** yfried has quit IRC12:56
*** gordc has quit IRC12:56
*** kadm__ has joined #openstack-infra12:56
*** ihrachyshka_ has joined #openstack-infra12:57
*** radez_g0n3 is now known as radez12:59
*** dguerri is now known as _dguerri12:59
*** sarob has joined #openstack-infra12:59
*** ivar-lazzaro has joined #openstack-infra13:00
*** _dguerri is now known as dguerri13:01
*** signed8bit_ZZZzz is now known as signed8bit13:02
*** claudiub has joined #openstack-infra13:03
*** heyongli has joined #openstack-infra13:03
*** kadm__ has quit IRC13:03
heyonglianteaya, ping13:04
*** ppai has quit IRC13:05
*** kadm__ has joined #openstack-infra13:05
*** ivar-lazzaro has quit IRC13:05
*** abregman_ has joined #openstack-infra13:05
*** afazekas has quit IRC13:05
*** derekh_ has joined #openstack-infra13:07
*** kadm__ has quit IRC13:07
*** mestery has joined #openstack-infra13:08
*** afazekas has joined #openstack-infra13:08
electricalfungi: morning :-)13:08
*** ociuhandu has joined #openstack-infra13:09
fungilil_cain: the pypy 2.2.1+dfsg-1ubuntu0.2 package provided in the ubuntu 14.04 lts amd64 updates repo (trusty-updates)13:09
fungielectrical: good afternoon(?) to you!13:09
lil_cainfungi: Deadly, thanks13:09
*** annegentle has joined #openstack-infra13:10
electricalfungi: hehe yeah. 2 pm13:10
*** derekh has quit IRC13:10
*** dkranz has joined #openstack-infra13:11
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Switch on docs and publish jobs for magnum
*** devvesa has quit IRC13:14
*** alaski has quit IRC13:14
*** Longgeek has joined #openstack-infra13:15
*** miqui has joined #openstack-infra13:16
*** alaski has joined #openstack-infra13:16
*** ppai has joined #openstack-infra13:16
*** bswartz has joined #openstack-infra13:18
*** peristeri has joined #openstack-infra13:18
sdaguefungi: has rax scheduled the world reboot yet?13:21
fungisdague: we did it yesterday13:21
sdagueoh, gotcha13:22
sdaguecool :)13:22
fungithey sent notifications to everyone to perform hard reboots of all hvpvm servers within 24 hours or they would do it for us13:22
*** ddieterly has joined #openstack-infra13:22
*** soren_ has quit IRC13:22
fungisp we engaged in that pleasant scramble yesterday evening13:22
*** ppai has quit IRC13:23
fungithough i'm marginally worried... for my personal servers i did the same, and rackspace keeps opening new tickets reminding me to reboot them even though i responded in the originals that i had done so and then closed them13:23
*** dboik_ has quit IRC13:23
electricalfungi: I'm working on a new module for jjb. logstash reporter. almost done i think13:23
fungielectrical: neat. we don't really use any jenkins plugins except for the couple which we have as glue into zuul and nodepool (gearman and zeromq plugins respectively), but it's nice to see a healthy ecosystem plugin support layer growing in jjb13:24
*** changbl has quit IRC13:25
annegentlefungi: yuck, I'll chase that.13:25
electricalfungi: hehe yeah indeed. I'll most likely only build the ones I'm using :-)13:25
*** bhunter71 has quit IRC13:25
fungiannegentle: the rackspace tickets? yeah makes me wonder if those are just repeats from their original list of affected servers, or if they really don't know whether anyone has rebooted anything13:26
annegentlefungi: right. I have more emails from tickets in my inbox this morning.13:26
mtreinishfungi: heh, I closed my rax account last week since it won't be free anymore13:26
fungiand are just going to reboot them all again even though we already did13:26
mtreinishso don't have to worry about reboots :)13:27
annegentlemtreinish: heh, this one's for all the clouds tho13:27
*** heyongli has quit IRC13:27
fungiannegentle: righth, for my personal servers alone i think i got 20 rackspace ticket notification e-mails in the last 24 hours (as i have instances in iad and syd and they treated them separately)13:27
annegentlefungi: I don't know much about the automation13:27
annegentlefungi: right. Our customers hate all the ticket flood.13:27
*** freerunner has quit IRC13:28
mtreinishannegentle: yeah, I saw the cve, but rax was the only cloud I was really using. I have my hp cloud account but I don't really use it for much13:28
electricalfungi: jenkins_jobs.errors.JenkinsJobsException: Unknown entry point or macro 'logstash' for component type: 'reporter'.  any idea? i created the logstash def in reporters but test fails.13:28
fungielectrical: is it in the entrypoints list? have you looked at a recent plugin addition commit for ideas?13:30
electricalfungi: haven't checked to be very honest. my fault.13:31
*** prad has quit IRC13:31
*** prad has joined #openstack-infra13:31
annegentlemtreinish: workin' for cloud creds :)13:31
electricalAh, i see what i'm missing13:32
fungielectrical: something like might be useful to review13:32
fungielectrical: so wondering if it's in setup.cfg13:33
electricalfungi: it wasn't no.13:33
electricalthat error is gone now. only got a discrepency between the yaml and xml file13:33
*** gordc has joined #openstack-infra13:33
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras and env markers
*** jgrimm is now known as zz_jgrimm13:35
linuxmaniaczaro: Thanks! :-)13:35
mordredfungi: so - fwiw, tox 2.0 broke shade functional tests13:35
*** enikanorov2 has joined #openstack-infra13:36
fungimordred: so i heard. started filtering the calling env13:36
mordredit's fine - I want that to be consuming a clouds.yaml input file _anyway_13:36
*** dustins has joined #openstack-infra13:36
mordredbut I was ignoring doing the work for that since the env var hack was working13:36
fungilong-term i think i like that change, since it's nice to be explicit about what parts of tox's calling environment should be seen in the test envs13:37
mtreinishmordred: fwiw, I think you should be able to just do passenv = * and everything will work again13:37
mordredShrews: ^^13:37
mtreinishannegentle: heh, yeah something like that :)13:37
mordredShrews: that might be an answer for the short term13:37
fungiyep, that's what keystone ended up doing13:37
mordredfungi: greghaynes had the idea yesterday that was similar to one I'd also had - which is to add a command line program to os-client-config that will take OS_ envvars from the environment and write them into a clouds.yaml file for you13:38
mordredfungi: so you could say "source foo.env ; os-client-config add-cloud devstack"13:38
mtreinishI've got the tempest review up, although I figured I'd try to limit it to the vars that are being used:
*** mriedem_away is now known as mriedem13:38
Shrewsmordred: oh, that's a neat idea13:38
mordredShrews: yah. should make it easy for people to consume new cloud accounts13:38
fungiit's also worth noting that tox's current documentation still mentions the {envbindir} parameter as supported, but 2.0 seems to have broken/dropped it. bknudson opened a bug report for holger about that13:38
mordredbtw - ec2 and google each have a command line tool you use for setting your local credentials13:39
mordredShrews: we shoudl also honestly make it able to take an env vars file on the command line too - because needing to source them is a little lame13:40
fungithat might be useful is os-cloud-config, maybe presented by openstackclient when it's missing credentials?13:40
fungier, s/is/in/13:40
*** dguerri is now known as _dguerri13:40
mordredfungi: there it could be "openstack cloud add" or something13:40
mordreddtroyer: ^^13:40
fungi<clippy>it looks like you're trying to openstack</clippy>13:40
* Shrews stabs fungi's clippy13:40
mordredShrews: we should also convince david-lyle to add "download a clouds.yaml snippet" to horizon13:41
*** bknudson has joined #openstack-infra13:41
mordredShrews: so that one of the things your horizon can do for you is give you a clouds.yaml for your cloud account - and os-client-config should then be able to merge that in to your current clouds.yaml locally13:41
fungithat's a helpful entry point for newbie users too, agreed13:42
*** pabelanger has joined #openstack-infra13:42
*** pabelanger has joined #openstack-infra13:42
dtroyeradding that to osc should be simple…13:42
dtroyerI'd rather see it there than in a library13:42
*** claudiub has quit IRC13:42
*** arielb has joined #openstack-infra13:43
*** arielb has quit IRC13:43
*** arielb has joined #openstack-infra13:43
Shrewsmordred: so not just env vars merge, but also separate clouds.yaml merges?13:43
fungiwhichever. the only reason i was thinking of putting the guts in a lib is so that more apps than openstackclient could reuse the same config writing backend13:43
*** _dguerri is now known as dguerri13:43
mordred"I have this cloud defined in this clouds.yaml, please add it to my others"13:43
dtroyerguts, sure, not cli13:43
fungiagreed. the cli for it goes in the client. full stop13:43
mordredguts -> o-c-c, cli -> osc13:43
*** dboik has joined #openstack-infra13:46
*** e0ne is now known as e0ne_13:47
*** Longgeek has quit IRC13:48
sdaguemordred: so I approached the python-novaclient functional tests differently, and just put stuff in a file13:48
sdaguemtreinish: so... the tempest tests are running without that, we just loosing some safety net?13:49
mtreinishsdague: yeah, the gate uses defaults anyway or just overrides things in the tox.ini13:50
mtreinishit's really for people using the options outside of the gate13:50
openstackgerritRichard Pijnenburg proposed openstack-infra/jenkins-job-builder: Add logstash reporter
mordredsdague: so ... just for sake of argument13:50
sdagueso... for outside of the gate, that seems like those should go in the config instead13:50
*** freerunner has joined #openstack-infra13:51
mordredsdague: why not use o-c-c?13:51
mordredand/or would you accept such a patch?13:51
electricalfungi: would love your feedback on it.13:51
sdaguemordred: mostly because I'm not sure exactly what that would look like13:51
mtreinishsdague: it's mostly for the telling tempest where the config file is, the only way we could pass that in was via an env var13:51
sdaguemtreinish: right, that one, fine13:51
sdagueuntil there is a real cli13:52
sdaguebut the rest?13:52
mordredsdague: ok. how about I make you a patch and you will like it or not13:52
sdaguemordred: that sounds completely reasonble to me13:52
mtreinishsdague: the rest are triggers for fixtures, I mean I guess we can move that into the tempest config13:52
mordredsdague: do you mind if I hijack that patch? or would you rather I do a followup one?13:53
mtreinishoh, and 1 is to set the discovery path (but tox should own that anyway, so we can run unit tests)13:53
sdaguemordred: follow up please13:53
sdagueso that we can unwedge for now13:53
mordredsdague: ok13:54
*** freerunner has quit IRC13:54
sdagueand contemplate better13:54
sdagueif you stack it on top, that would be cool13:54
mordredI shall do that13:54
*** freerunner has joined #openstack-infra13:54
mordredI'm also going to write the devstack patch I keep talking about about outputting a clouds.yaml file in the first place13:54
*** freerunner has quit IRC13:54
sdaguemordred: cool13:54
mordrednow that we have more than one thing that know how to read them13:54
mordredactually, I'll do that first13:55
*** ddieterl_ has joined #openstack-infra13:55
sdagueanyone know if we're actually tracking a bug here for the tox thing?13:55
mordredand then make the novaclient change make use of it13:55
*** mestery has quit IRC13:55
mtreinishsdague: I haven't seen one yet13:55
*** pabelanger has quit IRC13:55
mtreinishbut I didn't look too hard13:55
sdaguehey, lets do that then13:56
mordredsdague: I don't suppose you'd entertain a patch to s/OS_TENANT_NAME/OS_PROJECT_NAME/13:57
*** pabelanger has joined #openstack-infra13:57
*** ddieterly has quit IRC13:57
sdaguemordred: you can overload it13:57
mordredI know13:57
mtreinishit'll always be tenant to me :)13:57
sdaguebut we'll need a deprecation cycle13:57
mordredI was just reading a comment that says tenant is the new "right" thing13:58
sdaguebecause people are going to have stuff hardcoded13:58
sdagueand figure out how to warn them13:58
mordredand it was the comment I was more concerned with, since that's no longer accurate13:58
sdaguecomment changes are fine13:58
mordredand if they're learning by reading comments, we're perpetuating things13:58
fungii don't know of any one specific bug. i expect that any bugs open about this are particular to the different ways in which tox behavior has changed to impact various repos/jobs13:58
mordredI mean, I probably won't do that today13:58
sdagueI'll file one13:58
*** mtanino has joined #openstack-infra14:00
*** baoli has quit IRC14:00
openstackLaunchpad bug 1455102 in OpenStack-Gate "some test jobs broken by tox 2.0 not passing env variables" [Undecided,Confirmed]14:00
*** baoli has joined #openstack-infra14:01
*** ivar-lazzaro has joined #openstack-infra14:01
*** dboik_ has joined #openstack-infra14:02
openstackgerritMerged openstack-infra/system-config: Move afs servers to using o_p::template
*** tiswanso has joined #openstack-infra14:02
*** btully has joined #openstack-infra14:02
*** gokrokve has joined #openstack-infra14:03
funginibalizer: that ^ merged. i'll try to make sure it's puppeting successfully14:03
*** dkranz has quit IRC14:04
*** sigmavirus24_awa is now known as sigmavirus2414:05
*** dboik has quit IRC14:06
*** e0ne_ is now known as e0ne14:06
*** ivar-lazzaro has quit IRC14:06
*** rossella_s has quit IRC14:07
*** rossella_s has joined #openstack-infra14:07
mordredfungi: woot!14:08
*** sputnik13 has joined #openstack-infra14:08
mordredsdague: ok. so - for devstack->clouds.yaml - it seems like I should either make a new script that gets called aroudn the same time as create_userrrc - or I should make create_userrc do it since it already knows the things14:10
mordredsdague, dtroyer: ^^ do you have preference?14:10
*** mattfarina has joined #openstack-infra14:10
*** cody-somerville has quit IRC14:10
*** ildikov has quit IRC14:10
mordredalthough it might just want to be a different thing, because it wants to know some of the things that go into .stackenv too14:11
*** sarob has quit IRC14:11
mordredsdague, dtroyer: nevermind - let me play a while longer and come back and ask questions14:11
dtroyerI was thinking it would only have one cloud in it… were you thinking a distinct entry for each user account?14:12
*** markvoelker has quit IRC14:13
openstackgerritMonty Taylor proposed openstack-infra/puppet-nodepool: Consume puppet-diskimage_builder
mordreddtroyer: unsure?14:14
mordreddtroyer: I was originally thinking one cloud too - but the user account thing is an interesting one14:14
fungii am interested in support for multiple user accounts in one tenant, fwiw. that is actually my personal workflow for my systems14:17
fungii have a low-privilege user with read-only access to the api and a high-privilege one i use for creating/deleting servers, et cetera14:17
mtreinishmordred, dtroyer: something like this:
fungiso i switch between credentials depending on what calls i'm initiating, as a safeguard14:18
mordredfungi: yup. that's kinda what o-c-c is there for - although each one would be a "cloud"14:18
mordredbut yeah - named sets of cloud credentials14:18
fungioh, got it14:18
mordredmtreinish: I should port that to use o-c-c too ...14:18
mordredmtreinish: just kidding - that's a bit different usecase14:18
dtroyerbtw, this is also why I want to keep the ability to load a cloud and override individual things like USERNAME from the env14:19
mordredgiven the roles14:19
fungiyeah, that serves my purposes. i should get around to using that. at the moment i just have scripts sourcing the appropriate credentials, prompting me to decrypt the corresponding password and passing them into openstackclient14:19
mtreinishmordred: heh, yeah jamielennox|away brought that up before. They're similar but different enough14:19
mordreddtroyer: yeah - loading a cloud and overriding from the env gets _REALLY_ confusing for the common case though14:19
mtreinishmordred: yeah roles, and optionally specifying a network14:19
*** unicell1 has joined #openstack-infra14:20
*** unicell has quit IRC14:20
*** dkranz has joined #openstack-infra14:20
*** claudiub has joined #openstack-infra14:21
*** jlanoux_ has joined #openstack-infra14:26
*** ildikov has joined #openstack-infra14:26
*** fhubik has quit IRC14:26
*** akshai has joined #openstack-infra14:26
*** k4n0 has quit IRC14:26
*** fhubik has joined #openstack-infra14:26
*** mkerrin has quit IRC14:27
*** aswadr has joined #openstack-infra14:27
*** jlanoux has quit IRC14:28
openstackgerritMerged openstack-infra/release-tools: Add
openstackgerritMerged openstack-infra/release-tools: Add
openstackgerritMerged openstack-infra/release-tools: Add
openstackgerritMerged openstack-infra/release-tools: Move get_last_tag to functions
openstackgerritMerged openstack-infra/release-tools: Print release notes as part of releasing the library
*** eharney has joined #openstack-infra14:30
wznoinskhi, is anyone here?14:31
*** achanda has joined #openstack-infra14:33
mordredwznoinsk: there are over 300 people here ;)14:33
wznoinskI'm wondering is there an issue in gerrit, i.e.: doesn't have Jenkins and many other CI's in the CI isummary table at the top14:34
*** smarcet has joined #openstack-infra14:34
fungimy irc client claims 429, but some of those are bots14:34
*** claudiub has quit IRC14:34
openstackgerritMerged openstack-infra/release-tools: Add script to update .gitreview files
openstackgerritMerged openstack-infra/release-tools: Submit .gitreview to the right branch
fungiwznoinsk: have you checked to see if it's in the process of being tested still?14:35
wznoinsksorry, bad example, give me a sec14:36
fungiwznoinsk: looks like it's been in the check pipeline for 19 minutes, seems to be passing all jobs which have completed so far, and has an estimated 35 minutes to report14:36
*** toabctl has joined #openstack-infra14:37
*** tnovacik has quit IRC14:37
*** fhubik has quit IRC14:38
*** Shrews has quit IRC14:38
*** fhubik has joined #openstack-infra14:39
*** gilliard has quit IRC14:39
*** markvoelker has joined #openstack-infra14:40
*** mordred has quit IRC14:40
*** nelsnelson has joined #openstack-infra14:40
wznoinskfungi, yes, you're right, didn't realize there was another patchset already, no issue then14:41
*** gilliard has joined #openstack-infra14:41
*** fhubik is now known as fhubik_afk14:41
*** mordred has joined #openstack-infra14:42
*** ryanpetrello has quit IRC14:42
*** achanda has quit IRC14:42
*** Shrews has joined #openstack-infra14:43
*** ryanpetrello_ has joined #openstack-infra14:43
*** ddieterl_ has quit IRC14:43
*** bhunter71 has joined #openstack-infra14:45
*** Longgeek has joined #openstack-infra14:45
*** poconosguy has joined #openstack-infra14:46
*** poconosguy has quit IRC14:46
*** dguerri is now known as _dguerri14:48
*** _dguerri is now known as dguerri14:48
*** shardy_ has joined #openstack-infra14:48
*** dguerri is now known as _dguerri14:49
*** _dguerri is now known as dguerri14:50
*** zhiwei has quit IRC14:50
*** zz_jgrimm is now known as jgrimm14:50
*** shardy has quit IRC14:50
mordredsdague: so - create_userrc doesn't seem to honor all the same things as the things that wind up being set if you source openrc14:51
mordredsdague: for instance, identity api14:51
mordredsdague: for instance, identity api version14:51
sdaguemordred: that's entirely possible, I'm actually complete oblivious to the create_userrc path14:51
sdagueI've only ever used the openrc one14:51
mordredso openrc does not interact with the data written by create_userrc then?14:51
*** freerunner has joined #openstack-infra14:52
sdaguedtroyer: ^^^ ??14:52
*** ddieterly has joined #openstack-infra14:52
dtroyernope.  I don't recall who contributed create_userrc but we've never used it directly14:52
*** fhubik_afk is now known as fhubik14:53
*** shardy_ has quit IRC14:54
mordredso I should fairly much ignore it14:54
sdagueyeh, I think that's best14:54
*** shardy has joined #openstack-infra14:54
jeblairdtroyer, sdague: I refreshed to remove the trove bits from devstack14:55
*** cody-somerville has joined #openstack-infra14:55
*** cody-somerville has quit IRC14:55
*** cody-somerville has joined #openstack-infra14:55
*** armax has joined #openstack-infra14:55
*** yamahata has joined #openstack-infra14:55
sdaguemordred: how are you expecting to handle password in the general case14:55
sdaguejeblair: cool, is the project-config change landed?14:55
mordredsdague: I'm not sure I understand your question?14:55
sdagueI kind of lost track with the depends on spider14:55
sdaguemordred: so.... whatever this is, needs the password for the account14:56
*** fawadkhaliq has joined #openstack-infra14:56
sdaguewhich seems to conflict with a previous statement of "have horizon export you a working one"14:56
mordredoh - so - sorry - two different things there14:56
*** ryanpetrello_ has quit IRC14:57
jeblairsdague: the only non-merged changes are: 167447 - remove trove from devstack <- 168145 - Enable trove plugin for trove tempest job14:57
jeblairgah one more14:57
*** ryanpetrello_ has joined #openstack-infra14:57
mordredsdague: I imagine that for the import-thing-exported-by-horizon that we could have it prompt for a password if there is not one in the config - or maybe if there is a "prompt_for_password" flag in the exported config14:58
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for heat tox 2.0 bug 1455065
jeblair<- 168146 Revert "Temporarily add trove plugin configuration"14:58
openstackbug 1455065 in heat "New tox version broke integration tests" [Critical,In progress] - Assigned to Sirushti Murugesan (sirushtim)14:58
mordred(becaue it's possible the cloud may not use password auth, in which case prompting on missing password would be a mistake)14:58
*** ryanpetrello_ is now known as ryanpetrello14:58
sdaguemordred: ok, cool14:58
*** fhubik has quit IRC14:58
mordredsdague: for devstack, I intend to write the password to the file :)14:59
sdaguemordred: yep, for devstack that's fine14:59
sdagueI was mixing with the other case14:59
sdaguejeblair: ok, cool, I'm +2 on the remove, will let dtroyer +A14:59
jeblairsdague: ^ so 3 patches; we're using the new system already because of the temporary patch to devstack-gate (so it's ignoring what's set in project-config); once the devstack cleanup lands, we can merge the project-config change to set that explicitly, then remove the temporary patch to devstack-gate14:59
sdaguejeblair: great14:59
electricalpelix: fungi is there a specific reason that reporters are only available in maven2-moduleset type jobs? and not freestyle for example15:01
*** dangers_away is now known as dangers15:01
*** achanda has joined #openstack-infra15:01
*** ryanpetrello has quit IRC15:01
*** ivar-lazzaro has joined #openstack-infra15:02
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for neutron/manila tox 2.0 bug 1455102
openstackbug 1455102 in python-novaclient "some test jobs broken by tox 2.0 not passing env variables" [High,In progress] - Assigned to Sean Dague (sdague)15:02
*** ryanpetrello has joined #openstack-infra15:02
*** sks has quit IRC15:04
*** sergsh has quit IRC15:04
*** Longgeek has quit IRC15:04
sdaguefungi: so we had another burst on -
*** Longgeek_ has joined #openstack-infra15:05
*** Longgeek_ has quit IRC15:05
*** achanda has quit IRC15:05
sdaguewere there any other proposed fixes to deal with git hanging up on us?15:05
*** Longgeek has joined #openstack-infra15:05
*** tiswanso has quit IRC15:06
*** dboik_ has quit IRC15:06
*** baoli has quit IRC15:06
*** ivar-lazzaro has quit IRC15:06
jeblairsdague, fungi:
fungisdague: i'm not sure that's an accurate representation15:07
jeblairpulled a random from there ^15:07
fungisdague: looks like failures for other reasons are clouding that count15:07
sdaguefungi: ok15:07
fungisdague: that error appears if we retry15:07
jeblairdns resolution failure, followed by git failure.... looks like maybe abject network failure15:07
fungisdague: then if the job fails for another reason, e-r thinks that was the cause15:07
sdaguefungi: ok, so we need to update our signature15:08
jeblairand yeah, git eventually succeeded15:08
*** tiswanso_ has joined #openstack-infra15:08
*** dboik has joined #openstack-infra15:08
fungiin that example, "E303 too many blank lines (3)"15:08
*** dustins_ has joined #openstack-infra15:08
mordreddevstack's support for keystone v3 seems ... not really a thing15:09
mordredlike, I see no mention of USER_DOMAIN or PROJECT_DOMAIN anywhere15:10
mordredand v2.0 is hardcoded in many places15:10
mordredso, I'm guessing I should not worry too much about that for now?15:11
sdakemordred would you mind taking a look at
*** tnovacik has joined #openstack-infra15:11
sdagueyeh, interestingly, if we weren't retrying, this would have failed15:11
*** akshai has quit IRC15:11
sdagueso... that's good15:11
sdakeand jaeger too if hes awake ;)15:12
mordredsdake: it is marked WIP15:12
mriedemi just hit in the gate15:12
openstackLaunchpad bug 1383403 in OpenStack Core Infrastructure "fatal: unable to access '': gnutls_handshake() failed: A TLS packet with unexpected length was received." [Medium,Fix committed]15:12
*** dustins has quit IRC15:12
*** dguerri is now known as _dguerri15:12
*** _dguerri is now known as dguerri15:12
sdakesec mordred15:12
fungielectrical: not really sure why they're maven only to be honest. hopefully pelix or zaro know15:12
*** dboik has quit IRC15:13
*** tiswanso_ has quit IRC15:13
*** craigbr has joined #openstack-infra15:13
sdaguemriedem: now that retry logic is in there - seems no longer specific enough, we might need to prune that back15:14
*** arxcruz has quit IRC15:14
mriedemsdague: yes, i'm working on that15:14
*** bswartz has quit IRC15:14
fungisdague: mriedem: though also looks like devstack-gate doesn't retry `git remote prune origin` calls15:14
mriedemsdague: alas, i have to run to get a haircut b/c i need to look beautiful for the summit15:14
dtroyermordred: devstack and v3 are incubating…the first flag to turn off v2 is about to merge, morgan is working on the rest of a v2-free setup15:14
openstackgerritDebo~ Dutta proposed openstack-infra/project-config: Cognitive - Machine Learning as a Service
mordreddtroyer: also, there seems to be no support for glance v2, is that right?15:15
mordredI mean, Im asking because there seems to be support for nova and cinder api version switching, even though nova and cinder do  not have more than one version, and there is no support for switching versions for keystone and glance which do ... so it's a bit wat?15:15
dtroyerin devstack?  I don't recall anything explicit either way, just glance's defaults15:16
mriedemfungi: i can add a retry loop for git remote prune origin15:16
sdakeok mordred debo fixed that up15:16
*** bswartz has joined #openstack-infra15:16
mordreddtroyer: awesome15:17
sdaguemordred: so... basically that's a reflection of the involvment rate of the various teams15:17
sdaguecinder does have more than one version15:18
nfedotovHello all! For some reason it is impossible to clone repos from via git:// protocol " Connection reset by peer". Does somebody experiences similar issue?15:18
mordredsdague: really?15:18
mordredsdague: well, good on them for doing it right, beause so far it's never caused me pain :)15:19
sdagueyeh, v1 isn't yet removed. openstackclient doesn't actually work with cinder v2 api yet15:19
BobBallWhat's the easiest way to stop executing jobs in the zuul/jenkins setup while continuing to queue jobs to run?15:19
mordredShrews, greghaynes: also - see scrollback with me and sdage a little bit ago about passwords and importing files from horizon15:20
fungiBobBall: you can put your jenkins into prepare-for-shutdown15:20
BobBallOK - wasn't sure if that would break other things :) Thanks,15:20
greghaynesmordred: yea, there was convo late last night about tox2 being to blame15:20
fungiBobBall: all running jobs will complete, no new jobs will run, zuul will queue them up waiting for jenkins to come back online15:20
sdaguefungi: hmmm... there is some spiking -
sdaguethe git server at high load atm?15:21
*** skraynev has quit IRC15:21
sdaguethat just faceplanted on 2 jobs at about the same time on a clone15:22
mordredgreghaynes: yup. we have a short-term workaroudn for tox2 - and are looking at just making things work from clouds.yaml longer term15:22
BobBallAnd that's why I just stopped the libvirt+xen ci sdague :)15:22
fungiBobBall: zuul could probably be easily extended to do something similar. right now it has a signal handler callout to wait for changes being tested to report but queue any new trigger events in preparation to save its queues and exit. could probably add a pair of signals to quiesce and resume using most of the existing logic15:22
sdagueBobBall: ok, it's you? :)15:23
mordredsdague, greghaynes, Shrews, dtroyer:
*** skraynev has joined #openstack-infra15:23
BobBallI'm happy for you to say it's my fault... but I'm afraid I have no idea what's going on :)15:23
BobBallJust that it seems to be failing a huge percentage of the changes starting about 30 minutes ago ish for me15:24
*** links has quit IRC15:24
BobBallAnd the best part is that because 'logs' doesn't exist it fails to upload _anything_ for me! :)15:25
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Switch on docs and publish jobs for magnum
*** ihrachyshka_ has quit IRC15:26
sdaguemrodden: I think you have to fix at least one thing15:27
sdaguesorry mordred ^^15:27
mroddensdague, what did i break?15:27
sdaguethere is at least 1 typo, 1 ... I'm not sure that's safe, and then 1 of me giving you hard time on naming15:27
*** sabeen1 has joined #openstack-infra15:28
mordredsdague: awesome15:28
greghaynesmordred: also commented15:28
mordredsdague: the auth dict is because of keystone auth plugins15:29
mordredsdague: so taht you can do validation of parameters15:29
mordredsdague, greghaynes: awesome. thanks! updated15:31
mordredsdague: and yes - it's a bit bonghits15:31
*** nfedotov has quit IRC15:32
*** sabeen3 has joined #openstack-infra15:32
*** linuxmaniac has quit IRC15:34
*** sabeen1 has quit IRC15:34
Shrewsmordred: commented.  i think you mean ! -f for the clouds.yaml15:34
nibalizerfungi: cool, did everything with afs go okay?15:35
jeblairBobBall, fungi, sdague: starting around 14:20, the git fe's have seen an unusually high number of established tcp connections
funginibalizer: seems to not have failed to puppet anyway15:36
mordredShrews: doh15:36
Shrewsmordred: i'd even go with ! -e15:36
fungijeblair: nodepool image updates?15:37
fungijeblair: they kick off at 14:1415:37
jeblairfungi: ah maybe so then15:37
mordredShrews: ! -e ? why?15:37
*** jcoufal has quit IRC15:37
jeblairdib will make this better :)15:37
fungijeblair: might also not be our nodepool image updates, if other people reuse our nodepool.yaml example verbatim15:37
mordredjeblair: yah it will15:37
Shrewsmordred:  -e tests if it exists,  -f tests if it's a file, not directory15:37
mordredShrews: fair15:38
Shrewsmordred: just more logical15:38
BobBallfungi: The number of connections is way higher than earlier in the month though15:38
fungiBobBall: yeah, that's why i saw it may not be ours15:38
clarkbfungi: BobBall could also be more reliable clouds/image builds15:38
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Testing the ceph job.
*** linuxmaniac has joined #openstack-infra15:38
clarkbfor example bare-centos6 was not building for ~6 weeks15:38
* BobBall changes his nodepool.yaml to kick off the image builds at a different time to spread the load :)15:39
*** nadya has joined #openstack-infra15:40
*** nadya is now known as Guest7915615:41
*** Guest79156 is now known as _nadya_15:41
*** Swami has joined #openstack-infra15:42
fungimordred: 20008 connections through git-fe01 from so far this hour... normal?15:42
sdaguemordred: also... for this to work in the novaclient test case, we're going to need admin creds in there as well15:42
fungier, sorry, that was last hour (14:00-15:00 utc)15:42
mtreinishfungi: oh, that would probably explain why I can't clone trying to spin up a new devstack node15:43
fungiand another 5120 from it through fe0215:45
fungiso it made 25128 connections in an hour (a little over 7 per second)15:46
jeblairroot@hound:/var/log# less hound.log15:47
jeblair"hound.log" may be a binary file.  See it anyway?15:47
*** _nadya_ has quit IRC15:47
sdakeredirect loop at libertydesignsummit.sched.org15:48
* sdake groans15:48
*** sarob has joined #openstack-infra15:48
jeblairsdake: wfm?15:48
sdakerestarting chrome15:49
sdakeya it keeps going to linked in and back to libertydesignsummit15:49
sdakei got an error about linked in not authorized earlier as well15:49
sdakemaybe thats the cause15:49
fungiweird. not for me in firefox anyway15:49
jeblairsdake: oh, yeah, i'm not linking anything in.15:50
sdakeI mean I have my linkedin credentails set I think15:50
*** markvoelker has quit IRC15:50
sdakebut maybe there is a temporary breakage15:50
nibalizeralso "ERROR:open /home/hound/data/idx-403a35fe9c764811/raw/LICENSE: too many open files" from the hound ui15:50
*** gyee has joined #openstack-infra15:51
fungii didn't realize there was a linkedin integration in, but it wouldn't surprise me. i don't have a linkedin account/profile so probably would never have noticed it15:51
jeblairso if this were a real server, we'd have diagnostic information in cacti, etc. :/15:51
fungijeblair: i'm in favor of just turning it off for now and mordred can investigate when he's free15:52
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras and env markers
fungisince it's a non-production demo/test anyway15:52
jeblairlemme lsof first...15:52
electricalfungi: okay. looking forward to know :-)15:53
wznoinskhi, git clone works, while git clone git:// doesn't, using same socks proxy, could someone check their end?15:54
*** dtantsur is now known as dtantsur|afk15:54
*** yamamoto has quit IRC15:55
openstackgerritMerged openstack-infra/project-config: Mark apps-catalog py34 job as non-voting
fungijeblair: mordred: the count of connections works out to about 38 per repo we host per hour, so unsurprising if it's trying to remote update copies of all of them periodically (not sure how many separate git protocol connections that entails)15:55
*** ajmiller has joined #openstack-infra15:55
*** yamamoto has joined #openstack-infra15:55
*** ildikov has quit IRC15:55
sdakegroan redirct loop on both sched sites for summit15:56
fungiit's possible these are very short-lived and not contributing heavily to traffic volume or backend load15:56
*** dims has quit IRC15:56
*** clif_h has quit IRC15:56
sdakesent tweet to support hopefully they cna fix15:56
jeblairmordred, fungi, nibalizer: houndd had 1312 files like /home/hound/data/idx-2dbe7b1ea98063a5/tri open.  i stopped it.15:56
fungijeblair: thanks. let's see if that gets any better15:56
*** dims has joined #openstack-infra15:56
fungiwznoinsk: are you sure you have access to reach port 9418/tcp on the internet?15:57
wznoinskfungi, it wasn't working for last 1h and started to work a second before your msg ;-)15:58
mordredjeblair, fungi: ah, fun - so yeah, it sets up a poller thread to check for changes in the repos - we can lower the frequency in config15:58
fungiwznoinsk: okay, then that's the overload we're invesitgating15:58
jeblairfungi: i think it just git pulls frequently, so it should be fairly lightweight.  it also did not log any errors around the times we're interested, though it did see some earlier.15:58
fungiwznoinsk: in the future "stopped working" conveys a lot more information than "doesn't work" (which could also be interpreted as "never worked")15:59
jeblairmordred: i think it's not a problem under normal circumstances; i think at the moment we're wondering if it's related to the spike.  it may not be.15:59
*** markvoelker has joined #openstack-infra15:59
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Switch to environments in tripleo-heat-templates.
wznoinsktrue, will keep that in mind15:59
*** kmartin has joined #openstack-infra15:59
jeblairmordred: however, nibalizer did point out an error potentially related to leaking fds15:59
mordredjeblair: k. it also could be made more efficient - it currently doesn't grok watching local repos for updates - which I think it shoudl learn. if it did, then we could easily have an rsync cron job instead of a per-repo poller16:00
fungiwznoinsk: in that case, yes we're tracking a spike in activity for git protocol access to our farm16:00
mordrednibalizer: ah - lovely16:00
mordrednibalizer: wanna learn go?16:00
*** puranamr has joined #openstack-infra16:00
mordredsdague: re: admin creds - overriding username/password is easy to do and I'll include that in my novaclient patch16:01
jeblairfungi: i see an increase in established tcp corresponding with a decrease in tcp opens on fe0216:02
*** craigbr has quit IRC16:02
*** jlanoux_ has quit IRC16:02
mordredsdague: in the gate case, admin/admin and demo/demo are the admin and non-admin users, yeah?16:02
*** ivar-lazzaro has joined #openstack-infra16:02
*** yfried has joined #openstack-infra16:03
nibalizermordred: uh16:03
*** tonytan4ever has joined #openstack-infra16:03
nibalizermordred: are we all learning go?16:03
mordrednibalizer: NO16:03
Clintrust, all rust16:04
mordredClint: ++16:04
openstackgerritGal Sagie proposed openstack-infra/project-config: Update networking-ovn job
*** asrangne has joined #openstack-infra16:05
clarkbheh if you keep throwing rust and go around I will have to dust off my haskell16:05
mordredclarkb: just learn rust - it's got the good bits of haskell without the pedantic16:06
clarkbmordred: those are the good bits of haskell (and no rust really doesn't have the good bits of haskell)16:06
mordredclarkb: we may differ in what we think are the good bits then :)16:06
fungijeblair: so the next highest offenders are https not git protocol. an ip address in (6029) and a bunch of addresses spread through the top half of a /24 in (5955) for that hour16:07
*** ivar-lazzaro has quit IRC16:07
*** freerunner has quit IRC16:07
jeblairgiven a log entry in haproxy, can i tell what source port it used when connecting to the backend?16:08
clarkbfungi: jeblair fwiw I do think that it is the expensive to pack operation that hurts more than total connections, but git's logging isn't really good enough to track that down and prove it16:08
openstackgerritGal Sagie proposed openstack-infra/project-config: Update networking-ovn job
clarkbjeblair: iirc yes but I need to go look at an haproxy log really quickly16:08
mordredyah - I mean, this is one of the reasons that started offering public rsync iirc16:08
*** aswadr has quit IRC16:08
mordredbecause they hit scaling problems doing git things16:08
*** tiswanso has joined #openstack-infra16:09
*** dboik has joined #openstack-infra16:09
pleia2good morning16:10
nibalizerpleia2: o/16:10
clarkbjeblair: wow you know what, it looks like no16:10
clarkbhow can a proxy server not do that16:11
clarkb(I am reading the haproxy loggging doc section now to confirm)16:11
jeblairyeah, i can't see a way :(16:11
*** puranamr has quit IRC16:11
*** puranamr has joined #openstack-infra16:12
sigmavirus24clarkb: fwiw, I'm down with haskell too16:13
sigmavirus24especially given haskell and rust at least have half-way decent package managers16:13
*** tiswanso has quit IRC16:13
*** dboik has quit IRC16:14
clarkbit has been too long since I wrote any haskell, I should find and excuse to brush up16:14
clarkbbut breakfast now16:14
*** tqtran has joined #openstack-infra16:14
fungiclarkb: i thought xmonad kept you on your haskell toes?16:14
clarkbfungi I occasionally have to fix my config. trying it on tumbleweed was fun16:15
*** shardy has quit IRC16:16
greghaynesclarkb: you took the tumble?16:17
*** freerunner has joined #openstack-infra16:17
*** unicell1 has quit IRC16:17
clarkbon a vm16:17
clarkbhad to make xmonad work16:18
*** e0ne is now known as e0ne_16:18
jeblairfungi: gitfe02 has something like 28 connections from rax HK16:20
fungioh, looking at concurrent connections as opposed to number initiated over a span of time?16:22
jeblairfungi: yeah.  they seem to be very long-lived16:22
*** patrickeast has joined #openstack-infra16:22
zaroelectrical: that might be a jenkins limitation.16:23
pelixelectrical: I believe those just happened to be the early precursors to publishers and were only compatible with maven project types and were called reporters since that's how maven would refer to them.16:23
pelixSubsequently as the idea of publishers was created to be generic and plugins moved to use that naming as they were made compatible with other project types16:23
*** craigbr has joined #openstack-infra16:23
*** derekh_ has quit IRC16:24
*** afazekas has quit IRC16:24
*** dkranz has quit IRC16:25
*** e0ne_ is now known as e0ne16:25
fungijeblair: oh, i see. over ipv6 for git protocol16:27
btullyare there any current/known issues with the openstack git repos? gettling lots of timeouts and very slow connections16:27
fungibtully: yes, something is hammering our git server farm hard and we're trying to track it down16:29
clarkbconnection reset by peer is the error ya? I have checked the bw use on the frontends and that is below our theoretical limits16:29
clarkbbut rax sending resets for us was the old behavior that seems to be related to bw limiting16:29
jogoits been happening for about 2 hours16:29
fungijogo: yep, that matches our graphs16:29
openstackgerritMerged openstack-infra/elastic-recheck: Add query for heat tox 2.0 bug 1455065
openstackbug 1455065 in heat "New tox version broke integration tests" [Critical,In progress] - Assigned to Sirushti Murugesan (sirushtim)16:29
clarkbin that case I had confirmed that the connection that was reset on the jenkins slave side never showed up in the haproxy logs16:29
openstackgerritMerged openstack-infra/elastic-recheck: Add query for neutron/manila tox 2.0 bug 1455102
openstackbug 1455102 in python-novaclient "some test jobs broken by tox 2.0 not passing env variables" [High,In progress] - Assigned to Sean Dague (sdague)16:30
*** gokrokve_ has joined #openstack-infra16:30
*** whoops has joined #openstack-infra16:30
*** cody-somerville has quit IRC16:30
electricalzaro: pelix okay afaik all reporters work with freestyle ad well. I can run some tests if you guys want.16:30
*** dannywil_ has joined #openstack-infra16:30
*** krtaylor has quit IRC16:31
jeblairclarkb, fungi, jogo: the logstash lines are _all_ from hpcloud nodes, none from rax16:31
jeblairperhaps the fault is not at the git farm16:31
btullythanks fungi - good luck tracking the offenders down!16:31
jogojeblair: there is *one* from rax bare-trusty-rax-dfw-255264216:31
jogo1 out of 2416:32
fungiyes, entirely possible it's some connectivity problem, but wondering why people are seeing issues via git:// and not http(s)://16:32
*** ashleighfarnham has joined #openstack-infra16:32
SpamapSgreghaynes: are you working on a devstack patch to emit clouds.yaml?16:33
*** gokrokve has quit IRC16:33
jeblairoh, if you go back further, there are more hits from rax16:33
*** gokrokve_ has quit IRC16:34
greghaynesSpamapS: Theres a hacky patch already, im working on osc support16:34
*** yamamoto has quit IRC16:34
fungimemory usage on git05 is pretty bad, and git01 is also bad but not as bad16:35
SpamapSgreghaynes: ok, because it sounds like that would be a better answer to our functional test things than trying to ram through the OS_x vars we want. Yes?16:35
greghaynesSpamapS: yes16:35
greghaynesSpamapS: thats the plan16:35
clarkbfungi: it has 1GB free16:35
clarkbthouh that seems to rapidly change16:36
greghaynesSpamapS: basically, do ^ now, give osc a command to edit clouds.yaml based on env vars / args16:36
clarkbfungi: several git processes all using >500MB of ram16:36
fungiclarkb: right, but from a performance standpoint that means basically no caching in memory16:36
clarkblsof reports its for nova16:37
SpamapSgreghaynes: I have no idea what that means.16:37
funginot surprising16:37
*** dkranz has joined #openstack-infra16:37
mordredSpamapS: osc == python-openstackclient16:37
mordredSpamapS: we're going to add cloud config manipulation commands to it16:37
fungii'll see if filtering logs by hits to git05 paints any different of a picture16:37
SpamapSmordred: how does that help us get shade's functional testing working?16:38
mordredSpamapS: that know how to transform either env vars, or files of env vars into the clouds.yaml16:38
zaroelectrical, pelix : maybe try to approve both &  and then do a release today?16:38
mordredSpamapS: well, we rework shade's functional testing to consume clouds.yaml instead of env vars16:38
SpamapSsince we've decided not to rollback tox 2.0 incompatible changes (very frustrating IMO)16:38
mordredSpamapS: first using the devstack patch above - but then we clean up the devstack support by using the osc command16:38
*** craigbr has quit IRC16:38
*** claudiub has joined #openstack-infra16:39
yolandajeblair , mordred, can you give some feedback about ? the nodepool queues change16:39
*** wenlock has joined #openstack-infra16:39
mordredSpamapS: because we can provide this functionality to other people and not just our gate16:39
yolandatoday we had some hpcloud slowness and it should have been helpful16:39
*** BharatK has quit IRC16:39
fungijeblair: that rax hk address seems to be spreading its concurrent sessions across both git-fe lbs16:39
mordredSpamapS: not sure I follow the rollback frustration? I'm probably also frustrated though16:39
*** jlanoux has joined #openstack-infra16:39
zaroelectrical: if tox isn't working for you it might be that you need to install a compatible version of tox and virtualenv16:40
SpamapSmordred: the patch to cap tox at <2.0 has three -1's, mostly with the theory that "it's early in the cycle so its ok for everything to be broken"16:40
mordredSpamapS: well, pinning tox is a big deal, beause it means "pip install tox; tox -epy27" will be broken for people16:40
fungiSpamapS: experience has shown that if we hardcode a specific piece of tooling which is outside our self-testing loop, it's a massive undertaking to undo that later16:40
mordredso it's just masking a real world breakage16:40
SpamapSwhich frustrates me because our cycle has very little to do with our downstreams' cycles and breaking all tests right now means we are discouraging them from tracking trunk. :-/16:40
SpamapSfungi: yes, we are faced with two evils. :)16:41
SpamapSI choose the massive undertaking later, because later means we can plan, track, and execute it on a set schedule.16:41
mordredSpamapS: they'd have to also track our infra puppet to pin tox for us pinning tox to help them16:41
SpamapSNow means "zomg its on fire haaallp"16:41
SpamapSmordred: devstack is responsible for installing tox :-/16:41
fungiSpamapS: the options are 1. hide the broken from everyone and hope they fix it anyway with no real incentive to do so, or 2. leave the broken obvious so they can tell when they've fixed it all16:42
mordredSpamapS: only for devstack16:42
*** jistr has quit IRC16:42
*** unicell has joined #openstack-infra16:42
SpamapSfungi: "no real incentive" isn't true. We can set a date, say, L2, where it will be unpinned. Give people time to plan, work on fixes, and maybe even have an experimental job which unpins it.16:42
jeblairfungi, clarkb: i looked for session termination anomolies, and they only ones i see around the times we're looking at are a handful of "cL", which is "client timeout while sending the last data after the server has finished"16:42
*** asilenkov_ is now known as asilenkov16:42
SpamapSmordred: luckily, devstack is the primary victim of the env var change.16:43
fungiSpamapS: as i said, for much of our infrastructure that's outside our self-testing loop. we build new worker images and hope everyone's covered their bases16:43
mordredSpamapS: well, I believe I've got patches up to fix or provide vehicle for fixing all the things - although I need to do a shade one real quick16:44
SpamapSOK, I guess I don't really grok the relationship we have with the things we test. I'll pipe down and just go back to zomg its on fire and haaaalp myself. ;)16:44
*** enikanorov2 has quit IRC16:44
fungiSpamapS: for a lot of our tests, low-level plumbing like setuptools, pip, virtualenv, tox, et cetera aren't easily testable in automation. they're preinstalled for use _by_ tests16:45
fungithey could be made testable, but that too is a major undertaking and i'd be hesitant to just freeze our entire toolchain on the assumption that someone will come along and solve that eventually16:46
clarkbjeblair: show errors against the haproxy admin socket doesn't show anything for either haproxy server16:46
clarkbjeblair: and it is surprisingly difficuly to map a connection on a backend back up to the frontend to see what is causing it (trying to see if I can figure out if these high memory nova packs are consistently from one location)16:46
fungiSpamapS: which means the devstack fix boils down to "use an older tox in devstack than we do everywhere else" and that comes with its own problems16:47
SpamapSfungi: so, I see what you're saying. We don't have an easy way to have a "future" toolchain.16:47
fungiSpamapS: not now at least. would be nice!16:47
mordredyah. I have thoughts ...16:47
*** ildikov has joined #openstack-infra16:47
clarkbfungi: the other issue we run into is when people run tox for unittest they get latest tox and it doesn't work16:47
SpamapSSeems like something that will come up so infrequently as to not be a priority.16:47
clarkbfungi: this resulted in tons of questions every day when we had pinned tox16:47
clarkb"why does tox not work"16:48
SpamapSYeah, it's two evils, I completely acknowledge that.16:48
mordredSpamapS: btw: is what I just put together for novaclient functional testing - I think it probably wants to be made into a fixture - but for now cargo culting is probably fine16:48
pleia2it's really not obvious to folks which one to use when it's pinned16:48
zaroclarkb: check this out,
SpamapSAnd without an actual way to provide both evils to users, picking "most current software evil" is probably the right play.16:48
fungiyep. for the devstack case maybe that's less of a problem, but it's not really teh "devstack" case it's the "tempest" case so you're effectively saying "let's fix tempest by pinning tox in devstack, nobody uses tempest except by running it within a devstack-built environment"16:49
clarkbzaro: neat, so they are using jenkins now instead of that builder script?16:49
zaroclarkb: i'm wondering if it might be better to use that gerritforge setup to get our gerrit plugins instead of building it ourselves.16:49
clarkbzaro: or maybe using both I guess16:49
SpamapSpleia2: well .. we could put these things in a file in global reqs and advise people to install only those versions.16:49
clarkbzaro: thats a possibility16:49
SpamapSBut again, as fungi points out, you need a way to _ALSO_ have automated testing of unpinned, or people will never unpin.16:49
mordredclarkb, zaro: why would we use someone else's build farm?16:50
openstackgerritSwaminathan Vasudevan proposed openstack-infra/project-config: Make neutron-dvr job voting at the gate
mordredoh neat though! they use jjb16:50
clarkbmordred: becusae we don't need to build all the gerrit plugins (or really even gerrit itself)16:50
mordredclarkb: fair enough16:50
clarkbmordred: thats mostly an artifact of history16:50
jeblairwe do have a local patch16:50
zaroi think we still need to build gerrit since we do a fork.16:50
*** yamamoto has joined #openstack-infra16:51
clarkbjeblair: so doing more lsof digging, nova does seem to consistently need that memory overhead16:51
mordreddoes anyone else think that it's funny that is a jenkins and not a gerrit?16:51
zarobut the plugins should be ok to just grab from there instead of building it ourselves16:51
SpamapSmordred: yes that should go into oslo.test me thinks.16:51
SpamapSor oslo_test I suppose is what we call things now16:51
clarkbSpamapS: even with automated testing of unpinned I doubt many would care to fix16:51
jeblairclarkb: yes, i have observed this in the past; a given be can only handle a small number of nova clones at a time.  do you think the be's are overtaxed at the moment?16:51
clarkbjeblair: ya I think that may be the case, then as they fall behind haproxy may be telling clients togo away16:52
zarohaha! i didn't even think about that but, yeah what's up with that?  would be a good question for luca16:52
*** signed8bit is now known as signed8bit_ZZZzz16:52
jeblairclarkb: i don't see haproxy doing that though16:52
clarkbjeblair: we do have connection limits particularly on the git:// protocol because the git daemon itself does not do any connection management16:52
clarkbjeblair: ah in that case likely not the problem16:52
fungijeblair: clarkb: maybe not a great correlation, but those established connections from rax hk died down and so did memory consumption on the backends to which they were being shunted16:52
clarkbjeblair: though I want ot say the limits are above what we see on the backends16:53
clarkbiirc we would peak at ~30 connections per backend during normalish operations16:53
clarkb*concurrent on average as reported by cacti16:53
jeblairclarkb: i would expect that to show up as " s. " for the connection state (that's a regex match anything dot)16:53
jeblairclarkb: and that only shows up 6 times today, all well before the current issue16:54
jeblair(sD to be precise in these cases)16:54
clarkb that graph has ramped up over time, but its still quite low overall16:54
*** fawadk has joined #openstack-infra16:55
fungijeblair: clarkb: also comparing the netstat output to the haproxy logs, that wasn't one ip address from rax hk, it was more like 6016:55
funginetstat was truncating a bunch of addresses from the same /6416:56
jeblairfungi: oh i was using lsof and i saw 28 simultaneous when i took a snapshot16:56
SpamapSclarkb: I disagree that nobody would care to fix. One simply needs to give them a clear time frame when the pin will be advanced, and get it on their roadmap.16:56
electricalzaro: your patch looks good.16:56
clarkbjeblair: reading docs I agree, s. and likely sD or sQ16:56
SpamapSclarkb: just like python 3, now that EOL for 2 is imminent, people are actually doing it.16:57
clarkbSpamapS: our classic example is the last time we pinned tox fwiw16:57
fungijeblair: from the same singleton address? got it16:57
clarkbSpamapS: I had to fix it, and it was like pulling teeth16:57
SpamapSclarkb: Perhaps you should not have fixed it?16:57
clarkbSpamapS: I had to for python34 support16:57
electricalzaro: that should make everything work nicely on Python 3.x16:57
*** yamahata has quit IRC16:58
*** ddieterly has quit IRC16:58
jeblairclarkb, fungi: we've seen those addresses for the past few days at least, so it's not like they just started up today16:58
SpamapSOpen a bug in everything that's broken, target it at CURRENT_MILESTONE+1, and note in the bug that the pin will be advanced immediately following CURRENT_MILESTONE+116:58
fungiyep, may also be unrelated to this16:58
*** gokrokve has joined #openstack-infra16:58
*** fawadkhaliq has quit IRC16:59
*** ddieterly has joined #openstack-infra16:59
*** enikanorov2 has joined #openstack-infra17:00
clarkbSpamapS: the amount of managerial effort involved in that is o_O and for a large majority of the projects you will end up wedging things just as badly as you do if you let the broken in immediately or worse ecause instead of one broken you have accumulated 10 brokens that all need fixing17:00
clarkbSpamapS: the accumulated broken was part of the tox issue last time17:00
electricalyolanda: is there anything i can do / test to make the reporters open for other type of jobs?17:00
jeblairfungi, clarkb: also, the HK addresses transferred more data yesterday and the day before, and had more connections.17:01
jeblairthey transferred 37610899052 today (so far) and 60251402389 yesterday17:01
* jeblair just learned: |awk '{s+=$11} END {print s}'17:01
fungiooh! a list sum function?17:02
pleia2hah, nice17:02
jeblairyeah, sum column 11 in haproxy log17:02
fungii've always resorted to hacking separate perl or other madness17:02
fungithat's handy17:02
pelixzaro: found a chunk of bugs in python-jenkins around python 3 and post requests. Have some additions to the tests that force checking on the request to ensure it's valid on python 3.17:03
pelixMost of the api calls just needed to have their calls changed from using '' to trigger a POST opperation, to use b'' instead17:03
*** mestery has joined #openstack-infra17:03
electricalpelix: interesting. i didn't get that far so was unable to test :-)17:04
pelixbut for create_job and reconfig_job the tests currently don't encode what is being sent in. Should we assume the application will correctly encode the XML, or always test and encode it if not already encoded?17:04
*** krtaylor has joined #openstack-infra17:04
*** yamamoto has quit IRC17:05
electricalwe could always call encode just to be safe?17:05
yolandaelectrical, there was a spec for rewriting a bit of jjb17:05
SpamapSclarkb: I agree with all that you say, however, an additional factor is that the fixes for the 10 brokens can be done on a plan, where as right now the brokens must be done _immediately_ and interrupt any plans.17:05
*** e0ne has quit IRC17:05
yolandaalso you can send patches to add new modules for jjb17:05
clarkbjeblair: more info on the large nova pack processes, they seem to be IO bound with a ton of writes to fd 1, I think thats basically writing up a pipe back to the parent process which is speaking tcp17:05
*** sdake has quit IRC17:06
pelixelectrical: yeah, just decided to see if some of the other api calls worked on python 3 when testing your patch and discovered that Request have different enough behaviour that it was worth modifying the tests to build the request fully to check for exceptions.17:06
electricalyolanda: i did send that logstash reporter module for jjb. but reporters are limited to maven2 type jobs but i use freestyle mainly17:06
clarkbjeblair: but 8192 bytes at a time very slowly17:06
electricalpelix: ahh i see :-)17:06
*** sdake has joined #openstack-infra17:06
*** freerunner has quit IRC17:07
*** wenlock has quit IRC17:07
jeblairclarkb: are you sure it's not the reads that are slow?17:07
clarkbjeblair: this process isn't doing any reads (it may be the other end reading slowly off the pipe)17:07
*** pabelanger has quit IRC17:08
jeblairclarkb: something is reading off of disk, right?17:08
clarkbjeblair: but I think that may be why it uses so much memory, the entire thing is loaded into memory, then its writing it out17:08
jeblairclarkb: okay i see17:08
jeblairclarkb: so you think it's already read substantial data, and is now slowly streaming it out17:08
openstackgerritDarragh Bailey proposed stackforge/python-jenkins: Modify tests to exercise Request on python3
jeblairclarkb: (possibly limited by client bandwidth); and in the interim, it's sitting with the entire nova pack in memory?17:08
clarkbjeblair: yup, and while it does that slowly it has consumed that memory resource17:08
clarkbjeblair: ya17:09
pelixzaro: see change ^ for test changes, need to fix up a few things first17:09
zaroelectrical: if you haven't notice, i -1 your python-jenkins change :)17:09
electricalzaro: i saw yeah. want me to add the decode there as well ?17:09
fungielectrical: pelix: "always call encode just to be safe" introduces new problems, in particular trying to run a unicode-encoded string through .encode('utf-8') raises a UnicodeDecodeError exception17:09
*** pabelanger has joined #openstack-infra17:09
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: Do not automatically target bugs for stable releases
electricalfungi: uhg true.17:09
zaroelectrical: i don't have python 3 setup.  so yeah, just test to see if it's needed.17:10
*** ianw has quit IRC17:10
fungielectrical: er, a utf-8 encoded string through .encode('utf-8')17:10
electricalzaro: okay, will try if i can find it out and report back17:10
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Decouple openstack_project::server from Gerrit
*** dboik has joined #openstack-infra17:10
jeblairclarkb: it looks like it got close to using all the ram, but not quite?17:10
jeblairclarkb: the machine has not swapped17:11
clarkbjeblair: I did see an madvise go by too, so it may be freeing up part of that buffer as its write pointer moves along. It did a MADV_DONTNEED17:11
*** shashankhegde has quit IRC17:11
fungielectrical: basically to successfully use .encode() you need to first know or be able to somehow detect the existing encoding, if any17:11
clarkbjeblair: ya, I don't think it every completel ran out of memory but the pressure did affect the memory cache size17:12
fungiso that you can decode back to unicode and translate into the new encoding17:12
electricalfungi: makes sense.  found this:
clarkbjeblair: we may be trading pack files cached by kernel for pack file in $process memory space, which negatively affects the next pack process?17:13
*** gokrokve has quit IRC17:13
AJaegerI just noticed on the nova queue that job check-tempest-dsvm-multinode-full is NOT_REGISTERED. The job runs on node devstack-trusty-2-node. Anybody an idea what's wrong here?17:13
*** gokrokve has joined #openstack-infra17:13
fungielectrical: yep, that's a fairly reasonable way to do it, or use the various string handling utility functions from six17:14
electricalfungi: ah. okay. not so familiar with python yet ;-)17:14
electricalfungi: zaro i have to run and go out for dinner. will do some more checking / testing tomorrow. feel free to ping me on IRC, ill be able to read back17:14
electricalor email :-)17:14
*** dboik has quit IRC17:14
*** dboik has joined #openstack-infra17:15
*** dboik has quit IRC17:15
electricalfungi: ah interesting :-)17:15
fungiconvenience library to handle writing code compatible with both python 2 and 317:15
*** dboik has joined #openstack-infra17:16
jeblairclarkb, fungi: it makes sense that lots of slow clients cloning nova would run up the ram and established connections; i don't actually see how that results in connection errors for other clients :/17:16
*** gyee has quit IRC17:16
*** freerunner has joined #openstack-infra17:16
*** baoli has joined #openstack-infra17:16
*** cdent has quit IRC17:17
clarkbjeblair: maybe apache failed to fork/startthread and its nicely closing the connection for new clients? except I think most of the issues were over git protocol ya?17:17
*** gokrokve has quit IRC17:17
zarofungi: so i cannot repro on my local machine.17:17
fungiclarkb: yeah, it was apparently git protocol which saw the brunt of the impact17:17
*** gokrokve has joined #openstack-infra17:18
clarkbI am not seeing anything inthe apache error log that would support that theory17:18
zarofungi: it is happending on review-dev. now i'm wondering if it's something to do with the reverse proxy setup.  i don't have that setup on my local machine.17:18
*** tiswanso has joined #openstack-infra17:19
fungizaro: it's possible. i tried several ways of working around it in the proxy config, to no avail. but it definitely seems to be a behavior difference between 2.8 and 2.1017:19
jeblairclarkb, fungi: i do note some packet loss from my workstation to fe0217:19
*** Krinkle|detached is now known as Krinkle17:20
jeblairand 0117:20
zarofungi: behavior difference + some config on our gerrit servers17:21
*** yamahata has joined #openstack-infra17:21
jeblairclarkb, fungi: i'm browsing the logstash hits and they seem to be mostly on initial connections for 'git remote update'17:21
fungijeblair: clarkb: looking at the logs on git05, git-daemon definitely shows a log of "Disconnected (with error)" along with things like "git-upload-pack died of signal 13"17:22
*** bhuvan has joined #openstack-infra17:22
fungier, a lot of17:22
zarofungi: would it be possible to turn off the reverse proxy thing on review-dev for a few minutes to see if it still a problem?17:22
clarkbfungi: syslog?17:22
fungiclarkb: messages17:22
clarkboh right rhel17:22
fungi(this is sparta^H^H^H^H^H^Hrhel!)17:22
btran_Hi my Cinder CI testbed was running fine about a week ago.  But now I"m getting an problem with devstack gate getting configured and setup.17:23
jeblairclarkb, fungi: i have to run for a bit; the graphs and logstash seem to indicate that whatever was happening is over17:23
btran_here are my logs, the error occurs at the end of the logs:
*** abregman_ has quit IRC17:23
asselinbtran_, are you reusing nodes?17:23
fungizaro: the proxy is what's making gerrit reachable. you'd need to reconfigure gerrit to listen on a public address on 443 and add https to it somehow, or maybe just have it listen on port 8017:23
btran_asselin: sometimes yes, is that my problem. I need to run in clean setup all the time?17:24
asselinbtran_, yes17:24
asselinbtran_, seems the leftovers of a previous test is causing the script to fail17:24
clarkbin other news the ES cluster is green and happy again17:25
*** dizquierdo has quit IRC17:25
*** gokrokve has quit IRC17:26
btran_asselin: i see.... is anyone working on fixing that?  or will this script cleanup problem be here for some time?17:26
*** gokrokve has joined #openstack-infra17:26
asselinbtran_, everyone is using clean nodes. take a look at nodepool:
asselinbtran_, so yes, the script cleanup problem will be there for a long time....17:28
fungibtran_: the problem will always exist. if you're running jobs with root privileges you're running untrusted code that could make the entire system inoperable/unusable. someone could just propose a patch that deletes everything on the root filesystem17:28
zarofungi: ok. i'll give it a try.17:28
btran_asselin: yes that's our goal to setup nodepool, but currently there are some resource issues, thanks for the info17:28
btran_fungi:  ok thanks for data/info, I will rise the issue with my mgmt17:29
clarkbarg puppet lint hates me17:30
fungibtran_: you don't have to use nodepool necessarily, but it's the way we and a lot of other ci systems have solved the "use a fresh worker for every job" problem17:30
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Pass OS_ variables through to functional tests
SpamapSmordred: ^ if we land this we can do the clouds.yaml fixes decoupled from "all the other things"17:31
SpamapSgreghaynes: ^17:31
SpamapSShrews: ^17:31
SpamapSclarkb: ^17:31
*** gokrokve has quit IRC17:31
greghaynesSpamapS: Does that work in pre 2.0?17:31
greghaynesas in, not explode17:31
openstackgerritClark Boylan proposed openstack-infra/puppet-logstash: Logrotate the indexer.log logfile.
*** achanda has joined #openstack-infra17:32
greghayneshrm, looks like thats documented in 1.917:32
AJaegerfungi, clarkb, jeblair: I noticed on the nova queue that job check-tempest-dsvm-multinode-full is NOT_REGISTERED. The job runs on node devstack-trusty-2-node. Doy you have an idea what's wrong? The job has been around for a longer time, just renamed a month ago.17:32
greghaynesoh, says new in 2.017:32
fungiAJaeger: checking now17:32
SpamapSgreghaynes: yes17:32
SpamapSpassenv has been there for a while17:33
SpamapSjust not turned on by default17:33
btran_fungi:  yea, I could probably just deploy a fresh worker vm or revert to a clean snapshot of the vm for now, until we deploy/implement nodepool. thanks17:33
SpamapSI just saw that you can do OS_*17:33
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for d-g git prune bug 1455123
SpamapSseems like cheating..17:33
openstackbug 1455123 in OpenStack-Gate "fatal: unable to access '': GnuTLS recv error (-9): A TLS packet with unexpected length was received." [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)17:33
SpamapSbut might be pragmatic given the short life of the patch17:33
*** Apsu has quit IRC17:33
dimsAJaeger: adding check-requirements does not seem to trigger gate-magnum-requirements job ( i am looking at zuul status for a WIP patch 18311217:34
*** Apsu has joined #openstack-infra17:34
AJaegerdims: let me check...17:34
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Pass OS_ variables through to functional tests
AJaegerdims, check
AJaegerlet me adjust your WIP patch to trigger it...17:36
*** shashankhegde has joined #openstack-infra17:37
SpamapSI'm sad pypy isn't compelling enough for us to keep on top of it. :-(17:37
clarkbSpamapS: I thought I fixed it (anyother thing I fixed because no one else was willing to ...)17:38
dimsthanks AJaeger17:38
SpamapSclarkb: see sdague's message. Nobody is tending to it.17:38
clarkbSpamapS: have examples of failures? I can track those down and see if I am missing image builds or something like that17:38
openstackgerritMatt Riedemann proposed openstack-infra/devstack-gate: Add a loop to git_prune
clarkbhrm I seriously just fixed it on monday17:38
SpamapSclarkb: you shouldn't need to fix it. THats the point.17:38
AJaegerdims: check your change again ;)17:38
EmilienMjeblair: after reading the doc and code, I still have a question: is 'Depends-on' able to test a patchset from another project?17:38
fungiSpamapS: i can see the attraction, but then again i can also understand wanting to focus on having an openstack that works well on one python interpreter before trying to make it work well on more than one17:38
SpamapSclarkb: no fixers is a sign that it isn't in anybody's critical path.17:38
dimsAJaeger: aha! :)17:39
AJaegerdims: now zuul shows me "gate-magnum-requirements:"17:39
SpamapSfungi: err... two interpreters... (2.7 and 3.4+  ;-)17:39
clarkbSpamapS: ya17:39
AJaegerdims, so everything is fine...17:39
dimswhew thanks for the tip17:39
SpamapSalso as the swift guys pointed out.. it helps, but not enough to scale to where they want to scale.17:39
AJaegerdims: you're welcome!17:39
*** kmartin has quit IRC17:39
fungiSpamapS: in some ways, still three for a bit (2.6 for some stable branches), and if you consider how every distro modifies the python they provide it's actually a different python interpreter per platform we test on17:40
clarkbSpamapS: well you have to turn off JIT with eventlet in pypy so its pretty unfun for our needs17:40
openstackgerritSirushti Murugesan proposed openstack-infra/devstack-gate: Add Heat to features.yaml
clarkbSpamapS: if however we stopped eventleting that could potentially get much much better17:40
fungithen again if we stopped eventleting even normal cpython has the potential for significant speedups17:41
SpamapSclarkb: the few things left that really need eventlet (read: non wsgi) are the things that would get the most benefit from pypy. :-/17:41
clarkb(or fixed eventlet in pypy I suppose, but unsure if this is a fundamental issue with greentheads running outside of the interpreters thread model or something that is fixable)17:41
SpamapSIt's probably a fundamental incompatibility between pypy's magic and eventlet's magic.17:41
*** harlowja has quit IRC17:41
*** harlowja has joined #openstack-infra17:42
EmilienMjeblair: nevermind, I misread it and it works17:42
clarkbfungi: any luck with the 2-node trusty gearman registration?17:42
SpamapSAnyway, I think there's an axiom forming in my head about this. "You only get one magical thing in your software." or something like that.17:43
erlonkrtaylor: ping17:43
clarkbEmilienM: if you use depends-on with a change in another project zuul will include that change in your stack of zuul refs for your job17:43
EmilienMclarkb: ack, thanks17:43
clarkbEmilienM: this is why figuring out how to set up deps properly with beaker is important for us17:44
*** bhunter71 has quit IRC17:44
clarkbEmilienM: I think we can do it with zuul-cloner being called by that beaker setup script thing17:44
fungiAJaeger: clarkb: looking to see what node type zuul thinks this job should require, but it's definitely not registered since the restart17:44
clarkbfungi: should be devstack-trusty-2-node17:45
*** bhunter71 has joined #openstack-infra17:45
mordredSpamapS: yah - totally. I think the "pass through env vars for shade functional" is definitely step 117:45
fungiclarkb: we have one ready since before the zuul restart. i'll delete it17:45
*** signed8bit_ZZZzz is now known as signed8bit17:45
fungiin theory the next one that builds will get the job registered in zuul17:46
*** signed8bit is now known as signed8bit_ZZZzz17:46
*** kmartin has joined #openstack-infra17:46
clarkbfungi sounds good, thanks17:46
*** signed8bit_ZZZzz is now known as signed8bit17:46
AJaegerthanks, fungi !17:46
fungii'll let you know as soon as it's showing registered again17:47
fungithere's one building currently17:47
AJaegerdims, what are you doing with 183112 now? I'll ignore any review requests now ;)17:48
clarkbfungi: we might also consider slightly bumping the ready node count for that flavor since nova is using it in check now17:48
fungiclarkb: that's a good idea, yes17:48
*** johnthetubaguy_ has joined #openstack-infra17:49
EmilienMclarkb: I'm asking because in our Puppet jobs, it's pulling from master even if you have Depends-on thing17:49
*** bhunter71 has quit IRC17:49
EmilienMclarkb: I can show you an example17:50
crinkleactually our tests specifically do a git clone from master, we don't have a way to co-gate them17:50
fungicrinkle: you could replace git clone with zuul-cloner17:50
dimsAJaeger: haha, i am all set don't worry about that one now. the requirements check job probably version checks only when things change i guess. testing that theory17:50
*** johnthetubaguy has quit IRC17:50
*** johnthetubaguy_ is now known as johnthetubaguy17:50
crinklefungi: how does that work, and would that work for a user running tests locally?17:51
*** bhunter71 has joined #openstack-infra17:51
clarkbEmilienM: crinkle I know that was one of my dislikes of how beaker is currently set up17:51
clarkbI need to distill the current feedback on that spec and write one with work items to address the beaker things17:51
fungicrinkle: it 1. knows how to inspect the calling environment for ref names passed in via job parameters but also 2. takes advantage of the local git cache for other projects on our workers17:51
fungicrinkle: clarkb has a patch proposed to make zuul-cloner work even without zuul envvars present and fall back to normal git behaviors17:52
fungibut alternatively you could just add some conditional logic around the presence/absence of a ZUUL_REF in the env17:52
clarkbfungi: crinkle EmilienM ya thats the change that should let us use it in beaker as it should work on your laptop and in the gate17:53
fungiand use zuul-cloner when it's present but regular git clone somebranch when it's not17:53
EmilienMclarkb: nice17:53
*** e0ne has joined #openstack-infra17:53
*** marcusvrn has quit IRC17:54
fungiclarkb: AJaeger: check-tempest-dsvm-multinode-full is registered in zuul now17:54
*** teran_ has quit IRC17:55
*** marcusvrn has joined #openstack-infra17:55
*** teran has joined #openstack-infra17:55
AJaegergreat, thanks fungi!17:57
*** HeOS has quit IRC17:58
adam_gfungi, you mentioned earlier this week that the $gerrit_patch->$lp_project(s) postback is configurable somewhere. any pointers on where that lives? trying to replicate neutron's flow where patches to multiple neutron-* projects update bugs on the single neutron project17:59
AJaegerfungi: and already in use ;)17:59
*** teran has quit IRC17:59
openstackgerritMerged openstack-infra/system-config: Capitalize references to resoruces
fungiadam_g: yes, you can set a groups: neutron parameter on each of them in openstack-infra/project-config:gerrit/projects.yaml18:00
dimsAJaeger: thanks for setting up the transifex and helping with bootstrapping the projects for translations!18:01
adam_gfungi, ah! thanks18:01
*** BharatK has joined #openstack-infra18:02
*** dannywilson has joined #openstack-infra18:02
krtaylorerlon, pong18:02
erlonkrtaylor: Hi Kurt18:02
*** jlanoux has quit IRC18:03
erlonkrtaylor: I was looking for that cross section on summit, about CI integration but couldn't fund18:03
*** jlanoux has joined #openstack-infra18:03
*** patrickeast has quit IRC18:03
*** patrickeast_ has joined #openstack-infra18:03
erlonkrtaylor: did you manage to get the session?18:03
*** ivar-lazzaro has joined #openstack-infra18:04
*** tonytan4ever has quit IRC18:04
*** dannywil_ has quit IRC18:05
krtaylorerlon, no, but we do have an 2 infra working sessions and a BoF18:07
krtaylorerlon, see the links in the meeting agenda18:08
erlonkrtaylor: ill check18:08
AJaegerdims: since you went through the exercise, what about documenting some of it at ? Could you do this, please?18:09
dimsAJaeger: that's the least i can do :) am on it18:10
mordredclarkb: zuul-cloner as locally usable was the next hard thing you were going to tackly, right?18:10
openstackgerritMerged openstack-infra/subunit2sql: Add db api method to get uuid from test_id
openstackgerritMerged openstack-infra/subunit2sql: Add db api method to get per status counts for a test
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Group akanda sub-projects
erlonkrtaylor: whats is the title/day?18:12
*** hdd has joined #openstack-infra18:12
*** SotK has quit IRC18:12
clarkbmordred you mean for installation? if so yes18:13
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Decouple openstack_project::server from Gerrit
*** freerunner has quit IRC18:14
mordredclarkb: neat18:15
*** dimtruck is now known as zz_dimtruck18:15
krtaylorerlon, BoF ->
clarkbmordred I was just thinking about what a pip install zuul[cloner] might do to indtall only that tools deps and install only that entrypoint18:15
clarkbbut that may be a terrible idea18:15
mordredclarkb: no - I think tat's a GREAT idea18:16
*** zz_dimtruck is now known as dimtruck18:16
krtaylorerlon, Infra work session ->
erlonkrtaylor: thanks18:16
*** sabeen3 has quit IRC18:16
clarkbthat means entrypoints need rxtra like defs too18:16
clarkband extras need to be subtractive too18:17
clarkbso not super simple18:17
*** sabeen1 has joined #openstack-infra18:17
AJaegerdims: I guess you better enhance
krtaylorerlon, sure, np18:17
*** spzala has joined #openstack-infra18:18
*** gyee has joined #openstack-infra18:19
*** freerunner has joined #openstack-infra18:23
clarkbcan I get eyes on and those are bug fixes for log leaking in the logstash/ES system that cropped up yesterday with full /s18:23
openstackgerritMerged openstack-infra/shade: Pass OS_ variables through to functional tests
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: Allow JJB to set connection timeout to jenkins server
*** ildikov has quit IRC18:25
pleia2clarkb: to be clear, the ES stuff rotates itself, but logcheck does not?18:26
mordredclarkb: tchaypo is working on adding extras to reqs - excellent point about entrypoints though - that's an interesting problem18:27
clarkbpleia2 ya18:28
pleia2too many logs18:28
flaper87Do we use cloud archive's PPA in our gates? IIRC, the gate policy is to not use PPAs, am I right ?18:30
*** claudiub has quit IRC18:30
mordredflaper87: that is right18:31
mordredflaper87: we do not use its PPA18:31
clarkbwe did try at one time but it didnt work18:31
flaper87mordred: clarkb thank you!18:32
*** tonytan4ever has joined #openstack-infra18:32
AJaegerflaper87: could you approve some translation imports, please: and
flaper87AJaeger: of course18:33
AJaegerdolphm: what about ? I just rechecked and would appreciate an import, please18:34
AJaegerthanks, flaper8718:34
flaper87AJaeger: done18:34
*** jlvillal has quit IRC18:36
dimsAJaeger: i tried to doc a really simple step by step for an existing project (like mine was) here -
dimsif you like it we can move it to one of the other pages. just wanted to get the formatting right hence started a fresh page18:38
*** ildikov has joined #openstack-infra18:38
AJaegerdims: shall I move it and update on the fly?18:39
*** otter768 has joined #openstack-infra18:40
AJaegerdims: thanks!18:40
*** baoli has quit IRC18:40
AJaegeroh, there's even some initial content - let's merge with
AJaegeryour section complements nicely what's there...18:41
*** jlvillal has joined #openstack-infra18:41
AJaegerand some bits there are outdated ;(18:42
lifelesssdague: sorry, it was late and all:
*** fawadk has quit IRC18:42
*** atan8 has joined #openstack-infra18:43
*** Sukhdev has joined #openstack-infra18:43
*** BharatK has quit IRC18:44
*** otter768 has quit IRC18:45
sdaguelifeless: yep, no prob. So typically "gate is wedged" means tempest or something equivalent is blocking all jobs (or vast majority)18:46
*** changbl has joined #openstack-infra18:46
lifelesssdague: k18:47
sdaguethis was a lot narrower, so being more specific would be good, especially as a lot of base patches were passing fine18:47
lifelesssdague: so if you look at the backscroll18:47
lifelesswe got Kennan and SpamapS reporting it18:47
*** _nadya_ has joined #openstack-infra18:48
lifelesslate my local time, witching period US18:48
lifelessI'll try to be clearer in future18:48
lifelessbut I figured *something* out on the list would help diagnosis when more folk showed up18:48
sdagueyep, so that's fine. I just didn't feel like from your email on the list I had enough to go with. So just trying to figure out what would be better next time.18:49
*** fawadkhaliq has joined #openstack-infra18:49
lifelessagree on having a bug; I should have put the effort into file one18:49
sdagueAnd I think links to a few different failing jobs, and what classes of jobs are failing, would be really helpful18:49
sdagueyeh, even without the bug, examples are good18:49
lifelessI had a mental hat on of 'responding' rather than 'filing', which was a box I think18:49
*** Sukhdev has quit IRC18:50
*** maishsk_afk has joined #openstack-infra18:51
AJaegerdims, I'm on it...18:51
sdaguebecause I had a disconnect with "there's a thing, and this is patch" and that patch randomly failed grenade, which made me think this was going to require chaining. So after about 40 minutes of starting to set up the chaining patches, I got confused with the gate basically being green18:51
sdagueeven though the patch didn't land18:52
*** _nadya_ has quit IRC18:52
*** yamamoto has joined #openstack-infra18:53
openstackgerritYathiraj Udupi proposed openstack-infra/project-config: Added create permission to release group for project nova-solver-scheduler
*** ildikov has quit IRC18:54
*** markvoelker has quit IRC18:54
*** dannywilson has quit IRC18:54
openstackgerritAlexandre Viau proposed openstack-infra/project-config: New stackforge project: puppet-surveil
AJaegerdims, check again18:57
sdagueanyway, it's fine now, people are working through point patches to get individual projects up18:57
openstackgerritAlexandre Viau proposed openstack-infra/project-config: Surveil ACL: pushSignedTag requires surveil-release
dimsAJaeger: "extract_messages" section is not really needed, defaults should work fine18:58
*** yamamoto has quit IRC18:59
dimsright? (not sure)18:59
AJaegerdims: not sure either ;( If it works without, remove the content ;)19:00
*** markvoelker has joined #openstack-infra19:00
dimsAJaeger: let's let it be for now :)19:00
openstackgerritAlexandre Viau proposed openstack-infra/project-config: surveil-specs: use surveil acl
*** maishsk_afk is now known as maishsk19:01
dimsAJaeger: reads well. thanks!19:01
*** harlowja has quit IRC19:03
AJaegerdims, thanks to your quickstart!19:03
openstackgerritMonty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions
*** harlowja has joined #openstack-infra19:05
*** dannywilson has joined #openstack-infra19:05
*** claudiub has joined #openstack-infra19:05
sdakelots of create: patches going into gerrit acls ;-)19:06
sdakeI wonder if thats missing from the documentation19:06
*** hashar has joined #openstack-infra19:06
fungisdake: we didn't originally document it because it's only needed if you're going to create branches instead of having the openstack release management team do it and up until a couple months ago none of the projects outside the integrated release and libraries wanted stable branches19:07
AJaegersdake: please review
sdakeajaeger willd o19:08
fungisdake: the sudden rash of all projects wanting to create additional branches was unexpected19:08
*** Krinkle is now known as Krinkle|detached19:08
AJaegerfungi, 181646 documents it...19:08
AJaeger(me is just fishing for reviews ;)19:09
*** ildikov has joined #openstack-infra19:09
sdakefungi ya just adding comedy relief ;)19:09
sdakeajaeger looks good19:11
*** baoli has joined #openstack-infra19:11
clarkbfungi I think its related to openstack libs doing it19:12
AJaegerthanks, sdake !19:12
*** shashankhegde has quit IRC19:12
clarkbbut stackforge isnt required to19:12
sdakeajaeger say one of our guys wants to get this in before summit so we can announce in our 2pm tuesday session on kolla that we actuallly do real CI :)  any chancey ou can review?19:12
clarkbalso not really here niw waiting to see what verdict on summit travel is19:12
*** sc68cal has quit IRC19:12
*** johnsom_ has joined #openstack-infra19:13
sdakeI think the syntax is right, but I'm not certain19:13
*** baoli has quit IRC19:14
*** baoli has joined #openstack-infra19:14
*** BObH_ has joined #openstack-infra19:15
greghaynesclarkb: do we need to bribe the dr so theyll say you should go?19:15
clarkbmaybe but I doubt it will work19:15
lifelessclarkb: wat, you missing the summit?19:15
fungisdake: so... do you only want it to fail if tox -e images exits nonzero?19:15
sdakepro tip - you can buy codeine OTC in canada ;)19:16
*** smarcet has quit IRC19:16
BObH_Hey folks, looking for a pointer.19:16
sdakefungi right, if tox -e images exits with 0 that means we were successful I hope :)19:16
clarkblifeless dunno yet, depends on whether or not twins are cooperating19:16
lifelessclarkb: twins - eeek19:17
*** annegentle has quit IRC19:17
clarkblifeless indeed19:17
fungisdake: but if say tox -e setupenv errors you still want to run tox -e images19:17
sdakeI have two kids already19:17
sdakeyikes ;)19:17
sdakeif setupenv errors it should exit there19:17
*** johnsom_ has left #openstack-infra19:17
sdakeit should exit on each line19:17
sdakei was thinking it needed && but wasn't clear19:17
sdakeI am not a  pro on infra unfortunately like you guys :(19:17
fungisdake: okay, then you probably need set -e but also the quoting is wrong. i'll leave comments19:18
BObH_Who do I contact about this when I try to get to gerrit?, SIGN_IN< contackt+site+administrator when signing in to
sdakefungi thanks alot ! :)19:18
fungiBObH_: i don't think i've ever seen that. what's your launchpad username?19:19
fungiBObH_: okay, checking now19:19 actually19:20
openstackgerritMerged openstack-infra/infra-manual: Extend Gerrit ACL section
*** matrohon has joined #openstack-infra19:21
*** sc68cal has joined #openstack-infra19:21
AJaegerfungi, thanks ^19:22
*** ajmiller_ has joined #openstack-infra19:24
*** ajmiller has quit IRC19:24
fungiBObH_: found it. anteaya, rloo and gordc requested we disable that account on may 5 because it was leaving machine-generated comments (and useless ones at that) on proposed changes such as
openstackgerritMerged openstack-infra/elastic-recheck: Add query for d-g git prune bug 1455123
openstackbug 1455123 in OpenStack-Gate "fatal: unable to access '': GnuTLS recv error (-9): A TLS packet with unexpected length was received." [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)19:24
fungiBObH_: "Bob Hansen - Patch Set 5: - Merge Failed. - This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset."19:24
BObH_fungi, yes, that was definitely a mistake and has been corrected.19:24
*** stevemar has quit IRC19:25
*** freerunner has quit IRC19:25
*** harlowja has quit IRC19:25
fungiBObH_: okay, so it looks like you're running a zuul instance connected with that account. you've configured it to no longer comment on proposed changes for any projects?19:26
*** harlowja has joined #openstack-infra19:26
BObH_fungi, yes,19:27
fungiBObH_: thanks. try logging in again; i've reactivated that account now19:27
tim_owe are experiencing a config issue with our test setup, so I've halted our neutron builds for the time being while we triage the issue.  Is this the correct course of action?  Just checking...thanks!19:28
fungitim_o: if your system is reporting inaccurate results to proposed changes, then yes please disable it until you work out your configuration difficulties (otherwise someone is likely to ask us to disable the account for you)19:29
*** HeOS has joined #openstack-infra19:29
BObH_fungi, thank you. I'm ok to use the sanbox until I get this baked?19:29
openstackgerritMerged openstack-infra/project-config: Add notifications for all stable branches for kolla
fungiBObH_: sure, "sandbox" repositories are fair game19:30
tim_othanks, fungi!19:30
BObH_fungi, thanks again for your help19:30
fungiyou're both welcome19:30
*** ddutta has joined #openstack-infra19:31
dduttaHi @ajaeger19:32
sdakefungi ajeager thanks for the reviews on jeff's patch appreciate it :)19:33
sdakeajaeger that is :)19:33
dduttahi .... have a new project proposal for review19:33
openstackgerritJeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla
*** weshay has quit IRC19:33
jpeeleryes, hoepfully this is simple enough to get figured out quickly19:34
*** alexpilotti has joined #openstack-infra19:35
fungisdake: jpeeler: sorry, i got distracted by random other requests and just now pushed my comments, but looks like AJaeger set you straight in the meantime19:35
*** fawadkhaliq has quit IRC19:35
openstackgerritJeff Peeler proposed openstack-infra/project-config: Add image building verification to gate for Kolla
AJaegerddutta: as a stackforge project, you cannot publish on - your README needs adjustment at least19:36
jpeelerfungi: all good! feel free to take a look now if you can19:36
AJaegerThe source is also wrong in the README19:36
dduttaajaeger: thanks ...19:37
dduttaI just cloned with cookie cutter for this patch so I will just fix the docs right away19:37
AJaegerddutta: other than that LGTM, +219:38
dduttathanks ...19:38
AJaegersdake, jpeeler looks fine - will +2A once jenkins is happy (which I expect)19:39
dduttaAJaegar: I have checked in my source repos and removed the docs19:40
dduttathanks  a lot for the quick review19:42
*** angela-s has joined #openstack-infra19:45
*** ddutta has quit IRC19:48
*** shashankhegde has joined #openstack-infra19:48
*** cdent has joined #openstack-infra19:49
*** radez is now known as radez_g0n319:49
*** _nadya_ has joined #openstack-infra19:50
*** kmartin has quit IRC19:51
sdakeddutta don't forget python-cognitiveclient ;-)19:52
*** openstackgerrit has quit IRC19:52
*** openstackgerrit has joined #openstack-infra19:52
*** freerunner has joined #openstack-infra19:55
openstackgerritMerged openstack-infra/project-config: Add image building verification to gate for Kolla
*** rguillebert has joined #openstack-infra19:56
rguillebertI'm a pypy dev :)19:57
*** cdent has quit IRC19:57
*** markvoelker has quit IRC19:57
rguillebertI heard people need help for pypy support19:58
openstackgerritMerged stackforge/python-jenkins: Add utf8 to headers
*** dannywil_ has joined #openstack-infra19:58
fungihi rguillebert! sdague can probably explain the situation better than i, but in short openstack is lacking in a sufficient base of developers who want to spend time implementing and maintaining pypy support in our software19:59
*** annegentle has joined #openstack-infra20:00
*** dannywilson has quit IRC20:00
fungiincluding keeping an eye out for and triaging bugs and researching/reproducing/fixing test failures related to the same20:00
fungialex gaynor did some earlier work on getting some of our software tested and working on pypy but hasn't been around for a while to continue keeping up with it20:02
*** jtomasek has quit IRC20:02
rguillebertthat's bad :s20:02
rguilleberttheoretically I can help out, but I don't have that much time20:03
sdaguerguillebert: right, so right now our pypy jobs are just failing left right and center - - 55 fails in 24 hrs / 497 fails in 10 days20:04
sdaguerguillebert: so that's a concern, because that's how we got here in the first place. Without people that are committed to owning this, it's better to just remove it from our ci20:04
rguillebertyeah, that's a build config issue I think20:05
sdagueyep, still an issue, and no owners20:06
*** pc_m has quit IRC20:06
fungiand our build configs are openly maintained by the community, just need people to step up and identify the causes and submit appropriate patches20:06
fungiin a continuous fashion of course, not just as a one-time fix20:07
lifelessrguillebert: hi!20:07
rguilleberthi :)20:07
*** hdd has quit IRC20:07
lifelessrguillebert: the social structure here - individual projects are happy to make their patches etc work with pypy as long as CI is running the jobs20:07
lifelessrguillebert: but when the jobs break systematically - e.g. the current failure - it usually need someone deeply familiar with quirks - e.g. 'must have setuptools X' or whatever to figure it out20:08
lifelessrguillebert: tl;dr weeks or months of no-effort-needed, then the occasional firedrill20:09
fungiand to work within the design limitations of the systems we use to perform this testing as well20:09
rguillebertwell, I can be around and help with that sort of issue :)20:09
*** asrangne has quit IRC20:09
rguillebertas long as it's not a second full-time job :)20:09
*** marun_ is now known as marun20:10
*** asrangne has joined #openstack-infra20:10
*** markvoelker has joined #openstack-infra20:11
rguillebertdoes anyone know why the build uninstalls setuptools in the first place?20:12
jpeelerAJaeger: is there expected lag time for when the testing pipeline changes to just now merged review?20:13
*** kgiusti has quit IRC20:14
rguillebertand, can I run it locally?20:15
lifelessrguillebert: yeah, so there's a pile of context, let me give you a bootstrap20:18
lifelessrguillebert: all the CI jobs run in one-shot VM's20:18
lifelessrguillebert: there's automation to set those up, and they run from a custom base image (which is also automated)20:18
*** mriedem has quit IRC20:18
lifelessrguillebert: its entirely possible to run those locally (though not always simple)20:18
lifelessrguillebert: i'm not the best person to step you through that.20:19
lifelessrguillebert: also no, its not a full time job, not at all20:19
openstackgerritMatthew Treinish proposed openstack/requirements: Add os-testr to global requirements list
rguillebertgood :)20:19
lifelessrguillebert: uhm, why do we uninstall setuptools. I'm not sure we do. sdague ? fungi ?20:20
*** Krinkle|detached is now known as Krinkle20:20
*** baoli has quit IRC20:21
fungilifeless: looks like as a workaround for bug 129056220:21
openstackbug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] - Assigned to Ildiko Vancsa (ildiko-vancsa)20:21
lifelessrguillebert: the top of shows setuptools being upgraded by pip20:21
fungilifeless: specifically, many of the tox envs for pypy add a setuptools>3.420:21
fungiand then it looks like pip wants to uninstall the existing setuptools 2.2 in the virtualenv before replacing it with something later20:22
lifelessrguillebert: the reason it doesn't say 'running for setuptools' is because setuptools is a wheel20:22
lifelessrguillebert: and the pip ui around wheels is stil suboptimal20:22
lifelessfungi: deleting files before unpacking new ones is a common pip thing20:22
lifelessfungi: better than unpacking and then deleting :P20:22
openstackgerritJohn Griffith proposed openstack-infra/project-config: Add functional test job for cinderclient
*** bhunter71 has quit IRC20:23
lifelessclarkb: you said you think you've fixed the current thing?20:23
lifelesslike, is there anything specific for rguillebert to do now, other than lurk here and highlight on pypy?20:23
fungilifeless: too true. though for a while there was a situation where pip wouldn't immediately install the new thing after removing the old thing, thinking it could do it later on, and then run other things needing the deleted thing... mess20:23
sdaguelifeless: it's definitely not fixed20:23
sdagueby the ER graph20:24
lifelessso here's a recent one then20:24
fungilifeless: i think it's something similar, but not necessarily the same as what that bug report was working on20:24
*** Sukhdev has joined #openstack-infra20:25
*** miqui has quit IRC20:25
fungibecause the workaround claims to be for a debian bug fixed in pypy-2.2.1+dfsg-220:25
lifelessrguillebert: so ^ tht log20:25
lifelessrguillebert: shows that we're installing setuptools and the other test requirements20:25
sdague .... I hate media wiki search some times20:25
*** Krinkle is now known as Krinkle|detached20:26
lifelessthen it gets down to the actual install step20:26
rguillebertyeah, looks like setuptools should be installed alone20:26
sdagueanyone have the wiki for the etherpad list somewhere?20:26
rguillebertas in20:26
rguillebertupdate setuptools20:26
fungioh! trusty-updates has 2.2.1+dfsg-1ubuntu0.2 so actually doesn't have that fix still20:26
rguillebertand then do everything else20:26
rguillebertbecause it looks like setuptools gets removed for future updating20:27
lifelessrguillebert: It looks like that, but the pip inner loop there is just horrid UI20:27
lifelessrguillebert: its replacing setuptools immediately: uninstall, install (from a wheel)20:27
*** maishsk is now known as maishsk_afk20:27
*** mriedem has joined #openstack-infra20:27
lifelessone thing we should do I think, to make this analysis easier, is to use pip list not pip freeze20:28
lifelessbecause pip list shows the versions of setuptools and pip20:28
lifelessfreeze doesn't20:28
rguillebert'install', '-U', 'setuptools>3.4', '-r/home/jenkins/workspace/gate-requirements-pypy/test-requirements.txt']20:28
*** soren_ has joined #openstack-infra20:28
*** matrohon has quit IRC20:28
rguillebertthat's what it's doing right?20:28
lifelessrguillebert: yeah20:28
*** apmelton has quit IRC20:28
rguillebert-r and beyond should be done separately20:29
rguillebertI think20:29
*** kmartin has joined #openstack-infra20:29
fungias long as nothing else declares a conflicting install_requires on setuptools20:29
reedsdague, the list of etherpads for the design summit?20:30
* reed hates mediawiki as a whole, not just its search20:31
reedsdague, from link to
sdaguereed I found it20:31
sdagueI was just amazingly surprised that Search "Liberty" only returned 1 page20:32
sdaguewhich was completely unrelated to the summit20:32
lifelesssdague: - replied to your comemnt20:32
sdaguedid the lucene worker die?20:32
reedi rely on firefox history to find wiki pages20:32
lifelessfungi: what pip version will be on that node?20:33
reedmaybe the reboot yesterday left something borked?20:33
sdagueso... if search is that bad, we should probably either delete that box entirely, or make it go to google your thing20:33
sdaguewhich is how I found it eventually20:33
fungilifeless: in the global system environment, or the one bundled by virtualenv and used by tox?20:33
sdaguethe current model just induces anger20:33
reedsearch is not that bad actually20:33
reedlet me try one thing20:33
lifelessfungi: the one being invoked in the failing pip execution20:34
lifelessfungi: its clearly not 6.1.1, so not current pip or current virtualenv20:34
lifelessrguillebert: ok, so I'm going to see if I can put together a patch to make that happen20:34
reedif I search "upstream" i have relevant results20:34
lifelessrguillebert: I'd like to know *why* though, because as I said pip isolated each actual install20:34
reedliberty not so much20:34
lifelessrguillebert: I know its not visible in the UI, but setuptools is actually installed after the 'uninstalling' line20:34
*** weshay has joined #openstack-infra20:34
sdaguelifeless: so... I was pretty sure that was current behavior as of a couple of months ago20:35
lifelessrguillebert: wheel installs are invisible20:35
reed"outreachy"  There were no results matching the query.20:35
reedimpossible, something is broken on that search20:35
lifelesssdague: I don't know when it was fixed, but its fixed in 6.1.1 for sure, and up20:35
lifelesssdague: clarkb too was all paranoid and had to go manually try experiments to convince himself.20:35
reedshould probably file a bug20:35
reedsdague, I have the impression that the search results are just stale20:36
lifelesssdague: but I have limmited care here. Its fixed upstream. If you don't believe it, please file a reproduction upstream, as it is an upstream issue.20:36
fungilifeless: i wish tox logged that, even in the tox logfiles, but it looks like it does not. i'll jump on one of the machines and find out20:36
reedif I search "women" i get results to GnomeOutreachWomen20:36
*** apmelton has joined #openstack-infra20:36
reedwhich was renamed to Outreachy... but search Outreachy and get 020:36
sdaguelifeless: but if you are wrong later, you owes me a drink :)20:38
reedfiling a bug20:38
fungilifeless: currently for bare-trusty workers in hpcloud we seem to have virtualenv 1.11.4 installed, which comes with a bundled pip 1.5.420:38
sdaguelifeless: +A on patch20:38
lifelesssdague: I'm happy to owe you a drink20:39
lifelessfungi: ok, so I think thats a huge problem :)20:39
lifelesscause 1.5.4 is frankly bad20:39
vipulquestion for folks here that may have used the Openstack ansible modules that are in ansible proper.  Some of the modules seem really outdated (quantum) -- any recommendations of other / better implementations ?20:39
fungilifeless: clarkb has merged a patch to bump that to latest virtualenv, but the last successful bare-trusty image update in hpcloud was 101.80 hours ago, before that patch merged20:39
lifelessfungi: rguillebert: I propose that we get latest virtualenv + pip + setuptools on the system as a prelude to running any tox commands20:39
*** rossella_s has quit IRC20:40
lifelessfungi: ok cool.20:40
mordredvipul hi20:40
rguillebertyeah that sounds good20:40
vipulwassup mordred20:40
fungilifeless: this is why clarkb believes his patch has resolved the current pypy failure, we're just waiting for it to propagate to all our providers20:40
mordredvipul we're replacing them. I'll send you a link in 1520:40
*** otter768 has joined #openstack-infra20:40
fungilifeless: so i think this is probably good validation of that20:40
lifelesssdague: FWIW - the pip wheel loop is actually semantically wrong, because where we have open constraints, pip 8 will resolve a common set, which may not be latest, so we'd end up with the wrong wheels anyhow.20:41
mordredvipul you gonna be in Vancouver?20:41
lifelessfungi: alright20:41
fungier, last successful bare-trusty image update in hpcloud-b5 anyway20:41
lifelessrguillebert: so - we're in hurry up and wait stage now, while automation does its thing.20:41
vipulmordred: yes sir..20:41
*** maishsk_afk has quit IRC20:41
fungiwhich is where the example ran20:41
lifelessfungi: is that update pending, or broken?20:41
vipulmordred: i also looked at the shade stuff you showed a few months back.. is that still active?20:41
mordredcool. so, Monday afternoon session on the ansiblr modules20:41
mordredvery much so20:42
fungibut looks like all the bare-trusty images in hpcloud aren't updating for similar duration now, so likely someone merged something 4-5 days ago which broke bare-trusty image builds20:42
mordredthat's the basis of the new work ... it's what you want20:42
fungii'll try to hunt that down20:42
vipulOk great..20:42
vipulmordred: Ansible collaboration day the topic?20:42
clarkbfungi or base image got renamed20:42
sdaguelifeless: the patch has to recheck, because it last ran tests when we were enforcing pypy tests, but hopefully should land within a couple of hours20:42
fungiclarkb: also a distinct possibility20:42
mordredvipul yup!20:42
vipulok i'll be there...20:43
vipulso shade is going into ansible proper i take it?20:43
mordredvipul morning is about deployment, afternoon about modules20:43
mordredthe ansible mods are. shade is run through infra20:44
mordrednodepool is also moving to it20:44
*** otter768 has quit IRC20:45
vipulok good to know.  We'll start tinkering with it20:45
lifelesssdague: yeah, no rush20:46
lifelesssdague: I mean, its to let us hopefully test an alpha codebase :)20:46
*** annegentle has quit IRC20:47
lifelessrguillebert: thank you for stepping up; if you know another pypy fluent person that might be willing to do the same thing, we're all set.20:47
sdagueheh, sure, but I like the idea of actually testing the pip upstream commits to solve our problem in the system that exposes said problem20:47
fungialso interesting, our nodepool image update logs stopped after the 11th, so i'm doing a manual update now to see what's going on there20:47
jeblairfungi: potential logging misconfiguration?20:48
*** changbl has quit IRC20:48
fungijeblair: it's possible20:48
jeblairfungi: istr we landed some changes that touched logging20:48
*** hdd has joined #openstack-infra20:48
*** dustins_ has quit IRC20:48
reed!/story/2000265 not sure it should go to puppet-mediawiki though20:48
rguillebertlifeless, does that mean it's fixed?20:48
reedsdague, ^20:48
fungijeblair: just in hpcloud though. rax seems fine20:49
*** dannywil_ has quit IRC20:49
lifelessrguillebert: right now the proposed fix (new virtualenv) is stalled behind an unrelated problem20:49
*** isviridov_away is now known as isviridov20:49
lifelessrguillebert: so we'll get that fixed, and then see if its fixed. If it is, \o/. If it isn't "ping rguillebert" :)20:49
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Rename stackforge/octavia -> openstack/octavia
rguillebertI'm available :)20:49
sdaguerguillebert: thanks!20:50
openstackgerritMerged openstack/requirements: Uncap pytz and block pycadf 0.6.2
*** dannywilson has joined #openstack-infra20:51
*** Krinkle|detached is now known as Krinkle20:51
*** jaypipes has quit IRC20:53
*** _nadya_ has quit IRC20:53
reeddoes OpenStack need to respond to Venom vulnerability?20:54
*** Krinkle is now known as Krinkle|detached20:54
*** krtaylor has quit IRC20:55
fungireed: to what audience(s) are you thinking?20:55
*** ianw has joined #openstack-infra20:56
*** bswartz has quit IRC20:56
*** dannywilson has quit IRC20:56
*** dboik_ has joined #openstack-infra20:56
*** marun has quit IRC20:56
reedI was about to  write something there20:56
*** baoli has joined #openstack-infra20:56
*** marun has joined #openstack-infra20:56
*** dprince has quit IRC20:57
sdakefungi any chance you could approve this new project ajaeger already reviewed?20:58
*** dkranz has quit IRC20:59
*** dboik has quit IRC20:59
*** peristeri has quit IRC20:59
fungireed: probably someone could write an ossn similar to but i'd suggest starting a [security] tagged thread on the dev ml referencing that (and maybe referencing the thread which is going on the ops ml)20:59
*** dboik_ has quit IRC21:00
*** ildikov has quit IRC21:03
fungireed: but in short, venom is not a vulnerability in openstack software, it's a vulnerability in some software commonly used alongside openstack. we aren't the ones who provide fixes for it21:05
*** ayoung has quit IRC21:06
fungifixes come from the xen project, possibly by way of distributions packaging the software21:06
*** aysyanne has quit IRC21:06
*** gordc has quit IRC21:06
fungispecifically in the form of patches to or patched versions of qemu21:07
sdakefungi thanks bro21:08
*** chlong has quit IRC21:08
*** dizquierdo has joined #openstack-infra21:09
*** dstanek has quit IRC21:10
*** dannywilson has joined #openstack-infra21:10
jpeelerfungi: is there expected lag time for when the testing pipeline changes to what has been merged in project-config?21:11
*** rguillebert has quit IRC21:11
*** rguillebert has joined #openstack-infra21:11
fungijpeeler: generally no more than 45 minutes21:11
*** ildikov has joined #openstack-infra21:13
jpeeleryep, it has changed!21:13
jpeelerfungi: can sudo be used without issue?21:14
openstackgerritMerged openstack-infra/project-config: Cognitive - Machine Learning as a Service
openstackgerritMerged openstack-infra/project-config: Cognitive - Machine Learning as a Service
*** dstanek has joined #openstack-infra21:14
*** markvoelker has quit IRC21:14
fungijpeeler: for the job you were updating, it looked like you weren't using the revoke-sudo builder macro, so it should be able to sudo on the worker to do whatever it wants21:14
*** tiswanso has quit IRC21:15
*** ldnunes has quit IRC21:16
*** openstackgerrit has quit IRC21:22
*** openstackgerrit has joined #openstack-infra21:22
*** arielb has quit IRC21:23
*** rfolco has quit IRC21:24
*** markvoelker has joined #openstack-infra21:24
*** mwagner_lap has quit IRC21:25
clarkbfungi: any luck with hpcloud trusty nodes?21:25
*** tnovacik has quit IRC21:26
fungilogging is busted, not sure why yet. my test build is still underway21:26
fungiit's already well into cloning the git cache21:26
fungiwell, not cool. i'm sure it'll fail, it's just going to take plenty of time to do so apparently21:27
clarkboh, but it sounds so promising :)21:27
fungiyeah, it's going to spend an hour getting my hopes up first before dashing them on the floor like so many broken dreams21:28
*** annegentle has joined #openstack-infra21:28
*** fawadkhaliq has joined #openstack-infra21:28
*** Krinkle|detached is now known as Krinkle21:29
*** baoli has quit IRC21:29
*** jlanoux has quit IRC21:29
*** Sukhdev has quit IRC21:29
*** teran has joined #openstack-infra21:30
openstackgerritDebo~ Dutta proposed openstack-infra/project-config: Add python-cognitiveclient project for Cognitive
*** Sukhdev has joined #openstack-infra21:30
*** yamahata has quit IRC21:32
*** tonytan4ever has quit IRC21:33
*** shashankhegde has quit IRC21:34
*** yfried is now known as yfried|afk21:35
clarkbgreghaynes: so I am reading the GitPython source and it seems to be shelling out to git to do clones so now I am really confused21:37
clarkbgreghaynes: is what we call in zuul and that uses the Git object to run git clone21:37
greghaynesclarkb: hrm, you were mentioning something about an order of op's issue IIRC21:38
greghaynes(I havent fully paged in the context)21:38
greghaynesshould be simple to make a testcase for zuul_cloner that repros I think21:38
clarkbgreghaynes: well my original theory was that GitPython implemented git clone itself and did so poorly21:39
*** e0ne has quit IRC21:39
clarkbgreghaynes: but it seems to fork a `git clone` to do the work so my theory is not really sound anymore21:39
*** e0ne has joined #openstack-infra21:40
*** hashar has quit IRC21:40
openstackgerritJeff Peeler proposed openstack-infra/project-config: kolla: add sudo for
*** erlon has quit IRC21:41
fungii'm sort of sorry about this, but not really... <<<brace for impact>>>21:41
openstackgerritJeremy Stanley proposed openstack-dev/cookiecutter: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-dev/specs-cookiecutter: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/devstack-gate: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/elastic-recheck: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/gerritbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/infra-specs: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/jeepyb: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/jenkins-job-builder: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/nodepool: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/openstackid: Replace ci.o.o links with docs.o.o/infra
jeblairoh dear21:42
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-accessbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ansible: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-apparmor: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-askbot: Replace ci.o.o links with docs.o.o/infra
jeblairsome of our things have links don't they?21:42
openstackgerritJeremy Stanley proposed openstack-infra/puppet-asterisk: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-bugdaystats: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-bup: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-cgit: Replace ci.o.o links with docs.o.o/infra
fungi"some" yes21:42
openstackgerritJeremy Stanley proposed openstack-infra/puppet-diskimage_builder: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-drupal: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-elastic_recheck: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-elasticsearch: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-etherpad_lite: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-exim: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-gerrit: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-gerritbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-github: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-graphite: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-haveged: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-httpd: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-iptables: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-jeepyb: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-jenkins: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-kerberos: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-kibana: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-lodgeit: Replace ci.o.o links with docs.o.o/infra
sdaguejeblair: there are no links. The cake is a lie21:43
openstackgerritJeremy Stanley proposed openstack-infra/puppet-log_processor: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-logrotate: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-logstash: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-mailman: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-mediawiki: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-meetbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-mysql_backup: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-nodepool: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openafs: Replace ci.o.o links with docs.o.o/infra
sdagueand fungi is about to win at stackalytics for the day21:44
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openstackci: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openstackid: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-packagekit: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-pip: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-planet: Replace ci.o.o links with docs.o.o/infra
*** mattfarina has quit IRC21:44
openstackgerritJeremy Stanley proposed openstack-infra/puppet-project_config: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-redis: Replace ci.o.o links with docs.o.o/infra
jeblairwe're all going to win on review counts21:44
openstackgerritJeremy Stanley proposed openstack-infra/puppet-releasestatus: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-reviewday: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-simpleproxy: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-snmpd: Replace ci.o.o links with docs.o.o/infra
fungiyou'll catch up with me tomorrow, sdague21:44
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ssh: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ssl_cert_check: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-statusbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-storyboard: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-subunit2sql: Replace ci.o.o links with docs.o.o/infra
*** shashankhegde has joined #openstack-infra21:45
openstackgerritJeremy Stanley proposed openstack-infra/puppet-sudoers: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-tmpreaper: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ulimit: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-unattended_upgrades: Replace ci.o.o links with docs.o.o/infra
jeblairfungi: i'm not sure about that -- it might be a while before anyone else gets a test node :)21:45
openstackgerritJeremy Stanley proposed openstack-infra/puppet-unbound: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-user: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/puppet-zuul: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/pypi-mirror: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/shade: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/storyboard: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/storyboard-webclient: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/tripleo-ci: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/yaml2ical: Replace ci.o.o links with docs.o.o/infra
openstackgerritJeremy Stanley proposed openstack-infra/zuul: Replace ci.o.o links with docs.o.o/infra
mordredoh my21:46
fungiit's fewer than 100 repos, at any rate. i'm sure we'll chew through the jobs21:46
greghaynesoh my indeed21:46
*** yfried|afk is now known as yfried21:46
*** mriedem has quit IRC21:47
dougwigwhoa, someone ate their wheaties.21:47
fungizuul was looking bored21:48
sdagueyeh, it was at a pretty low job count21:48
sdaguecan't let it get out of shape21:48
lifelessfungi: what was the tl;dr on the image?21:48
*** ociuhandu has quit IRC21:48
fungilifeless: my test update just finished failing and i'm looking through the log now21:49
*** spzala has quit IRC21:49
fungiClientException: Unknown Error (HTTP 500)21:50
fungibubbling up from novaclient21:50
mtreinishfungi: shooting for #1 in liberty, I have faith you can take the top spot this cycle21:50
openstackgerritMerged openstack-dev/cookiecutter: Replace ci.o.o links with docs.o.o/infra
fungithis is at the point is should have tried to make a snapshot21:50
fungidid we maybe have something change around the snapshot method (either in nodepool or in novaclient)?21:51
fungithe actual template system seems to puppet successfully21:51
*** weshay has quit IRC21:52
jeblairthis is going throw my +/- ratio out of whack21:53
mordredjeblair: :)21:53
fungicould that maybe be a quota problem in hpcloud?21:54
fungilike we have too many snapshots or something?21:54
nibalizerwe're seeing hpcloud errors downstream so...21:54
jeblairfungi: i hope you left one error in the stack to make sure we're really reviewing it21:54
mordredjeblair: hehe21:55
mordredjeblair: you know what - I AM actually looking at the darned things - because I'm a moron21:55
clarkbgreghaynes: did we hit the git thing on trusty?21:55
*** dkranz has joined #openstack-infra21:55
jeblairmordred: i am too.  i mean, i'm putting my name on it.21:55
fungijeblair: i suspect there may be an error or two. i'm mostly going back through the non openstack-infra ones i proposed updates to because those are less regular21:55
clarkbgreghaynes: I just cobbled together a thing and it seems to be ok so far21:55
jeblairfungi: i'm saving suggestions for improvement for a later time21:56
greghaynesclarkb: pretty sure yes, I want to say all the dib tests were failing21:56
mordredjeblair: yah. still ... I feel mildly silly anyway21:56
clarkbfungi: we did switch to shade but its using the openstack clients from shade so not much really changed21:56
clarkbfungi: also I think that was less than 100 hours ago21:56
greghaynesyea, if you look at that traceback its not even hitting shade21:57
*** yfried is now known as yfried|afk21:57
*** hdd has quit IRC21:57
*** jgrimm is now known as zz_jgrimm21:57
greghaynesthe big change was really using os-client-config21:57
fungioh! here's a possibility... could the cli not be in sync with the daemon for image update?21:57
kambizdoes anyone know if there's a way to get ops in a channel on freenode, if the channel is not registered and there are no ops in the channel ?21:57
mordredfungi: whatcho mean?21:58
jeblairfungi: found a -1.  ;)21:58
mordredoh - well... _installing_ shade will have upgraded theclient libs21:58
mordredjeblair: nice catch!21:58
fungijeblair: awesome!21:58
fungimordred: in the past we've had at least one instance of the nodepool cli was calling into a different code path than the daemon was using to do some functions21:58
clarkbfungi: no they should be the same since we rebooted21:59
mordredjeblair, fungi: shade has docs - should the homepage be those and not the system-config?21:59
fungimordred: probably21:59
clarkbgreghaynes: more progress, things get weird after doing the repo.reset() in the cloner21:59
clarkbgreghaynes: before that it actually looks pretty sane21:59
mordredif so - I found a -1 to!21:59
jeblairmordred: yeah, though that was part of the 'improvements' that i was leaving for later21:59
jeblairmordred: that's true for several of those21:59
mordredwell, I'm keeping the -1 since I left it anyway :)22:00
fungimordred: though the homepage on it went to the system-config docs previously (we didn't have the "right" url for it before either)22:00
jeblairi'll +2 it either way.  mordred and fungi can fight it out :)22:00
jeblairmordred: you should +2 fungis and then write a bunch of patches to improve the links ;)22:00
fungior feel free to hijack mine22:00
*** dkranz has quit IRC22:01
fungii won't mind (or likely even notice)22:01
*** e0ne has quit IRC22:01
*** dkranz_ has joined #openstack-infra22:01
*** teran_ has joined #openstack-infra22:01
fungiso... yeah this is calling directly from the provider manager into novaclient22:02
*** Rockyg has joined #openstack-infra22:02
openstackgerritMerged openstack-infra/puppet-apparmor: Replace ci.o.o links with docs.o.o/infra
fungitrying to do a createImage call22:02
jeblairmordred: you missed 18323922:02
mordredjeblair: I may not subscribe to that repo22:02
clarkbgreghaynes: so flipping the reset and prune order in the cloner works22:02
mordredjeblair: repo?22:02
clarkbjeblair: ^ any idea why that would fix things?22:02
jeblairmordred: jjb22:03
mordredjeblair: yeah - I don't gertty jjb22:03
clarkbI need to read up on prune22:03
greghaynesclarkb: prune docs make it sound like it should be safe...22:03
jeblairclarkb: help me context switch please22:03
*** teran has quit IRC22:04
clarkbjeblair: the zuul-cloner breakage that greghaynes ran into with diskimage-buidler on our dib built test nodes22:04
*** markvoelker has quit IRC22:04
jeblairmordred: well, you can just ctrl-o "183239" and review it :)22:04
clarkbjeblair: git.exc.GitCommandError: 'git branch -d -r origin/fetch_master' returned with exit code 128 returns stderr: 'fatal: HEAD not found below refs/heads!'22:04
clarkbjeblair: that happens if we git clone, git reset, then git prune. It does not happen if we git clone, git prune, git reset22:04
clarkbmy reading of this is that order shouldn't matter and yet..22:04
greghaynesclarkb: I wonder if the prune is just surfacing the bug, and its actually broken reguardless22:05
*** SotK has joined #openstack-infra22:05
jeblairclarkb: ok.  iirc, we decided we should (a) make dib not leave weird things in the repo and (b) make zuul-cloner more resilient if weird things are left in the repo22:05
greghaynesjeblair: yep, this is b22:05
*** bhuvan has quit IRC22:05
clarkbprune relies on
clarkband now I think I know the issue22:05
clarkbyou have to make a new repo object each time because lol22:06
mordredjeblair: ctrl-o did not work for me22:06
clarkbI will confirm and write a patch22:06
*** isviridov is now known as isviridov_away22:06
jeblairmordred: what happened?22:06
mordredjeblair: it said "change not found"22:06
jeblairclarkb: cute22:06
openstackgerritJames Carey proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks
clarkbhrm except zuul is doing that for us22:06
jeblairmordred: try again?22:06
jeblairmordred: and if you have debug logs from that, i'd love a copy22:07
mordredjeblair: now it worked22:07
mordredjeblair: and no - I didn't start with debug this time22:07
jeblairmordred: any reason you don't always run with debug?22:07
jeblairmordred: it will really help me fix bugs for you :)22:07
fungimordred: jeblair: for 183305 those urls exist and are the counterparts to the original links. is the concern that they were the wrong links originally and we should point to the application manuals for those rather than their pages in our system configuration manuals?22:08
mordredjeblair: I'll try to remember that in the future22:08
*** doug-fish has left #openstack-infra22:08
mordredjeblair: you know, you could have it email you tracebacks ...22:08
jeblairfungi: oh, did i get that backwards, sorry.22:08
jeblairfungi: i was wrong; review updated.22:09
mordredclarkb: for lxml - shouldn't we ensure => absent for lxml?22:09
clarkbmordred: no because zuul still relies on that package22:09
clarkbmordred: its a avoid trouble with puppet problem22:10
mordredclarkb: but you removed the sentry22:10
clarkbmordred: but it will conflict with the entry zuul uses22:10
jeblairmordred: i just double checked by opening a random nova change, and it worked.  :/22:10
mordredclarkb: this is on puppet-nodepool22:10
clarkbmordred: yes, I know22:10
mordredok. that seems strange to me and like evil is happening22:11
clarkbmordred: the idea is that zuul and nodepool are coninstallable, which is why we guarded the python-lxml install previously22:11
clarkbmordred: but that means we cannot ensure present in one and ensure absent in the other22:11
mordredso - shouldn't we need this same thing for zuul22:11
clarkbpuppet will explode22:11
*** bhunter71 has joined #openstack-infra22:11
fungii'm getting the feeling that this createImage 500 error from the api in hpcloud is causing nodepool not to actually write out the corresponding image logs there22:11
clarkbmordred: no because pip generally figures it out for us22:11
mordredif pip install -U is causing carnage on nodepool it's going to on zuul too22:11
*** Swami has quit IRC22:11
jeblairfungi: logging should be continuous22:11
clarkbmordred: no it won't because no shade on zuul22:11
fungihrm, so not buffered until complete?22:11
jeblairfungi: correct22:11
clarkbmordred: the problem is shade specifically installs keystoneclient.22:12
mordredclarkb: what depends on lxml?22:12
jeblairfungi: unless something changed22:12
clarkbmordred: so right now on zuul we don't have to go down that upgrade path and its fine22:12
clarkbmordred: it will only cause carnage if we update things (like we did with shade) such that pip decides it should upgrade lxml22:12
mordredright. but if the two are supposed to be co-instalalble22:13
clarkbmordred: remember pip dep resolution doesn't look at the whole tree at once, so if you hide lxml behind something that does not need an upgrade you are fine22:13
mordredthen shade will carnage the zuul install if they co-nstall22:13
mordredand we know it22:13
*** dizquierdo has quit IRC22:13
clarkbmordred: no zuul will just get latest lxml via nodepool22:13
clarkbmordred: it will work fine22:13
mordredexcept it will still have python-lxml instaled22:13
clarkbmordred: yes22:13
*** spzala has joined #openstack-infra22:13
mordredseems half-done to me and I'm pretty sure it's going to break someone22:14
mordredbecause I do not bleieve pip will do the right thing with pyhton-lxml installed ia pacakges22:14
mordredI might be wrong - but it seems like playing with fire22:14
*** D1337d has joined #openstack-infra22:15
mordredfungi: HP cloud is doing maint for the thing22:15
mordredfungi: so our fail there may be related22:15
openstackgerritJeff Peeler proposed openstack-infra/project-config: kolla: add sudo for
clarkbmordred: two things, first pip installs to /usr/local and thats earlier in python path so that bit should just work. Without puppet package sanity I can't think of a better solution and still preserve coinstallability22:16
fungimordred: oh... maybe the real problem is shadowed by venom maintenance22:16
fungigot it22:16
clarkbmordred: we could force zuul to do the unnecessary build lxml from source22:16
*** eharney has quit IRC22:17
*** sdake has quit IRC22:17
*** ddieterly has quit IRC22:17
mordredclarkb: I just remember carnage when we've had competing system and python things ... it may be fixed now - but I haven't proven to myself that it's fixed22:18
*** sdake has joined #openstack-infra22:18
clarkbmordred: if we do do that we would haev to make this as an intermediate step anyways before ensuring absent otherwise puppet will break22:18
mordredclarkb: although your words make sense and is how I'd like to have it work22:18
clarkbmordred: what kind of carnage? I have never seen it reak22:18
clarkbmordred: the only thing similar to carnage I have seen is if yu upgrade pip during a puppet run pip provider in puppet gets confused about where to get pip in its path22:18
clarkbmordred: but thats mostly a puppet process caching things problem22:18
*** bknudson has quit IRC22:20
*** spzala has quit IRC22:20
*** nelsnelson has quit IRC22:20
jpeelersorry to nag, but can i get a quick review on ?22:20
*** annegentle has quit IRC22:20
*** ddieterly has joined #openstack-infra22:21
*** ivar-lazzaro has quit IRC22:21
*** ayoung has joined #openstack-infra22:22
*** ivar-lazzaro has joined #openstack-infra22:24
*** markvoelker has joined #openstack-infra22:25
*** rlandy has quit IRC22:25
openstackgerritMerged openstack-infra/infra-manual: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/gerritbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack/requirements: Allow testing with specific pips
openstackgerritMerged openstack-infra/infra-specs: Replace ci.o.o links with docs.o.o/infra
*** ildikov has quit IRC22:28
clarkbjeblair: new thing I am noticing about zuul-cloner is that it kills refs/heads/HEAD somewhere22:28
*** ivar-lazzaro has quit IRC22:28
openstackgerritMerged openstack-infra/jeepyb: Replace ci.o.o links with docs.o.o/infra
*** ildikov has joined #openstack-infra22:29
clarkboh wait, no its complaining that the root HEAD is not defined in refs/HEADS22:30
clarkber refs/heads22:30
*** ivar-lazzaro has joined #openstack-infra22:30
clarkbwhich should be ok if I do the full cloen I think22:30
clarkbits just in an unhappy state halfway through our edits22:31
clarkbgreghaynes: jeblair I have figured it out22:33
clarkbwill patch with a lengthy commit message22:33
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Add priority efforts section
openstackgerritMerged openstack-infra/elastic-recheck: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/nodepool: Replace ci.o.o links with docs.o.o/infra
clarkbya, so when we do a reset HEAD points to refs/remotes/origin/master it does not point to refs/heads/SOMETHING. git branch explodes in this case which is the error message we saw: stderr: 'fatal: HEAD not found below refs/heads!'22:34
clarkbso we can fix this by pruning first then reseting so that git branch is not run until later after we have done a checkout of the final desired state22:34
* SpamapS uploads python-shade to Debian unstable22:35
SpamapSzigo: hey, we need to talk about adding python3 packages to the openstack clients. :)22:35
clarkbgreghaynes: jeblair so this really has little to do with dib and is a zuul-clner bug, we just happen to only tickle it if prune is not a noop22:36
zigoSpamapS: I'd like to do so, but before this happens, we need Python3 support for pysaml2 !!!22:36
zigoSpamapS: That's currently the only remaining blocker.22:37
zigoSpamapS: It's needed to fix python-keystoneclient.22:37
zigoThen when this is done, I can add it to all packages.22:37
zigoSpamapS: Would you help?22:37
SpamapSzigo: patches awaiting merge, or more?22:37
zigoSpamapS: *huge* work to be done on this upstream.22:37
SpamapSzigo: yeah I can always find a little time for python3 :)22:37
zigoSpamapS: I started to write it, and the patch I wrote ended up being like 5000 long.22:38
zigoSpamapS: Then I did some crap which broke stuff, and gave up...22:38
zigoSpamapS: I'd need to restart it.22:38
*** teran has joined #openstack-infra22:38
greghaynesclarkb: yea, the dib thing definitely isnt a 'bug', just silly22:38
*** soren_ has quit IRC22:38
clarkbgreghaynes: ya its weird enough that we never hit this corner before :)22:39
*** teran_ has quit IRC22:39
clarkbanyways I should have a patch up in a bit, want to make sure I grok this before writing the commit message that justifies changing the order22:39
clarkbbut changing the order does fix22:39
zigoSpamapS: Would you try?22:39
SpamapSzigo: poking now22:39
openstackgerritMerged openstack-infra/infra-manual: Add ChangeLog and AUTHORS to .gitignore
zigoSpamapS: I can send you my patch file.22:40
openstackgerritJeff Peeler proposed openstack-infra/project-config: kolla: add sudo for
clarkbSpamapS: zigo I wonder if the keystone auth lib split out thing will remove need for pysaml2 in py3k for the base use case?22:40
clarkbSpamapS: zigo might be worth finding out from morganfainberg or jamielennox|away as that may speed things along with less effort22:41
openstackgerritMerged openstack-infra/puppet-bugdaystats: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/project-config: Replace ci.o.o links with docs.o.o/infra
*** marun has quit IRC22:41
SpamapSclarkb: no, the thing everybody needs is the thing that needs saml.22:41
sdakecould a couple cores fix our busted gate by reviewing:
zigoclarkb: As much as I know, the keystone_authtoken thing is *planned* to move out from keystoneclient to the middleware, but that's not done yet.22:41
openstackgerritMerged openstack-infra/puppet-github: Replace ci.o.o links with docs.o.o/infra
*** otter768 has joined #openstack-infra22:41
zigoclarkb: We'd better just fix pysaml2 anyway...22:42
sdakethanks jeblair22:42
clarkbSpamapS really? I dont think we use it22:43
openstackgerritMerged openstack-infra/puppet-ansible: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-elasticsearch: Replace ci.o.o links with docs.o.o/infra
SpamapSclarkb: hm, actually, now that I read deeper, it's just a plugin.22:43
morganfainbergzigo: auth token has been removed from ksc22:43
openstackgerritMerged openstack-infra/puppet-askbot: Replace ci.o.o links with docs.o.o/infra
SpamapSzigo: ^22:43
openstackgerritMerged openstack-infra/puppet-drupal: Replace ci.o.o links with docs.o.o/infra
mordredzigo, SpamapS: also - keystoneauth has been release the first version22:43
morganfainbergzigo: we are waiting post summit for a 2.x release of ksc because we don't want to be fixing broken things at the summit.22:44
zigomordred: keystoneauth ? As a separate package?22:44
openstackgerritMerged openstack-infra/puppet-elastic_recheck: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-asterisk: Replace ci.o.o links with docs.o.o/infra
SpamapSso yeah, I don't even see pysaml2 in the reqs for master keystoneclient22:44
morganfainbergKeystoneauth has been split which is just session object etc.22:44
mordredzigo: yah - intended to be used by other libs to auth against keystone instead of keystoneclient22:44
morganfainbergAnd we have a separate package just for saml auth plugin in ksc22:44
mordredso that keystoneclientis about talking to the keystone rest api22:44
morganfainbergSo the base one doesn't need py saml22:44
zigomordred: Wonderful.22:44
mordredit's very exciting22:44
SpamapSmorganfainberg: but keystoneauth will still need pysaml2 for clouds that use saml yes?22:44
zigoSpamapS: Then we need all clients to switch to keystoneauth instead of ksc.22:45
*** annegentle has joined #openstack-infra22:45
mordredSpamapS: only the keystoneauth plugin for saml22:45
mordredwhich will be different22:45
morganfainbergSpamapS: only if you want the plugin22:45
openstackgerritMerged openstack-infra/puppet-exim: Replace ci.o.o links with docs.o.o/infra
mordredkeystoneauth is puggable22:45
openstackgerritMerged openstack-infra/puppet-accessbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-etherpad_lite: Replace ci.o.o links with docs.o.o/infra
morganfainbergAnd that is in a separate package.22:45
openstackgerritMerged openstack-infra/puppet-cgit: Replace ci.o.o links with docs.o.o/infra
SpamapSRight, so can just make that its own package that has no python3 equiv until pysaml2 is ported.22:45
openstackgerritMerged openstack-infra/puppet-gerritbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-bup: Replace ci.o.o links with docs.o.o/infra
mordredit's almost like we're doing this right :)22:45
openstackgerritMerged openstack-infra/puppet-kerberos: Replace ci.o.o links with docs.o.o/infra
*** otter768 has quit IRC22:46
morganfainbergKeystoneauth has 0.1.0 out today. Please file bugs (Lp) or complaints for what we need to fix before we move to 1.x which will be the stable release we start from.22:46
morganfainbergOur target for 1.x is around milestone 122:46
sdakethanks fungi22:46
morganfainbergTo add to g-r so we can get everyone using it where appropriate in Liberty.22:46
zigoGosh, how come there's argparse in requirements.txt of keystoneauth ... :(22:47
morganfainbergmordred: phsaw doing it right or doing it in a way that people stop complaining at me ;).22:47
zigoLet's get rid of these, PLEASE !22:47
clarkbzigo: because python2.622:47
clarkbzigo: but its fine22:47
*** annegentle has quit IRC22:47
zigoclarkb: I thought we're not supporting it anymore...22:48
morganfainbergzigo: it isn't done. We just did the split most requirements will get removed as we continue cleanup.22:48
*** ajo has quit IRC22:48
clarkbzigo: the libs still support it until the last server release that supports it goes away22:48
morganfainbergKeystoneauth needs to work with 2.6 until clients don't need to.22:48
openstackgerritMerged openstack-infra/puppet-gerrit: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-graphite: Replace ci.o.o links with docs.o.o/infra
morganfainbergIf a client needs 2.6, ksa needs 2.622:48
clarkbmorganfainberg: right22:49
morganfainbergBut we do plan on eliminating Oslo.serialization from the requirements and a few others.22:49
morganfainbergBefore 1.x release.22:49
SpamapSmordred: /win 2022:50
SpamapSmordred: forgot to ^5 you, then go to window 2022:50
*** Swami has joined #openstack-infra22:50
openstackgerritMerged openstack-infra/storyboard: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-httpd: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-jeepyb: Replace ci.o.o links with docs.o.o/infra
zigoSpamapS: Ok, so to go back to reality: if we get a patch for pysaml2 to support Py3, then the next day, I can start adding Py3 support to client packages.22:51
zigo(ie: it wont need to wait until we do a major work everywhere...)22:51
zigoSo, in other words, it seems more realistic to just port pysaml2 to Py3 now.22:51
zigoThe other Py3 blocker for packages is qpid (impacting oslo.messaging and it's dependencies), though we had a thread about it on the list, so I guess you know already.22:52
mordreddrop qpid22:52
SpamapSzigo: has to be done anyway22:52
SpamapSzigo: you could just make the drivers separate packages. :)22:53
morganfainbergSpamapS: ++22:53
SpamapSqpid will be deprecated in Liberty, but not removed until "M"22:53
SpamapSso I'd say do that anyway22:54
SpamapSpython-oslo-messaging-qpid or something.22:54
*** hemna is now known as hemnafk22:55
SpamapSpysaml2's test suite requires mongodb22:55
*** banix has joined #openstack-infra22:55
* SpamapS stabs22:55
mtreinishSpamapS: heh, so it's webscale testing :)22:57
openstackgerritClark Boylan proposed openstack-infra/zuul: Fix git prune order in zuul-cloner
clarkbgreghaynes: jeblair jhesketh ^22:57
SpamapSmtreinish: should use the /dev/null driver instead22:57
openstackgerritJeff Peeler proposed openstack-infra/project-config: make kolla jobs non-voting
*** markvoelker has quit IRC22:58
morganfainbergSpamapS: it's ok, I'm going to make Keystoneauth require mongodb but only for the release you work with *shiftyeyes*22:59
*** sabeen1 has quit IRC22:59
*** ociuhandu has joined #openstack-infra22:59
greghaynesclarkb: fun. lgtm22:59
mordredclarkb: lg23:00
*** ashleighfarnham has quit IRC23:00
*** pabelanger has quit IRC23:01
jheskethclarkb: lgtm, approved23:01
*** rcarrillocruz has quit IRC23:01
*** bswartz has joined #openstack-infra23:01
clarkbawesome, ty23:02
SpamapSmorganfainberg: :|23:04
*** redrobot has quit IRC23:04
clarkbmordred: 182895 is another recent "fix this thing" change23:04
*** ayoung has quit IRC23:04
*** Trozz has quit IRC23:04
clarkbjhesketh: ^ not sure how interested you are in log cleanup :)23:04
*** markvoelker has joined #openstack-infra23:05
*** redrobot has joined #openstack-infra23:05
*** redrobot is now known as Guest1989523:05
*** SamYaple has quit IRC23:05
*** freerunner has quit IRC23:06
jheskethclarkb: I'm interested in anything that helps infra :-).23:07
*** ashleighfarnham has joined #openstack-infra23:07
pleia2jhesketh: ^523:07
*** ddieterly has quit IRC23:07
*** davideagnello has quit IRC23:08
jheskethclarkb: doesn't elasticsearch go back 10 days or something? Should we clean out everything older than 14 days instead?23:08
*** asrangne__ has joined #openstack-infra23:08
clarkbjhesketh: the indexed logs inside of elasticsearch go back 10 days. The logs we are cleaning up here are the normal text file logs for the elasticsearch service itself23:09
clarkbjhesketh: basically its turtles, but good suggestion since we may need to know why $thing that is indexed was indexed23:09
SpamapSzigo: whats the story on python-memcached ? Looks like it may be dead?23:09
clarkbjhesketh: I can update to 14 days23:09
jheskethRight, yes that was my point23:09
jheskethI think it likely won't matter but 14 days makes more sense23:10
*** chlong has joined #openstack-infra23:10
openstackgerritClark Boylan proposed openstack-infra/puppet-elasticsearch: Cleanup old ES logs
clarkbjhesketh: yup ^ its a sane idea23:10
*** asrangne has quit IRC23:11
jheskethclarkb: thanks, lgtm23:11
*** davideagnello has joined #openstack-infra23:11
jeblairclarkb: zuul-cloner change makes sense.  thanks23:11
clarkbjeblair: it ended up being far more time to debug than fix :)23:13
clarkbthose are the best bugs23:13
*** panda has quit IRC23:13
*** dstufft has quit IRC23:14
*** bhuvan has joined #openstack-infra23:14
*** panda has joined #openstack-infra23:14
*** dangers is now known as dangers_away23:14
*** whoops has quit IRC23:14
*** teran_ has joined #openstack-infra23:14
*** salv-orlando has joined #openstack-infra23:15
*** dstufft has joined #openstack-infra23:15
*** atan8 has quit IRC23:16
*** jamespage has quit IRC23:17
*** gnuoy has quit IRC23:17
*** teran has quit IRC23:17
jlvillalIs there a way in Zuul or Jenkins to see the results of previous partial test runs?  Test ran and finished some of the tests, but then before everything was done a new patch was uploaded.  Can I still find that old job?23:18
*** gnuoy has joined #openstack-infra23:19
*** jamespage has joined #openstack-infra23:19
*** yamamoto has joined #openstack-infra23:20
openstackgerritMerged openstack-infra/puppet-haveged: Replace ci.o.o links with docs.o.o/infra
clarkbjlvillal: yes, simplest way is probably to search for it in logstash, but the log urls are deterministic. so for its last 2 of change number/change number/patchset/pipeline/jobname/job-uuids23:20
openstackgerritMerged openstack-infra/puppet-kibana: Replace ci.o.o links with docs.o.o/infra
jlvillalclarkb: Thanks.  I will go look.23:21
openstackgerritMerged openstack-infra/puppet-logrotate: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-mediawiki: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/shade: Improve error message on auth_plugin failure
openstackgerritMerged openstack-infra/puppet-mailman: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-jenkins: Replace ci.o.o links with docs.o.o/infra
*** hdd has joined #openstack-infra23:22
openstackgerritMerged openstack-infra/puppet-iptables: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-meetbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-mysql_backup: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-nodepool: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-openstackid: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-packagekit: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-openafs: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-pip: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-planet: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-simpleproxy: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-reviewday: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-releasestatus: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-project_config: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-redis: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/project-config: Group akanda sub-projects
mordredmerge bomb23:23
mordredjhesketh, clarkb: I think we should be able to do this one now: 16761023:25
*** atan8 has joined #openstack-infra23:26
*** bhuvan has quit IRC23:26
mordredclarkb: also, I went ahead and dropped my -1 on your other one - I still think we should go do _something_ there23:26
*** bhuvan has joined #openstack-infra23:26
mordredbut maybe not today23:26
*** banix has quit IRC23:26
*** atan8 has quit IRC23:26
*** yamamoto has quit IRC23:27
fungiyeah, retrying image updates in hpcloud, it's completely hosed in there at the moment23:28
fungii guess their maintenance is still underway23:29
*** enikanorov2 has quit IRC23:29
*** Longgeek has quit IRC23:30
*** Longgeek has joined #openstack-infra23:30
*** Sukhdev has quit IRC23:30
clarkbmordred: the removal of python-yaml seems odd since nodepool depends on that too23:31
*** banix has joined #openstack-infra23:31
*** sigmavirus24 is now known as sigmavirus24_awa23:33
openstackgerritMerged openstack-infra/puppet-nodepool: Install lxml from source to accomodate pip -U
*** Krinkle is now known as Krinkle|detached23:33
clarkbmordred: are we sure that removing that won't make nodepool unhappy?23:33
clarkbmordred: otherwise it looks good, the diskimage_builder module is installed etc23:34
openstackgerritMerged openstack-infra/puppet-ssh: Replace ci.o.o links with docs.o.o/infra
mordredclarkb: nodepool is going to get it from its pip install anyway though, no?23:35
*** dimtruck is now known as zz_dimtruck23:36
clarkbmordred: ya, so why would we need to have added the python-yaml package?23:36
openstackgerritMerged openstack-infra/puppet-ssl_cert_check: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-statusbot: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-snmpd: Replace ci.o.o links with docs.o.o/infra
* fungi wonders if that package was a hold-over from when we weren't doing pip install -U23:38
*** zhiwei has joined #openstack-infra23:39
mordredI would vote on that ^^23:39
fungifor a while there, iirc, we installed distro packages for any python libs we could and just relied on pip to fill in the otherwise unpackaged gaps23:39
mordredclarkb: where did we get on nibalizer's server/template stuff?23:41
openstackgerritMerged openstack-infra/puppet-storyboard: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-lodgeit: Replace ci.o.o links with docs.o.o/infra
clarkbmordred: its up for review, I retopic'd it then VENOM and all that fun happened23:41
openstackgerritMerged openstack-infra/puppet-logstash: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-log_processor: Replace ci.o.o links with docs.o.o/infra
*** bhuvan has quit IRC23:42
*** angela-s has left #openstack-infra23:43
mordredoh - they both landed23:44
mordredso now next step is moving things from server to template?23:44
clarkbya to flip the manifest they use as their base23:45
mordredbecause server and template _should_ be equiv now, yeah?23:45
clarkbmordred: I +2'd the dib module one23:45
clarkbmordred: server calls template so ya its just an unnecessary layer23:45
mordredoh - server is just calling template23:46
mordredwell, there is one more - we now need to pass the exim flag to template in server23:46
*** ashleighfarnham has quit IRC23:48
openstackgerritMerged openstack-infra/zuul: Fix git prune order in zuul-cloner
openstackgerritMonty Taylor proposed openstack-infra/system-config: Pass exim flag to template rather than exim class
clarkbgreghaynes: one we get new images built (tomorrow morning assuming hpcloud cooperates) you can remove your git clones from the job23:49
openstackgerritMerged openstack-infra/puppet-subunit2sql: Replace ci.o.o links with docs.o.o/infra
*** ajmiller_ has quit IRC23:51
greghaynesclarkb: ok23:51
clarkbfungi: is hpcloud just not making images at all?23:51
nibalizermordred: yep so now its 'git grep openstack_server' and find and fix things23:52
openstackgerritMonty Taylor proposed openstack-infra/system-config: Pass exim flag to template rather than exim class
nibalizerwhich is gonna be a fir bit of churn23:52
nibalizeroh that looks interesting23:52
clarkbbut we can constrain it which is the nice thing23:52
*** rguillebert has quit IRC23:52
openstackgerritMerged openstack-infra/puppet-tmpreaper: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-sudoers: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-ulimit: Replace ci.o.o links with docs.o.o/infra
openstackgerritMerged openstack-infra/puppet-unattended_upgrades: Replace ci.o.o links with docs.o.o/infra
*** _nadya_ has joined #openstack-infra23:54
openstackgerritMerged openstack-infra/puppet-unbound: Replace ci.o.o links with docs.o.o/infra
*** _nadya_ has quit IRC23:58
*** ildikov has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!