Monday, 2013-09-30

*** guohliu has quit IRC00:03
*** matsuhashi has joined #openstack-infra00:05
dhellmannsdague, mordred : I'm going to just start incrementing X every release ;-)00:15
mordreddhellmann: ++00:33
*** crank has quit IRC00:34
*** yaguang has joined #openstack-infra00:42
*** alexpilotti has quit IRC01:01
*** nosnos has joined #openstack-infra01:05
*** xchu has joined #openstack-infra01:07
*** darraghb has joined #openstack-infra01:13
*** ericw has joined #openstack-infra01:27
*** alexpilotti has joined #openstack-infra01:27
*** yaguang has quit IRC01:31
*** ericw has quit IRC01:32
*** ekarlso has quit IRC01:33
*** ericw has joined #openstack-infra01:59
*** jpeeler has joined #openstack-infra02:00
*** jpeeler has quit IRC02:00
*** jpeeler has joined #openstack-infra02:00
*** ericw has quit IRC02:03
*** darraghb1 has joined #openstack-infra02:13
*** darraghb has quit IRC02:15
*** matsuhashi has quit IRC02:16
*** mrda has joined #openstack-infra02:20
*** matsuhashi has joined #openstack-infra02:21
*** darraghb has joined #openstack-infra02:23
*** darraghb1 has quit IRC02:25
*** cyeoh_ is now known as cyeoh02:29
*** ericw has joined #openstack-infra02:29
*** matsuhashi has quit IRC02:29
*** matsuhashi has joined #openstack-infra02:30
*** ericw has quit IRC02:34
*** xchu has quit IRC02:53
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
*** ericw has joined #openstack-infra02:57
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
*** guohliu has joined #openstack-infra03:24
*** crank has joined #openstack-infra03:30
*** changbl has quit IRC03:34
*** crank has quit IRC03:35
*** dcramer_ has joined #openstack-infra03:36
openstackgerritMatthew Treinish proposed a change to openstack-infra/reviewstats: Fix typo in mkoderer username
*** darraghb has left #openstack-infra03:44
*** changbl has joined #openstack-infra03:47
lifelessmordred: ugh, the pbr tests decode the setup output? grumble.03:53
lifeless(subprocess itself doesn't..)03:53
openstackgerritlifeless proposed a change to openstack-dev/pbr: Add a test for command registration.
*** changbl has quit IRC03:54
*** thingee_zzz is now known as thingee03:55
mordredlifeless: aroo?03:57
lifelessmordred: I checked for b" testr " in the output.03:57
lifelessmordred: subprocess normally gets you bytes out03:58
lifelessmordred: the failure on py33 was apparently a bytes-text mismatch, or so someone else suggested03:58
mordredyup. sounds about right03:58
*** dims has quit IRC03:58
lifelesswhich means something is magically decoding the subprocess output03:58
lifelessso just a surprise03:58
mordredyes. the run_command function03:58
mordredI believe we added that in one of the attempts to deal with 2.6-3.303:59
mordredI'm open to other suggestios03:59
mordredthe whole thing is a nightmare03:59
mordredand I hate it03:59
lifelesswill poke later03:59
mordredeverytime I have to touch it, it makes me want to cry04:00
mordredwell, let the current stack of things merge first -there is a patch that mucks with the mechanics on run_shell_command that needs to go in to fix ... somebody04:00
mordredI think osx04:00
mordredor somethingt04:00
mordredmaybe windows04:00
mordredthen going back through and re-sensible-izing it sounds GREAT04:01
mordredand would be SUPER SUPER welcome04:01
*** changbl has joined #openstack-infra04:07
*** thingee is now known as thingee_zzz04:14
mordreddstufft: it looks like pip might normalize _ to - when processing requirements names?04:17
mordredis that correct? or is something on my end doing something unexpected?04:17
*** AlexF has joined #openstack-infra04:22
*** dkliban has quit IRC04:26
*** dkliban has joined #openstack-infra04:29
*** crank has joined #openstack-infra04:31
mordreddstufft, lifeless: I just tried devpi again and it was able to make it through installing all of openstack requirements this time04:34
*** crank has quit IRC04:35
dstufftmordred: all of python packging does that04:38
*** vipul-away is now known as vipul04:38
dstufftmordred: it's a bit more than that, i'm trying to simplify it a bit04:38
mordreddstufft: ok. my testing of a local mirror which is static files at file:/// seems to be confuseled by it04:39
* mordred just wants to cry04:40
*** dkehn_ has joined #openstack-infra04:40
*** salv-orlando_ has joined #openstack-infra04:40
*** michchap_ has joined #openstack-infra04:42
dstufftmordred: a directory full of file:// ?04:42
dstufftmordred: what does your file structure look like04:42
*** zigo_ has joined #openstack-infra04:43
*** bnemec has joined #openstack-infra04:43
*** rahmu_ has joined #openstack-infra04:43
*** SergeyLukjanov has joined #openstack-infra04:43
mordredsorry, that wasn't clear04:43
mordreddstufft: are stuff makes stuff that looks like this:04:43
*** ArxCruz_ has joined #openstack-infra04:44
*** sdake__ has joined #openstack-infra04:44
*** jpipes has joined #openstack-infra04:44
*** pabelanger_ has joined #openstack-infra04:44
mordredso, on my test run thing, django_compressor is appropriately in django_compressor04:44
mordredso, on my test run thing, django_compressor is inappropriately in django-compressor04:44
*** belliott_ has joined #openstack-infra04:44
mordredbut on pypi.o.o it's correct04:44
*** Reapster_ has joined #openstack-infra04:45
dstufftmordred: so your local filesysem has BASEDIR/django_compressor/index.html ?04:45
mordredno. I think that's the issue.04:45
dstufftwhat does your local filesystem have04:45
mordredthe exact same as that web server, dirs full of tarballs04:46
mordredexcept the webserver is doing autoindex04:46
mordredand on the filesystem, it needs an exact match first time,04:46
dstufftoh oops, I didn't click on that04:46
dstuffta file04:46
dstufftI mean04:46
mordredbecause there is nothing to fall back to crawling04:46
dstufftI just looked at the top level04:46
dstufftpip has some logic to fake an index.html I think04:46
mordrednot on file:/// urls04:46
mordredit actually explcitly does not use it on those04:47
mordredfor reasons unbeknownst to me04:47
dstufftmordred: are you using --index-url or --file-links04:47
dstuffter find-links04:47
*** tvb has quit IRC04:47
dstufftmordred: moment04:47
mordredk. thanks (this is probably an error somewhere in our stuff, but knowing what the 'right' thing is will certainly be helpful)04:48
dstufftmordred: I think you need to add an index.html04:48
mordredfor the file:// ?04:48
dstufftmaking sure04:48
mordredeasier thing in this case would actually probably be just starting an apache and pointing it at a dir04:49
*** ArxCruz has quit IRC04:49
*** salv-orlando has quit IRC04:49
*** jaypipes has quit IRC04:49
*** michchap has quit IRC04:49
*** pabelanger has quit IRC04:49
*** bnemec_ has quit IRC04:49
*** afazekas has quit IRC04:49
*** dkehn has quit IRC04:49
*** zigo has quit IRC04:49
*** sdake_ has quit IRC04:49
*** rahmu has quit IRC04:49
*** belliott has quit IRC04:49
*** Reapster has quit IRC04:49
*** salv-orlando_ is now known as salv-orlando04:49
*** belliott_ is now known as belliott04:49
*** rahmu_ is now known as rahmu04:49
dstufftmordred: can you do an install with -vvv04:50
mordredit'll take a bit - but yeah04:50
dstufftI think for --index-url it literally looks for an index.html file to be inside the directory04:52
*** salv-orlando has quit IRC04:59
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
mordreddstufft: that'll add the -v -v05:11
mordreddstufft: actually, it looks like we are doing the index.html for the local dirs05:11
mordreddstufft: so I'm curious to see what the discovery sequence looks like05:11
*** basha has joined #openstack-infra05:12
*** jcoufal has joined #openstack-infra05:16
*** Ryan_Lane has joined #openstack-infra05:29
*** crank has joined #openstack-infra05:31
openstackgerritMonty Taylor proposed a change to openstack-infra/pypi-mirror: Ensure that wheel is installed
*** Ryan_Lane has quit IRC05:34
*** moted has quit IRC05:36
*** tvb has joined #openstack-infra05:41
notmynamemordred: <-- issues with the swift ppa being gone. seems that it was getting some amount of use05:55
*** matsuhas_ has joined #openstack-infra05:57
*** AlexF has quit IRC05:58
*** matsuhashi has quit IRC05:58
*** nosnos_ has joined #openstack-infra05:58
*** nosnos has quit IRC05:59
*** Ryan_Lane has joined #openstack-infra05:59
*** Ryan_Lane has quit IRC06:06
*** AlexF has joined #openstack-infra06:07
*** crank has quit IRC06:09
mordrednotmyname: oh excellent06:11
mordredttx: ^^06:11
*** nicedice has quit IRC06:12
notmynamemordred: ya. the hard thing is that if it were used people were getting bad (ie old, unsupported) code06:12
notmynamemordred: perhaps we have an old doc somewhere that is still pointing to it06:12
notmynamemordred: do you know if any stats are tracked about how much LP resources like that are used?06:12
mordrednotmyname: I'm guessing that's probably the case06:13
*** AlexF has quit IRC06:13
mordredand no, I don't believe there was any stat for that (I know ttx was trying to look in to whether it was in use or not and couldn't get a good answer)06:13
notmynamemordred: yup. <-- search for ppa06:13
notmynamewhich is an old doc that needs updating anyway06:13
*** SergeyLukjanov has quit IRC06:14
mordredalso is up to date enough to have zuul06:14
mordredbut still references the ppas06:14
notmynamemordred: so maybe if we're lucky it's just a new user trying to do stuff for the first time. if we're not lucky, it's a prod setup trying to update06:14
mordredwell, if they are a prod user looking for ubuntu packages, we should be able to point them to cloud-archive06:15
mordredI'm answering about the ppas being deprecated for a while and recently removed06:15
mordredI'll mention cloudarchive - unless you want to respond instead?06:15
notmynamenope. go for it06:16
notmyname /punt06:16
*** afazekas has joined #openstack-infra06:16
*** davidhadas_ has quit IRC06:17
*** pblaho has joined #openstack-infra06:21
*** ekarlso has joined #openstack-infra06:21
*** basha has quit IRC06:23
*** mrda has quit IRC06:24
*** ekarlso has quit IRC06:26
*** ekarlso has joined #openstack-infra06:26
notmynamemordred: still up?06:28
notmynamemordred: since I'll be traveling tomorrow, I'll send an email.06:30
*** basha has joined #openstack-infra06:33
*** Ryan_Lane has joined #openstack-infra06:34
*** Ryan_Lane has quit IRC06:38
*** odyssey4me has joined #openstack-infra06:45
*** michchap_ has quit IRC06:54
*** michchap has joined #openstack-infra06:54
*** yaguang has joined #openstack-infra06:56
*** matsuhas_ has quit IRC06:57
*** dkliban has quit IRC06:58
ttxnotmyname, mordred: wow. I was expecting something like this07:00
*** nosnos has joined #openstack-infra07:02
*** nosnos_ has quit IRC07:02
ttxbut there was no way of marking PPAs "deprecated" in a way those users would have noticed, anyway07:02
openstackgerritDirk Mueller proposed a change to openstack/requirements: Raise/Relax WebTest requirement to match Keystone
*** Ryan_Lane has joined #openstack-infra07:04
*** crank has joined #openstack-infra07:04
*** matsuhashi has joined #openstack-infra07:05
*** davidhadas has joined #openstack-infra07:06
*** michchap has quit IRC07:07
*** markmc has joined #openstack-infra07:08
*** basha has quit IRC07:09
*** Ryan_Lane has quit IRC07:09
*** michchap has joined #openstack-infra07:09
mordredttx: I think we should do some searching in the wiki for PPA07:12
*** flaper87|afk is now known as flaper8707:13
*** slong has quit IRC07:13
mordredttx, notmyname: (
mordredis what messed those guys up07:14
*** tkammer has joined #openstack-infra07:14
ttxmordred: I'll do some wiki searching07:15
*** davidhadas_ has joined #openstack-infra07:33
*** davidhadas has quit IRC07:33
*** Ryan_Lane has joined #openstack-infra07:35
saschpezul: ping, you still owe me a package on the gate :-)07:37
*** Ryan_Lane has quit IRC07:39
*** matsuhashi has quit IRC07:40
*** matsuhashi has joined #openstack-infra07:42
*** matsuhashi has quit IRC07:42
*** AJaeger has joined #openstack-infra07:42
*** SergeyLukjanov has joined #openstack-infra07:44
*** matsuhashi has joined #openstack-infra07:44
*** matsuhashi has quit IRC07:45
*** kpavel has joined #openstack-infra07:47
*** matsuhashi has joined #openstack-infra07:48
*** matsuhashi has quit IRC07:48
*** davidhadas_ has quit IRC07:50
*** boris-42 has joined #openstack-infra07:51
*** yassine has joined #openstack-infra07:59
*** Ryan_Lane has joined #openstack-infra08:05
*** derekh has joined #openstack-infra08:05
*** hashar has joined #openstack-infra08:05
*** hashar_ has joined #openstack-infra08:09
*** hashar has quit IRC08:10
*** hashar_ is now known as hashar08:10
*** flaper87 is now known as flaper87|afk08:12
*** matsuhashi has joined #openstack-infra08:13
*** mkerrin has joined #openstack-infra08:14
*** Ryan_Lane has quit IRC08:14
*** dizquierdo has joined #openstack-infra08:14
*** flaper87|afk is now known as flaper8708:23
*** fbo_away is now known as fbo08:25
*** dizquierdo has quit IRC08:28
*** basha has joined #openstack-infra08:38
*** boris-42 has quit IRC08:39
*** Ryan_Lane has joined #openstack-infra08:40
*** Ryan_Lane has quit IRC08:44
*** jcoufal has quit IRC08:59
*** dizquierdo has joined #openstack-infra09:02
*** johnthetubaguy has joined #openstack-infra09:13
*** tkammer has quit IRC09:26
*** Ryan_Lane has joined #openstack-infra09:41
konganyone can see this problem, AttributeError: 'module' object has no attribute 'find_resourceid_by_name_or_id'. Jenkins in stable/grizzly is failed because of it. Please see
*** Ryan_Lane has quit IRC09:41
*** Ryan_Lane has joined #openstack-infra09:41
*** thomasbiege has joined #openstack-infra09:41
*** yaguang has quit IRC09:42
*** boris-42 has joined #openstack-infra09:42
*** yaguang has joined #openstack-infra09:44
*** Ryan_Lane has quit IRC09:46
*** thomasbiege has quit IRC09:50
*** guohliu has quit IRC09:50
ttxnotmyname, mordred: more fun at
uvirtbotLaunchpad bug 1233042 in swift "swift-core repository failed to fetch" [Undecided,Invalid]09:54
*** boris-42_ has joined #openstack-infra10:07
*** boris-42 has quit IRC10:07
*** Ryan_Lane has joined #openstack-infra10:11
*** Ryan_Lane has joined #openstack-infra10:11
*** Ryan_Lane has quit IRC10:16
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add dynamic string and choice params.
*** pcm_ has joined #openstack-infra10:30
*** pcm_ has quit IRC10:32
*** pcm_ has joined #openstack-infra10:33
*** michchap has quit IRC10:34
*** yaguang has quit IRC10:34
*** michchap has joined #openstack-infra10:34
*** Ryan_Lane has joined #openstack-infra10:42
*** amotoki has joined #openstack-infra10:44
*** Ryan_Lane has quit IRC10:46
*** HenryG has joined #openstack-infra10:47
*** alexpilotti has quit IRC10:59
*** ArxCruz_ has quit IRC11:08
*** Ryan_Lane has joined #openstack-infra11:12
*** Ryan_Lane has quit IRC11:16
*** thomasbiege has joined #openstack-infra11:22
*** thomasbiege has quit IRC11:22
*** tvb has quit IRC11:25
*** tvb has joined #openstack-infra11:26
sorenI seem to remember that you guys found a way to cache the virtualenvs used during tox runs. Is that still functional?11:26
sdaguesoren: I thought it was just a feature of new tox, but mordred would know for sure once he's up11:36
*** lcestari_ has joined #openstack-infra11:39
*** kong has quit IRC11:41
sorensdague: Just saw this from 2011: Mentions pre-caching virtualenvs11:42
*** Ryan_Lane has joined #openstack-infra11:43
sdaguesoren: you have PIP_CACHE enabled?11:43
sorensdague: I have a pip cache.11:43
sorensdague: The problem with pip caches is that pip still has to interrogate pypi to learn what the newest version is.11:44
sdagueright, fair11:44
sorenWell, interrogate pypi and install the packages again, but it seems like talking to pypi is the lengthy bit.11:44
*** rfolco has joined #openstack-infra11:46
sorenI'd just like to cache the virtualenvs and do e.g. daily expiration + expiration every time pip-requirements changes or something like that.11:46
*** Ryan_Lane has quit IRC11:47
*** dims has joined #openstack-infra11:48
*** jcoufal has joined #openstack-infra11:56
*** dkranz has joined #openstack-infra11:56
*** cody-somerville has quit IRC11:56
*** AlexF has joined #openstack-infra12:02
*** boris-42_ has quit IRC12:08
*** sandywalsh has joined #openstack-infra12:08
*** ArxCruz has joined #openstack-infra12:11
*** dims has quit IRC12:12
*** dims has joined #openstack-infra12:12
*** Ryan_Lane has joined #openstack-infra12:13
*** hemnafk has quit IRC12:15
*** hemnafk has joined #openstack-infra12:16
*** Ryan_Lane has quit IRC12:17
*** pabelanger_ has quit IRC12:24
*** adalbas has joined #openstack-infra12:24
*** pabelanger has joined #openstack-infra12:24
*** hashar has quit IRC12:26
*** thomasm has joined #openstack-infra12:26
*** Clabbe has joined #openstack-infra12:26
*** sdake has quit IRC12:27
ClabbeWhat happens in openstack-ci if a git change cant be automatically merged?12:27
*** sdake has joined #openstack-infra12:27
*** basha has quit IRC12:27
*** adalbas has quit IRC12:31
*** hashar has joined #openstack-infra12:32
*** schwicht has joined #openstack-infra12:35
*** weshay has joined #openstack-infra12:37
*** hashar has quit IRC12:39
*** hashar has joined #openstack-infra12:39
*** Ryan_Lane has joined #openstack-infra12:43
*** adalbas has joined #openstack-infra12:43
*** acabrera has joined #openstack-infra12:43
*** Ryan_Lane has quit IRC12:47
fungiClabbe: it doesn't get merged, the corresponding gerrit review gets a comment stating as much, and someone needs to rebase/fix it before it gets reviewed and approved again12:55
fungi(usually with another patchset to the same change in gerrit)12:56
*** amotoki has quit IRC12:58
mordredsoren: so ...12:58
mordredsoren: tox should re-use its virtualenv in the same workspace12:58
mordredbut if you're talking about reusing a virtualenv between various jenkins workspaces or something12:59
mordredno, virtualenv itself gets screwy12:59
*** dprince has joined #openstack-infra12:59
mordredwe did a BUNCH of hacks to get that to work and eventually just ditched it and went for running our pypi mirror12:59
mordredsoren: if you don't want to set up the pypi mirroring stuff we did, I recommend running devpi12:59
mordredwhich I just verified works last night13:00
Clabbefungi: thx13:01
openstackgerritA change was merged to openstack-infra/config: Add #stackforge-libra to gerritbot
*** pblaho has quit IRC13:03
*** dkranz has quit IRC13:03
*** pblaho has joined #openstack-infra13:04
*** julim has joined #openstack-infra13:07
*** mriedem has joined #openstack-infra13:08
*** julim has joined #openstack-infra13:09
*** OlivierSanchez has joined #openstack-infra13:10
*** nosnos has quit IRC13:13
*** dcramer_ has quit IRC13:13
*** Ryan_Lane has joined #openstack-infra13:14
*** rnirmal has joined #openstack-infra13:15
*** dkranz has joined #openstack-infra13:16
*** Ryan_Lane has quit IRC13:18
*** matty_dubs|gone is now known as matty_dubs13:20
*** OlivierSanchez has quit IRC13:20
*** openstackgerrit has quit IRC13:22
*** openstackgerrit has joined #openstack-infra13:22
*** tizzo has joined #openstack-infra13:23
mordredttx: another fun thing to search for is pip-requires13:25
mordredhub_cap: this is out of date:
notmynamemordred: about to tag a release for python-swiftclient so that we can get beyond the most painful dependency issues13:28
openstackgerritafazekas proposed a change to openstack-infra/devstack-gate: Using the root user for tempest run
mordrednotmyname: woot!13:30
mordrednotmyname: I believe that that means that no more openstack projects will depend on it and I can remove it from the mirror13:31
notmynamemordred: writing my tag message13:31
afazekasmordred,lifeless :
*** OlivierSanchez has joined #openstack-infra13:33
sorenmordred: devpi, eh? Thanks for the pointer.13:33
sorenmordred: I suppose I could just configure Jenkins not to wipe the workspace, but the sort of bugs caused by that can be soo hard to spot.13:34
mordredsoren: I recommend wiping the workspace and using somesort of pypi mirror13:36
*** adalbas has quit IRC13:36
*** OlivierSanchez has quit IRC13:40
*** amotoki has joined #openstack-infra13:40
notmynamemordred: final proposal on the etherpad13:41
mordrednotmyname: yes. that looks good to me13:42
notmynamemordred: ok, thanks13:42
*** OlivierSanchez has joined #openstack-infra13:43
notmynamemordred: tag pushed to gerrit. sending email now13:44
*** Ryan_Lane has joined #openstack-infra13:44
*** dcramer_ has joined #openstack-infra13:44
mordredsweet. I'm going to watch the mirror jobs that are kicked off by this so I can see if anyone else is left touching d2to113:44
*** jergerber has joined #openstack-infra13:45
mriedemdoes anyone know if the gate is supposed to be installing devstack from master against a stable/grizzly patch or not?13:45
mordredas of yesterday, the count was python-swiftclient, oslo.sphinx, diskimage-buidler and triple-image-elements - but I believe all 4 have now made new releases13:45
uvirtbotLaunchpad bug 1231989 in openstack-ci "quantumclient being tested against devstack master rather than devstack stable/grizzly" [Undecided,Confirmed]13:45
mriedemsdague: ping13:45
sdaguemriedem: pong?13:45
mriedemsdague: looking at this:
mriedemit's blocking a patch to fix quantumclient in stable/grizzly13:46
mordredmriedem: client libs currently are tested against master13:46
mordredmriedem: we don't take stable patches to client libs13:46
mriedemmordred: and master has neutronclient but stable/grizzly has quantumclient13:46
*** tizzo has quit IRC13:47
mordredmriedem: can you explain more about the patch you're wanting to land?13:47
mriedemmordred: this is a cliff/pyparsing issue with quantumclient13:47
mordredyeah, we have to ninja that13:47
mriedemmordred: yeah, let me start at the beginning...13:47
mordredcan you point me at the patch?13:47
mriedemmordred: this is my nova stable/grizzly patch:
*** AlexF has quit IRC13:47
mordred(quantumclient is a very special snowflake)13:47
mriedemthat fails in the check b/c of quantumclient deps for cliff13:47
mordredyes. with you so far13:48
mriedemwhich is being addressed here:
mriedembut that's failing because of this bug:
uvirtbotLaunchpad bug 1231989 in openstack-ci "quantumclient being tested against devstack master rather than devstack stable/grizzly" [Undecided,Confirmed]13:48
*** Ryan_Lane has quit IRC13:48
mriedemfrom the check logs on my nova patch: "error: Installed distribution pyparsing 1.5.7 conflicts with requirement pyparsing>=2.0.1"13:49
mriedemmost people here are probably still having nightmares from the cliff/pyparsing issue from last week?13:49
mriedemso in this patch, thomasm is simply removing the pyparsing requirement from quantumclient so cliff can pull it in implicitly:
thomasmmriedem, Yeah. I am about to revert the 3rd patch to the second, given the stuff, and my only wanting to change the thing causing trouble for a backport.13:51
mriedembut then jenkins fails because it's running devstack on master which is looking for the neutron bash completion file but it should be using quantumclient13:51
mordredthomasm, mriedem ok. this is the deal13:51
mordreda) please do remove d2to113:51
mordredb) also please update to match the current in the projects, namely: setup_requires=['pbr'], pbr=True)13:52
mordredc) jenkins tests for the quantumclient patch will and are expected to fail13:52
mordredIf we're happy with this patch, and someone can verify that it does what we think it does (by hand)13:53
mordredthen I will force merge it and release a new quantumclient to pypi13:53
*** adalbas has joined #openstack-infra13:53
notmynamemordred: when you are on a plane, do you use the wifi to do code reviews or do you download everything up front?13:54
mordrednotmyname: wifi13:54
mriedemthomasm: i'm assuming you'll revert your patch to set 2 and change as mordred points out?13:54
* notmyname has a 5 hour flight today :-(13:54
mordrednotmyname: although I also tend to download all the code first13:54
mordrednotmyname: I just wrote a script yesterday that you can use to make sure that you have all of the code in all of the openstack repos up to date on your laptop!13:55
mriedemthomasm: once you get those changes up i can cherry pick them locally and run an install to verify it's OK13:55
*** flaper87 is now known as flaper87|afk13:55
mordrednotmyname: I recommend the wifi. it makes the flight much more productive time13:55
notmynamemordred: called "download ALL THE THINGS!!!!"13:55
mordredespecially for a cross-country13:55
*** OlivierSanchez is now known as ojacques13:57
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Add the semver documentation
mordredmriedem: awesome. let me know when it's ready and I'll get it in13:59
*** julim has quit IRC13:59
mordreddhellmann: thanks for the bullet catch. the semver doc is different than upstream - most notably it ports in PEP440 semantics for how to deal with pre-release versions14:00
*** basha has joined #openstack-infra14:00
mriedemmordred: thanks14:01
*** julim has joined #openstack-infra14:01
*** che-arne has joined #openstack-infra14:01
*** dkranz has quit IRC14:02
*** dguitarbite has joined #openstack-infra14:02
openstackgerritA change was merged to openstack-infra/reviewstats: Fix typo in mkoderer username
*** pblaho has quit IRC14:04
*** ojacques has quit IRC14:04
*** dkliban has joined #openstack-infra14:05
thomasmmriedem, Yes, I will14:05
thomasmmriedem, Doing that now14:05
mriedemthomasm: ok, thanks, i'll see the change when it comes up14:05
*** pblaho has joined #openstack-infra14:05
thomasmmriedem, thanks! =]14:05
*** OlivierSanchez has joined #openstack-infra14:06
dhellmannmordred: do the upstream folks not want to address pre-release versioning?14:06
*** AlexF has joined #openstack-infra14:06
* dhellmann wonders why we want a separate copy of that document14:06
*** adalbas has quit IRC14:06
mordreddhellmann: they address is differently14:09
mordreddhellmann: in ways that are incompatible with both python and distro versioning14:09
mordreddhellmann: the one in pbr is, I believe, going to be step one towards writing a pep14:09
dhellmannmordred: oh, that's helpful14:09
mordreddhellmann: I was more snarky about it, but lifeless and I hashed it out for a while over the weekend14:09
dhellmannmordred: what will the new pep address?14:10
mordreddhellmann: I think my main beef isn't so much the version itself, but the fact that their use of - as a seperator between main version and pre-release version is potentially confusing to a human looking at the version14:10
mordreddhellmann: the pep would basically be pep440 semantic versioning guidelines for python14:11
mordredso, if a python project claims to use semantic versioning, this is how that's implemented14:11
*** dguitarbite has quit IRC14:11
mordredthere isn't _that_ much that has to change from upstream, honestly14:11
mordredthe important bits (the actual releases) are spot on14:11
mordredbut the two main changes are actually being stricter about what you can and can't do in a pre-release section, and changing how that section is signfied14:12
mordredto X.Y.Z.(a|b|rc)[0-9]+14:13
dstufftwell PEP440 also doesn't say that 2.1 and 2.2 must be compatible14:13
dstufftit sort of suggests it14:13
*** odyssey4me has quit IRC14:14
dstufftbut i'd think with a "PEP440 based semver" we'd remove the suggestion and just make PEP440 define the version spec, and redirect to PEPWhatever for the semantics14:14
mordredthat's why I like your suggestion of an additional pep to talk about what the versions should _mean_14:14
mordredwhere PEP440 talks about how they should be physically constructed14:14
*** Ryan_Lane has joined #openstack-infra14:14
thomasmmordred, mriedem quantumclient branch is already updated with neutron's and it was given a new set of requirements:
mordredthis will be my first pep - I'm gonna have to go read up more on the process14:15
dhellmannis it worth pointing out the differences between what pbr and the proto-pep say to do and what the upstream version says, to ensure a casual reader doesn't miss the differences14:15
thomasmmordred, mriedem:
mordreddhellmann: yeah. that's a good idea14:16
dhellmannmordred: that can be a second changeset, if you like14:16
*** DennyZhang has joined #openstack-infra14:16
mordreddhellmann: I'm hoping that maybe some people will iterate on the doc for a bit after it's in and before we do a pep14:16
dstufftmordred: essentially it's write the PEP, submit it to the PEP eidtors (Nick is one of them), then post it on distutils-sig, set the BDFL-Delegate to Nick, discuss on mailing list, adjust with feedback until discussion is over or you're ready and then ask for pronouncement from the BDFL Delegate14:17
dstufftthat's it in a nutshell14:17
mriedemthomasm: good point, i forgot that mark was working on that14:17
thomasmLet me recheck my stable fix. =]14:18
dstufftmordred: and if you make any major changes post it again so peolpe can view the changes easily14:18
dstufftbefore acceptance14:18
*** changbl has quit IRC14:18
dstufftmordred: packaging PEPs that don't touch CPython are easier than ones that do (assuming the idea is sane :) )14:18
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Move savanna under openstack org
*** Ryan_Lane has quit IRC14:19
*** dkranz has joined #openstack-infra14:19
*** adalbas has joined #openstack-infra14:19
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Move savanna under openstack org
*** esker has joined #openstack-infra14:19
*** tvb has quit IRC14:19
eskerHi All... How might one get a project listed here:
*** anteaya has joined #openstack-infra14:20
eskerMust it be fully recognized as "incubated" first?14:20
*** OlivierSanchez is now known as ojacques14:22
mordredesker: nope. there's all sorts of crazy stuff there. what sort of thing do you want listed?14:22
eskermordred: thanks... this:
eskermordred: FWIW, headed down the incubation path... just haven't fully prepped it yet.14:23
*** tkammer has joined #openstack-infra14:24
*** markmcclain has joined #openstack-infra14:25
mordredesker: so, what you want to do is go to: look for the "Part of" field and put openstack there14:26
eskermordred... ah, okay... had missed that.  Thanks!14:27
mordredesker: sure thing!14:27
*** rcleere has joined #openstack-infra14:28
*** markmcclain has quit IRC14:30
*** markmcclain has joined #openstack-infra14:34
jog0I am getting timeouts on logstash.o.o today14:37
*** rcleere_ has joined #openstack-infra14:37
bashacan someone look at
bashaThe failures doesnt seem to be coz of the change14:37
*** rcleere has quit IRC14:37
*** rcleere_ is now known as rcleere14:37
jog0so elastic-recheck is down and timing out14:37
bashaand rechecks are'nt working14:37
jeblairjog0: elasticsearch is 502ing for me14:38
jog0jeblair: hmm I can go to but its slow14:39
jog0I don't see any 502s14:40
*** AlexF has quit IRC14:40
jeblairjog0: "14:40
jeblairIf it helps, I received a 502 Proxy Error from: api/search"14:40
jeblairjog0: is what eventually happens after several minutes14:40
mordreddstufft: you know things about both packaging and django ...14:40
*** thomasem has joined #openstack-infra14:40
mordreddstufft: all of our django-related thigns have this cargo-culted into their
mordredfor scheme in install.INSTALL_SCHEMES.values():14:40
mordred    scheme['data'] = scheme['purelib']14:40
dstufftmordred: yea, that's because distutils doesn't have package_data IIRC14:41
jog0jeblair: hmm, anyway back to finishing up my slides for today14:41
dstufftdata files get installed somewhere different then where .py files do14:41
dstufftpackage_daa files get installed alongside14:41
mordreddstufft: but setuptools does, no?14:41
dstufftDjango won't depend on setuptools14:41
mordredah. but so if it's an external project that uses django and does depend on setuptools, this is not needed14:42
dstufftshouldn't be, you just need to properly setup package_data14:42
*** AlexF has joined #openstack-infra14:42
mordredawesome. well, that we do14:42
mordredsweet! thank you!14:42
*** thomasm has quit IRC14:42
*** blamar has joined #openstack-infra14:42
*** DennyZhang has quit IRC14:44
*** Ryan_Lane has joined #openstack-infra14:45
*** datsun180b has joined #openstack-infra14:46
jeblairthe elasticsearch cluster status is 'green'14:46
*** dkranz has quit IRC14:48
*** Ryan_Lane has quit IRC14:49
jog0jeblair: hmm still seeing timeouts14:50
fungigreen timeouts. greaaaat14:51
jeblairjog0: agreed14:51
jeblairfungi: agreed.  :)14:51
jog0it may be related to how fast new data gets processed14:53
jog0because I am cheacking if new job is in elasticSearch when it timesout14:53
mordredwow. I just submitted patches to every stackforge project that's still referencing d2to114:54
mordredblerg, is what I have to say14:54
fungielasticsearch and the logstash workers don't seem to be particularly heavily loaded14:55
clarkbelasticsearch may have gone sideways did we check status?14:55
Alex_Gaynormordred: so the docs tests for oslo.version don't appear to be passing, known?14:55
mordredAlex_Gaynor: yeah - I noticed on that patch - lemme fix14:55
*** tizzo has joined #openstack-infra14:56
*** basha has quit IRC14:57
clarkbelasticsearch4 is not part of the cluster, I think something happened14:58
Alex_GaynorYay. A patch was just approved which fixes a bug with 163 rechecks!14:58
mordredAlex_Gaynor: WOOT14:59
jeblairclarkb: good morning!15:00
clarkbjeblair: morning15:00
jeblairclarkb: i checked status, it said green...15:00
*** che-arne has quit IRC15:00
jeblairclarkb: how did you find es4 was gone?15:00
mordredAlex_Gaynor: patch submitted for oslo.version which should fix the doc issue, and I rebased the sequence of mark-patches on top of it15:00
openstackgerritVictor Sergeyev proposed a change to openstack-dev/hacking: Add H234 check to avoid (x)range() function usage
clarkbjeblair: yeah, it is grean because we have one replica so no data lost with one node gone. I saw green and 13 nodes in cluster, we should have 1415:00
clarkbjeblair: then I fired up bigdesk against a local port forward to one of the cluster nodes15:01
clarkbI am going to restart elasticsearch on elasticsearch415:01
*** thingee_zzz is now known as thingee15:01
*** dkranz has joined #openstack-infra15:01
jeblairclarkb: ok.  what's the math to get to 14?15:01
clarkbjeblair: 8 worker nodes + 6 indexer nodes15:02
clarkbjeblair: the logstash workers join the cluster themselves in order to index data15:02
jeblairclarkb: gotcha15:03
clarkbbigdesk also shows a few nodes are under some memory pressure related to cache so I am going to run `curl -XPOST 'http://localhost:9200/_cache/clear'` locally against my portforward to clear all cached data. This will temporarily slow stuff down but should be good for us longer term15:03
*** ojacques is now known as OlivierSanchez15:07
*** michchap has quit IRC15:08
clarkbelasticsearch3 seems to want to use all the memory for some reason... seems to float between 13 and 15 of 15.9GB of heap available. May need to kick it more later15:09
clarkbI am going to walk to the conference venue now15:10
*** michchap has joined #openstack-infra15:11
*** Ryan_Lane has joined #openstack-infra15:11
mordredhub_cap: ping15:11
clarkbran cache clearing once more as I am hoping 4 can pick up some of the slack as stuff is rebalanced15:11
mordredhub_cap: you use pexpect in trove in the guestagent code15:11
eskermordred: another question... are OpenStack github repos afforded any "special" status in Launchpad?  Or do I simply "import a branch hosted somewhere else"15:11
mordredbut that package isn't in global requirements or our mirror15:12
hub_capwhat u dont like ugly expect code in production mordred?15:12
eskermordred: referring to this specifically:
mordredthat leads me to believe it may not be being used?15:12
hub_caprearry? :(15:12
hub_capoh no it is15:12
mordredor at least not being tested15:12
hub_capbut its possibly we build it in to the image15:12
mordredok. so, you're suggesting that we need to add pexec to the global requirements list15:12
hub_capwe dont exactly pip install the reqs.txt in the guest15:12
hub_capwell im not suggesting anything yet :)15:13
* hub_cap is not a fan of our pexpect code15:13
mordredhub_cap: lemme rephrase - does it belong in your requirements.txt list?15:13
hub_capbut if its necessary to add to global- then we can15:13
hub_capmordred: likely yes15:13
hub_capfor now15:13
sdagueclarkb: how far behind is ES now time wise from when runs complete?15:14
*** SergeyLukjanov has quit IRC15:15
openstackgerritMonty Taylor proposed a change to openstack/requirements: Add pexpect which is used by trove
hub_capmordred: would u like me to submit a review sir?15:16
mordredhub_cap: already done15:17
mordredhub_cap: once that requirements change lands, this: should be able to pass the gate, at which point I'd love it if you could merge it15:17
clarkbsdague < 1 minute I think15:17
sdaguenice :)15:17
clarkbyour wsgi patch helped15:17
Alex_Gaynormordred: should I assume pexpect has an FFE?15:18
hub_capit will be done mordred15:18
*** changbl has joined #openstack-infra15:18
*** pblaho has quit IRC15:19
jeblairfor some reason, we're not watching elastic-recheck in here...15:20
jog0clarkb jeblair: did any ES settings change this weekend?15:20
jeblairjog0: did you read clarkb's report on the cluster health?15:20
*** kpavel has quit IRC15:20
jog0jeblair: I just saw someone says its green here15:20
jeblairjog0: more complicated.  read scrollback from clarkb15:21
jog0so no15:21
jog0I will just ask him in person in a few minutes15:21
jog0jeblair: ahh just read makes sense15:22
*** hashar has quit IRC15:22
sdaguejog0: so are we going to do the elastic-recheck summit session in QA or Infra?15:24
* sdague is starting to rough out some things this week, so want to make sure to save it a space if I need to15:24
*** thomasem has quit IRC15:24
jog0sdague: your choice15:25
jog0I haven't submitted one yet15:25
sdaguejeblair: you have an opinion?15:25
*** che-arne has joined #openstack-infra15:25
jog0feel free to submit one yourself15:25
*** matsuhashi has quit IRC15:25
sdaguejog0: there's going to be so much stuff under my name.... I'd rather have other folks do it :)15:25
*** thomasm has joined #openstack-infra15:25
jog0I can do it after my presentation this morning15:26
jog0just ping me around 1115:26
*** SergeyLukjanov has joined #openstack-infra15:26
clarkbjog0: you want to go in the theater not the pub15:27
sdaguejog0: yeh, not a big deal. We don't need the actual content for a couple weeks, just getting ahead of stuff to make sure less important things don't get landed15:27
clarkbjog0: but the pub can give yo udirections if you find it first15:27
*** mrodden1 has quit IRC15:27
*** thomasm has quit IRC15:28
jog0clarkb:  thanks on my way15:28
*** thomasm has joined #openstack-infra15:30
clarkbis the neutron test a lot more reliable after we got the q-vpn d-g revert in?15:32
mordredAlex_Gaynor: it's probably worth talking about, I don't think we should assume that, no15:33
sdagueclarkb: so one of the half a dozen neutron fails seems to have quietted15:33
sdagueit wasn't just one fail, elastic-recheck has been helpful at slicing them apart15:33
clarkbsdague: woot. I was just glancing at zuul status and it seems to have a lot of green neutron tests15:33
clarkbdouble woot to elastic-recheck being helpful15:33
Alex_Gaynormordred: I'm pretty confused by pexpects versioning, the 2.4 release, which is the highest version, is from 2008, but there are two releases from 2012 with lower release numbers.15:34
mordredAlex_Gaynor: it doesn't seem like a big risk to me- it's in ubuntu already, it's MIT license, it's only used by an incubated project15:34
mordredAlex_Gaynor: oh excellent15:34
clarkbmordred: is that for trove?15:34
* mordred did not follow the external link15:34
mordredclarkb: yes15:34
clarkbso I suggested to them that they just write the tests properly...15:34
mordredclarkb: pexpect is used in their guest agent15:35
Alex_Gaynor++clarkb :)15:35
mordrednot in the test suite15:35
clarkbI thought it was for tests that check output of commands15:35
sdagueyeh, honestly what it's giving us today is already pretty great, what it looks like by the end of icehouse... I think will be awesome15:35
mordrednope. it's for the guest agent which runs commands and does things based on what they do15:35
sdaguemordred: pexpect in a guest agent seems kind of terrible as well15:35
* mordred doesn't specifically care- other than he wants d2to1 out of their requirements file, and is pretty sure the patch to remove it will bounce on the requirements gate15:36
mordredhub_cap: this looks like a really terrible library15:38
clarkbmordred: I think you would enjoy this. We are in an old school theater that has been converted into a theater, restaurant pub thing. Sitting on couches with working wifi and electrical outlets at my feet :)15:38
mordredclarkb: nice!15:38
clarkbmordred: expect is terrible. I don't think any implementation of it can be non terrible as a result15:38
hub_capmordred: pexpect?15:39
hub_capyes yes im well aware of pexpect being a TERRRRRRRRIBLE way to impl things in the guest15:39
hub_capill file a bug to remove it, id love for it to dissapear15:40
*** tvb has joined #openstack-infra15:40
*** basha has joined #openstack-infra15:41
eskerAre OpenStack github repos afforded any "special" status in Launchpad?  Should one "import a branch hosted somewhere else?"  When having a look at Neutron on (for example) it indicates that doesn't know where code is hosted... yet lists tons of branches15:41
eskerI'm trying to determine how to wire launchpad up to:
*** dkranz has quit IRC15:43
*** thomasbiege1 has joined #openstack-infra15:43
mordredesker: hi!15:44
mordredesker: we don't do anyting with code on launchpad15:44
mordredesker: so don't set up code imports15:44
*** thomasbiege1 has quit IRC15:44
eskermordred: so leave it hanging then... okay, thanks15:46
*** MikeSpreitzer has joined #openstack-infra15:48
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add elastic-recheck to infra gerritbot
MikeSpreitzerHow can I tell whether made it into Grizzly?15:49
uvirtbotLaunchpad bug 1090595 in cinder "Deleting volumes set as PVs results in unresolvable error_deleting status" [Undecided,Fix released]15:49
MikeSpreitzer(I mean the fix, of course)15:49
jeblairMikeSpreitzer: bill rich left a comment indicating the fix was in this change:
clarkbMikeSpreitzer: there is no code commit linked in the bug so it may be a little hard to check. easiest thing would be to clone cinder, checkout stable/grizzly and look in the history15:50
jgriffithMikeSpreitzer: jeblair clarkb
*** jswarren has joined #openstack-infra15:53
jog0mtreinish: ping15:54
*** dkranz has joined #openstack-infra15:54
*** tjones has joined #openstack-infra15:55
MikeSpreitzerjeblair: the comment I see from Bill Rich says "Change I40d12bd1 is submitted for review"15:55
*** tjones has quit IRC15:55
jswarrenHello.  I'm seeing an odd failure in tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops:
MikeSpreitzerIs that the comment to which you are referring?  How did you map that to the URL you pasted a few minutes ago?15:55
jeblairMikeSpreitzer: copy/pasted the change id into the search field in gerrit15:56
jeblairMikeSpreitzer: (it also shows up in the git log, so one could search 'git log' for it as well)15:56
jswarrenIt repeats the same 7 lines several times with "sudo: 3 incorrect password attempts" at the end.15:56
MikeSpreitzerjeblair: thanks15:57
jeblairjswarren: it's probably trying to run a command not supported by the neutron rootwrap15:57
*** basha has quit IRC15:57
clarkbjeblair: fungi: elastic-recheck was running on jog0's personaly machine but he is at this conference with me and can't get on the box to restart it after the elasticsearch derpage. Any chance one of you guys want to poke at getting it running on logstash.o.o?15:57
jswarrenjeblair: OK, thanks.  What should I do?15:58
fungiclarkb: what's still missing, just the implementation change you had up on gerrit?15:58
clarkbfungi: yup I think that is it (well and an init script)15:59
clarkbfungi: because you added the gerrit user right?15:59
fungiclarkb: i did, yes15:59
mriedemthomasm: did your recheck fail too?15:59
jeblairjswarren: oh, that's clearly not because of that change.  check the recheck bug page for a similar bug and leave a 'recheck bug ###' comment; see for details15:59
clarkbawesome. then yes I believe just the puppet change to install it16:00
jeblairclarkb, fungi: i -1'd because i registered an irc bot user; i'll update the change to reflect that16:00
jswarrenjeblair: Yeah, I did that.  Could not find a bug that matched.16:00
mriedemthomasm: mordred: looks like a different failure now on stable/grizzly patches:
mriedemcan't find test results16:00
jswarrenThere's one that fails on the same test, but the logs look nothing alike.16:00
fungijeblair: yeah, i saw the comment and the entry for it in hiera. i can update it16:00
openstackgerritA change was merged to openstack-infra/config: Set mysql max_connections to 1024 on nodepool
jeblairfungi: it's okay i'll update it16:01
fungiclarkb: what's current invocation like? does it background and disassociate properly already, create a pidfile, et cetera?16:01
thomasmmriedem, Let me check, it just finished16:02
clarkbfungi: yes, I think so. It uses python-daemon like our other things iirc16:02
*** dizquierdo has quit IRC16:02
mriedemmtreinish: ping16:03
fungiideal. thanks clarkb16:03
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Deploy elastic-recheck on
*** acabrera is now known as alcabrera16:04
thomasmmriedem, It failed, but I'm having trouble finding where in the logs.16:05
thomasmspecifically check-tempest-devstack-vm-neutron16:05
*** mrodden has joined #openstack-infra16:05
thomasmAll of the other gates are passed16:05
mriedemthomasm: i opened a bug:
uvirtbotLaunchpad bug 1233256 in openstack-ci "stable/grizzly patches failing in jenkins - unable to find test results" [Undecided,New]16:05
thomasmmriedem, Great, thanks!16:05
mriedemthomasm: here is my nova patch that has failures, it's the same in all 3 that failed:
mriedemlooks like the code is trying to find nosetests results but the test is using testr16:06
thomasmmriedem, my failure for neutron looks the same.16:07
jeblairsdague, jog0: i'm happy to have elastic-recheck summit sessions under the infra umbrella; it clearly has some crossover, so i'd understand if some topics fit better in qa.16:07
jeblairttx, sdague:  your input on that would be appreciated16:07
*** guohliu has joined #openstack-infra16:07
*** hemnafk is now known as hemna16:08
*** thomasbiege1 has joined #openstack-infra16:08
*** thomasbiege1 has quit IRC16:08
mriedemthomasm: i think my bug is invalid, there are a lot of quantum test failures in my py27 results16:10
thomasmmriedem, I think it may be two separate issues, yeah.16:10
*** yassine has quit IRC16:10
*** ruhe has joined #openstack-infra16:11
thomasmmriedem, I'm trying to spot where the failure is occurring for the neutron gate16:11
*** markmc has quit IRC16:11
*** dguitarbite has joined #openstack-infra16:12
mordredjeblair: we're getting some interesting topic suggestions for the infra track :)16:12
thomasmBecause it failed before it couldn't find the results, unless it's just logging the fact that it couldn't, but tried before failing at 'Execute shell'.16:12
jeblairmordred: where do i look?16:12
openstackgerritA change was merged to openstack/requirements: add an explanation to the sqla requirement
*** flaper87|afk is now known as flaper8716:14
mordredjeblair: also,
*** rnirmal has quit IRC16:15
*** ruhe has quit IRC16:15
jeblairmordred: a couple look like "openstack plumbing"16:15
jeblairmordred: i can't see that16:15
jeblairmordred: (topic 9)16:16
mordredjeblair: oh - I believe I'm still the topiclead16:16
mordredttx: you may want to make jeblair that16:16
jeblairmordred: is that just a list of infra cfps?  (== search for 'infra' on index page?)16:17
mordredjeblair: yup16:17
mordredjeblair: and also the screen where you can review and accept them16:17
ttxmordred: yes, I received a few requests around that. Even if in theory they are not formally PTLs yet I should probably already switch16:17
jeblairttx: when do we/they formally become ptls?16:18
mordredttx: btw - similar to the tc thing - perhaps for next summit we should just use gerrit for summit session voting :)16:18
ttxjeblair: when elections end... that should be Thursday16:18
ttxjeblair: but that's a minor just want all PTLs to start at the same date, not some before others just because there was not enough candidates16:19
* mordred goes to approve a bunch of things real quick before jeblair gets the keys16:19
jeblairttx: ok.  no rush, i can wait till then.  :)16:19
jeblairmordred: i agree; i'm excited about all of them except the 2 that were misfiled!16:19
notmynameanteaya: ttx: I never saw a voting email. but maybe they only went to the elections with more than one candidate?16:19
ttxnotmyname: yes16:20
notmynamettx: ah, carry on then :-)16:20
mordredjeblair: yes. also, I feel like in new orleans I kept saying "we should have a summit session about that" - and then I got drunk and don't remember any of them16:20
jeblairfungi: aprv ?16:20
* mordred voted in all the elections16:20
jeblairmordred: it turns out you actually just said "i think we should have a summit session about $DRINKNAME" a lot.16:20
mordredjeblair: ah. no wonder they all sounded like great ideas16:21
jeblairmordred: i mean, yeah, "we should have a summit session about sazerac" sounds plausible and awesome.16:21
ttxI remember that one16:21
mordredjeblair: we SHOULD16:22
*** reed has joined #openstack-infra16:23
mriedemthomasm: markmcclain: bug report for the neutron client proxy issue in stable/grizzly checks:
uvirtbotLaunchpad bug 1233264 in neutron "stable branch patches failing in check queue due to missing 'find_resourceid_by_name_or_id'" [Undecided,New]16:25
jog0mordred: the open bar at openstack on ales has already opened16:26
*** julim has quit IRC16:27
*** jcoufal has quit IRC16:28
*** guohliu has quit IRC16:30
fungijeblair: i was still writing the initscript for a follow-on patchset to 4749716:33
fungijog0: i see the pidfile is hardcoded to /tmp/, is that where it's going to stay?16:34
jeblairfungi: ok, np.16:34
*** moted has joined #openstack-infra16:34
jeblairfungi: i can patch e-r to make the pid configurable and default to /var/run/elastic-recheck.pid16:34
jeblairjog0: ^16:34
*** matty_dubs is now known as matty_dubs|lunch16:35
jeblairfungi: i can patch e-r to make the pid configurable and default to /var/run/elastic-recheck/  <-- correction16:36
* anteaya crawls back up for air after submitting expense reports16:37
*** fbo is now known as fbo_away16:38
*** afazekas has quit IRC16:38
fungijeblair: that seems ideal to me. i doubt jog0 will mind since he's running by hand (and is about to stop doing so anyway)16:39
fungii'll submit this initscript on that assumption16:39
jeblairfungi: do you think it should be 'elastic-recheck' or 'recheckwatchbot'?16:40
*** mikal has quit IRC16:40
*** matty_dubs|lunch is now known as matty_dubs16:40
jog0fungi: mtreinish wrote that part AFAIK he took it from one of the many apps in the config repo16:41
fungijeblair: it looks like elastic-recheck everywhere16:41
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Deploy elastic-recheck on
fungias far as config file name passed on the cli, executable name, et cetera16:41
jog0fungi: I do not mind16:42
jog0fungi: and I am really bad at naming things too16:42
mriedemthomasm: markmcclain: fyi
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
* mordred is getting closer that working ^^16:48
*** SergeyLukjanov has quit IRC16:48
*** mikal has joined #openstack-infra16:49
*** tjones has joined #openstack-infra16:54
mtreinishfungi: yeah I wrote that part and naming wasn't really finalized then16:56
mtreinishit doesn't really matter to me what we call it16:56
fungimtreinish: no sweat. /var/run/elastic-recheck/ seems a fine default to me16:56
mordredjeblair: I don't thing image pre-caching is working in d-g anymore16:57
mordredjeblair: search for IMAGE_FNAME=cirros-0.3.1-x86_64-uec.tar.gz16:58
jeblairmordred: that sucks16:58
mordredjeblair: this is not a direct problem that I'm seeing, I just happened to notice an image being download during a devstack run on a d-g node16:58
mordredsdague: &&16:59
mordredsdague: ^^16:59
*** derekh has quit IRC17:00
sdaguemordred: hmmm... interesting17:01
mordred(I mean, we have other fish to fry this week, but still17:03
sdagueso there was some restructure on the hypervisor code in devstack, I wonder if a path changed that tripped up the caching17:04
*** sodabrew has joined #openstack-infra17:04
* anteaya puts lunch aside17:06
anteayawelcome sodabrew17:07
anteayawe figured out why all of our nodes have been unresponsive:
anteayathe url to send reports to the dashboard had been reverted, but we fixed that17:08
anteayaand now we only have 10 unresponsive nodes, but I just cleared the queue and restarted the worker about 15 minutes ago, so that number is dropping17:08
fungiyeah, something (possibly a stray puppet agent run i may have inadvertently fired on the master) unset it last week17:08
anteayaI do have a question about pending tasks17:09
anteayaso far we have 34 and I am not seeing the number drop, perhaps it is too earlier to tell though since the worker restart17:09
anteayajust glad you are here sodabrew, thanks for joining us17:10
*** nati_ueno has joined #openstack-infra17:10
pleia2not supposed to declare backscroll bankruptcy for a *weekend* away, busy weekend17:10
anteayamorning pleia217:11
anteayamordred was chatty17:11
anteayawe need to get him on an airplane17:11
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: doc: fix up strict errors
*** reed has quit IRC17:12
sodabrewHi @anteaya I'm not actually around at the moment. Will be later on today though, around 1 or 2pm Pacific time.17:12
*** ekarlso has quit IRC17:13
anteayasodabrew: I kind of ambushed you, sorry17:13
anteayajust really hoping to make some progress on this17:13
anteayawhen you are available17:13
sodabrewthanks! catch you later on17:14
zaromordred: i'm not sure how these disagreements get resolved ->
*** sodabrew has quit IRC17:16
*** reed has joined #openstack-infra17:16
*** julim has joined #openstack-infra17:16
*** gyee has joined #openstack-infra17:17
zaromordred: do you want to provide an opinion there?17:17
*** dkliban has quit IRC17:18
mordredzaro: yes - but I will have to read more things first17:18
*** tkammer has quit IRC17:19
mordredpleia2: come on - you'll enjoy reading an hour-long conversations between lifeless and I on semantic versioning17:19
mordredpleia2: it's like Game of Thrones, except with less talk about winter17:20
*** dkliban has joined #openstack-infra17:20
*** dcramer_ has quit IRC17:21
thomasmmriedem, Ahh, thanks!17:21
*** morganfainberg has quit IRC17:22
*** ekarlso has joined #openstack-infra17:22
*** morganfainberg has joined #openstack-infra17:22
pleia2mordred: slightlyless blood too17:22
*** rockyg has joined #openstack-infra17:24
*** sarob has joined #openstack-infra17:24
*** thomasbiege1 has joined #openstack-infra17:25
*** SergeyLukjanov has joined #openstack-infra17:26
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
*** thomasbiege1 has quit IRC17:28
jeblairjog0, fungi, mtreinish:
mriedemmarkmcclain: how do you get patches through jenkins if the change is on the quantumclient branch?17:29
*** MikeSpreitzer has left #openstack-infra17:29
mriedemah, nm, force merge from mordred17:30
*** johnthetubaguy has quit IRC17:30
zaroclarkb: pleasant drive to bend?17:30
*** thomasbiege1 has joined #openstack-infra17:31
clarkbzaro: No, it was crazy. But I made it in one piece. So many cars up on the berms by the freeway or in ditches upside down and facing the wrong direction17:31
clarkbI think we counted 6 between seattle and portland17:31
*** jerryz has joined #openstack-infra17:32
*** morganfainberg has quit IRC17:33
anteayaclarkb: wow, glad you are safe17:33
*** mdrnstm has joined #openstack-infra17:33
anteayawas their ice, or just rain?17:33
clarkbanteaya: a lot of rain and wind17:33
zaroanteaya, pleia2 : you were in seattle on the good week.  this weekend has been horrible17:33
clarkbthe weather in bend is nice though. crisp and dry17:33
anteayadry is nice17:34
*** mdrnstm is now known as morganfainberg17:34
anteayacrisp is okay if you are dry and have a hat17:34
*** thomasbiege1 has quit IRC17:35
mtreinishjeblair: it looks good to me, I'm running the tests to ensure everything works (I doubt it broke anything though)17:36
jeblairmtreinish: can we have jenkins run them?17:37
mtreinishjeblair: right now they require a working config file. (because they touch gerrit, es, and other things)17:38
mtreinishso we could add them to jenkins but I think it would be better to make them run independently first17:38
jeblairmtreinish: yep, definitely.  zuul has a fake gerrit implementation if you feel like going that route...17:38
mtreinishjeblair: really cool that would be useful for testing the gerrit stuff17:39
jeblairmtreinish: clarkb and fungi are working on getting a _real_ gerrit spun up in git-review tests, but that's not quite working yet.17:39
jeblairmtreinish: (i'd go with the fake one for now)17:39
*** ekarlso has quit IRC17:39
*** ekarlso has joined #openstack-infra17:39
*** tjones has left #openstack-infra17:39
mtreinishjeblair: yeah the gerrit tests for elastic-recheck just leave a comment on review-dev
sdaguejeblair: so... question on devstackgate vs. devstack-tempest vs. sudo17:40
jeblairsdague: ya (did you see afazekas patch?)17:40
sdagueI guess not yet17:40
fungijeblair: mtreinish: well, it does work but it's a little racy/flaky or slow (depending on how we do it) tearing down and spinning up a new gerrit for each test17:40
mtreinishfungi: I'm fine with faking it, the test is just making sure we call gerritlib correctly for leaving a comment17:41
jeblairfungi: yeah, i think e-r's use of gerrit is very similar (actually a subset) of what zuul does, so it's probably a good match.17:41
mtreinishit was easier to just point it to a real gerrit and look for the comment :)17:41
fungimtreinish: yeah, that's probably fairly safe to just stub out with a fake then17:41
sdaguejeblair: ok, so that's one approach17:42
sdagueI was also thinking we could just drop in a new sudoers file for the debug commands we want17:42
mordredjeblair, sdague: that doesn't seem like the _best_ approach on first glance17:42
sdagueyeh, I'd rather not run as root17:42
sdagueas it masks too many other things17:42
jeblairsdague: i haven't formed an opinion yet17:42
mordredI _do_ want additional interesting information17:42
jeblairsdague: btw, what's the problem? :)17:42
sdaguethe issue is we need to run ip and iptables as root17:43
sdagueto dump out their contents17:43
jeblairsdague: does tempest assume it's running on-host?17:43
mordredyeah - it would certainly be good if tempest did not assume that :)17:43
sdaguehowever, it's behind a flag you need to set if you wan it17:43
jeblairsdague: i thought 'point tempest at an external (multi|single-node install) was a primary use case17:43
sdaguejeblair: it is, this is sitting behind some debug flags17:44
jeblairsdague: where are the debug flags set?17:44
*** hogepodge has joined #openstack-infra17:44
sdaguejeblair: one sec...17:45
sdagueso, it's in the wrong place, but it's currently in tempest.conf.sample17:47
jeblairsdague: also is this to get ip/iptables content before and after individual tests? (as opposed to, say, a sort of final state dump that could just be done by d-g at the end of the run?17:47
sdaguelets say, this really shouldn't have gone in like this17:47
sdaguejeblair: yeh, this is point in time dump17:47
sdaguewhat we really need to find out is when neutron didn't bring up a network in time, what was going on17:48
sdaguehow much was pulled together17:48
*** flaper87 is now known as flaper87|afk17:48
sdaguelong term, what we probably want is a neutron API to put that to the logs17:50
jeblairsdague: do you know why tempest is being run as 'stack' atm?17:50
sdaguejeblair: because it's the non root user17:51
sdaguehonestly, besides that, I don't know that stack is important relative to any normal user17:51
*** sdake__ is now known as sdake_17:52
jeblairsdague: so, dropping in a sudoers.d file sounds like the best approach atm; adding to a different account would be even better, only dropping it in when needed is even better than that, and not needing it at all is even better than that...17:52
sdagueyeh, we can't not need it at all unless we move the functionality back to neutron17:52
jeblairright; so the best compromise i can think of atm is to create a tempest user and make a sudoers.d file for it that allows that17:53
sdagueso, how about I propose a d-g change to switch users17:53
sdagueok, I'll work on that17:53
jeblairthat minimizes the potential impact to testing17:53
jeblaircool, thx17:53
hogepodgeI'm about to send a new review to add the puppet-vswitch module to stackforge. I don't expect I did everything right the first time through, so comments and suggestions are certainly welcome.17:54
*** melwitt has joined #openstack-infra17:55
*** rnirmal has joined #openstack-infra17:55
openstackgerritChris Hoge proposed a change to openstack-infra/config: Added puppet-vswitch project to Stackforge
*** lcestari_ has quit IRC17:58
SergeyLukjanovhi all, are there any thoughts on sorry for spamming - it's on review 2.5 months ;)17:59
*** dcramer_ has joined #openstack-infra18:00
*** lcestari_ has joined #openstack-infra18:01
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user
sdaguejeblair: ok, see what you think18:01
*** mrmartin has joined #openstack-infra18:02
SergeyLukjanovdoes anyone know will be design summit talks displayed at
*** dcramer_ has quit IRC18:09
openstackgerritA change was merged to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour
fungiSergeyLukjanov: ^ thanks again for your help (and patience!) on that one18:09
SergeyLukjanovfungi, thank you for dozens of reviews :)18:09
SergeyLukjanovfungi, the next step, I think is to update projects.yaml in os-infra/config18:09
fungiSergeyLukjanov: yes, and then we can rip the mapping out of jeepyb once the projects.yaml update merges18:10
SergeyLukjanovfungi, yeah, I'll try to do it tomorrow18:11
fungiSergeyLukjanov: thanks again!18:11
SergeyLukjanovbtw, could be merged too I think18:12
sdagueSergeyLukjanov: design summit schedules are typically not set until the week before the event18:13
sdagueat the PTLs are busy getting the release out before that point18:13
*** basha has joined #openstack-infra18:13
SergeyLukjanovsdague, just looking at how to plan own schedule for summit18:14
openstackgerritFelipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Added support for Git shallow clone parameter
*** dguitarbite has quit IRC18:15
*** dguitarbite has joined #openstack-infra18:15
*** freyes has joined #openstack-infra18:16
openstackgerritA change was merged to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad
*** fbo_away is now known as fbo18:20
*** MarkAtwood has joined #openstack-infra18:21
*** dcramer_ has joined #openstack-infra18:21
*** guitarzan has quit IRC18:23
*** guitarzan has joined #openstack-infra18:23
*** _david_ has joined #openstack-infra18:25
*** cody-somerville has joined #openstack-infra18:29
*** afazekas has joined #openstack-infra18:30
*** _david_ has quit IRC18:34
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Add elastic-recheck to openstack-infra channel
jog0jeblair: ^18:34
jeblairopenstackgerrit: heh, i think i proposed that already18:34
jeblairjog0: ^18:34
jog0jeblair: done18:36
*** basha has quit IRC18:38
lifelessmordred: pleia2: semver is coming18:39
*** sarob has quit IRC18:40
mriedemmordred: ping?18:40
*** sarob has joined #openstack-infra18:40
openstackgerritA change was merged to openstack-infra/config: Add elastic-recheck to infra gerritbot
fungijog0: ^18:41
fungijeblair: jog0: since 49018 merged (pidfile relocation for e-r), are we okay to merge (initscript look fine?)18:41
dkranzfungi: Any idea why is not merging? Has all reviews and no dependencies and +1 from jenkins.18:42
clarkbdkranz: recheck vs reverify18:42
dkranzclarkb: Ah! Thanks.18:43
clarkbreverify is for the gate, recheck for check tests18:43
dkranzclarkb: Yeah, I didn't do the recheck and didn't notice.18:43
*** sarob has quit IRC18:45
*** openstackgerrit has quit IRC18:47
*** openstackgerrit has joined #openstack-infra18:47
jog0fungi: one issue18:47
dkranzclarkb: Is there a way to deal with a patch that has a dependency that was abandoned, other than resubmitting fresh?18:47
openstackgerritDiane Fleming proposed a change to openstack-infra/config: Updated api.yaml for the netconn-api v1.0 and v2.0 refs - pom location changed
clarkbdkranz: you can unabandon the dependency and merge it18:48
clarkbdkranz: submitting without the dep is probably simplest18:48
fungijog0: ahh, thanks!18:48
dkranzclarkb: Thanks.18:48
jog0fungi: but I want that patch merged ASAP18:52
fungijog0: so the initscript is going to need to run something like 'python /opt/elastic-recheck/elastic_recheck/ /etc/elastic-recheck/elastic-recheck.conf' instead?18:52
*** sarob has joined #openstack-infra18:52
fungiokay, coming right up18:53
jog0fungi: \o/18:53
*** gyee has quit IRC18:54
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Deploy elastic-recheck on
fungijog0: hopefully ^ that does what we expect (though s-s-d may get confused, not sure)18:57
fungiin the interim i can launch it by hand now that i know what you were running18:57
*** vipul has quit IRC18:57
*** mrodden1 has joined #openstack-infra18:57
*** vipul has joined #openstack-infra18:58
*** mrodden has quit IRC18:58
jog0fungi: I don't see any bugs so LGTM18:59
jog0fungi: thats a good idea18:59
fungijog0: is installing executables just a matter of adding a proper [entry_points] section to setup.cfg in the elastic-recheck project?19:00
jog0fungi: I *think* so19:01
jog0also renamining things away from 'bot' I think19:01
jog0patches welcome19:01
mordredentrypoints ftw19:01
mordredshould all just work19:01
jog0mordred: unlike
openstackgerritJeremy Stanley proposed a change to openstack-infra/elastic-recheck: Install an executable for the elastic-recheck bot
fungijog0: mordred: that ^ ?19:04
*** dkehn_ is now known as dkehn19:04
*** pmoosh has joined #openstack-infra19:05
*** sarob has quit IRC19:08
*** sarob has joined #openstack-infra19:08
*** sarob has quit IRC19:12
*** AlexF has quit IRC19:12
*** adalbas has quit IRC19:15
*** pmoosh has quit IRC19:16
*** pmoosh has joined #openstack-infra19:17
*** qba73_ has joined #openstack-infra19:17
*** beagles has quit IRC19:20
*** alexpilotti has joined #openstack-infra19:22
openstackgerritA change was merged to openstack-infra/config: Deploy elastic-recheck on
openstackgerritA change was merged to openstack-infra/config: Update projects when requirements change
*** reed has quit IRC19:27
*** boris-42 has joined #openstack-infra19:27
*** ryanpetrello has joined #openstack-infra19:28
*** ArxCruz has quit IRC19:29
*** alcabrera has left #openstack-infra19:29
*** MarkAtwood has quit IRC19:31
mordredooh. exciting!19:31
zarofungi: i'm working on bug 108310119:37
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add basic syntax highlighting to cgit
uvirtbotLaunchpad bug 1083101 in openstack-ci "Set up private gerrit for security reviews" [High,In progress]
zarofungi: need your help to figure out what groups to create and which permissions to assign them ->
*** adalbas has joined #openstack-infra19:38
*** nicedice has joined #openstack-infra19:39
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add basic syntax highlighting to cgit
*** sandywalsh has quit IRC19:40
*** thingee is now known as thingee_zzz19:40
openstackgerritKhai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews
fungizaro: i think the workflow we'd discussed previously was that we'd want to sync existing group memberships continuously from the public gerrit, but that projects would be acl'd so that changes to them were only visible to their respective core group members instead of everyone. probably need to hash the workflow details out with ttx again now that it's closer to reality19:41
*** hashar has joined #openstack-infra19:43
*** qba73_ has quit IRC19:43
fungizaro: unfortunately group sync probably also means we have to sync user accounts as well (since the group membership tables are joined on user id number)19:45
*** beagles has joined #openstack-infra19:46
zarofungi: that patch sets up replication to the new gerrit so that should include the groups.  not sure about users though.19:46
zarofungi: do you know if user info is save to the underlying git?19:47
fungizaro: what do you mean by user info?19:47
fungiyou mean individual account preferences?19:47
fungithose are in tables in mysql, at least in 2.4.x19:47
zarofungi: yes, user and their account profile.19:48
fungithey're in the db19:48
fungijust like the groups are19:48
zarofungi: aren't groups in refs/meta/config?19:48
zarofungi: ohh, those are only the permission settings.19:49
openstackgerritA change was merged to openstack-infra/elastic-recheck: Install an executable for the elastic-recheck bot
zarofungi: i think maybe replication only replicates the git stuff?19:50
mordredyes. it does19:52
zarofungi: yep, gerrit replication only replicates the git stuff.19:52
mordredyou'd need to set up mysql replication to replicate mysql19:52
openstackgerritEndre Karlson proposed a change to openstack-infra/config: Add RTFD hook to trigger auto-build of docs
mordredand that might get weird is19:52
jeblairfungi: merged (elastic-recheck executable)19:52
zaromordred: why replicate?  can't we just point to same mysql?19:52
*** sandywalsh has joined #openstack-infra19:53
mordredzaro: because then a setting made in the security gerrit might modify the global gerrit19:53
*** AlexF has joined #openstack-infra19:53
mordredzaro: but - if we make a mysql user for the security gerrit that does not have write privs, it might be ok19:54
zaromordred: ohh of course.19:54
zaromordred: probably not, if we want comments, votes, etc.. right?19:55
*** beagles has quit IRC19:55
*** beagles has joined #openstack-infra19:55
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Correct elastic-recheck puppet dependencies
fungijeblair: ^ yes, i know19:56
*** pblaho has joined #openstack-infra19:56
*** alexpilotti has quit IRC19:56
fungijeblair: i'm trying to hand-patch-and-apply it to logstash.o.o through a puppet development environment currently to ferret out anything else which was also missing19:57
fungimordred: zaro: yes, i think the original workflows discussed would probably need to be able to overwrite some tables in the security gerrit with their current content from the public gerrit19:59
fungimordred: zaro: and possibly also perform some etl on them in flight19:59
fungi(because not sure what breakage we're likely to uncover which could require filtering/overriding some rows, fields, whatever)20:00
*** tvb has quit IRC20:00
*** mrmartin has quit IRC20:02
zarofungi: mordred : does our puppet setup the gerrit mysql?20:02
fungijeblair: oh, you said 49033 not 47497. i'll adjust 49042 to suit20:03
*** sodabrew has joined #openstack-infra20:04
*** changbl has quit IRC20:07
zarofungi: ok. i see where mysql for gerrit gets setup.  i'll take look further to see how it can get replicated.20:08
*** thedodd has joined #openstack-infra20:08
*** davidhadas has joined #openstack-infra20:09
mordredjeblair: what is our policy on channel op rights for devs in team channels?20:10
*** thedodd has quit IRC20:11
*** thedodd has joined #openstack-infra20:11
*** dguitarbite has quit IRC20:12
jeblairmordred: we need a bot/script to manage channel perms; i don't want to be in the business of handing out perms manually20:12
*** AlexF has quit IRC20:13
jeblairmordred: i expect with that in place, we'd have a policy something like core devs or ptls could get ops on their related channel20:13
*** sarob has joined #openstack-infra20:16
*** jcoufal has joined #openstack-infra20:21
lifelessthat policy would be fine for -deployment, FWIW.20:21
*** adalbas has quit IRC20:23
anteayasodabrew: are you available to chat?20:23
jcoufalHi everybody, can I ask for a help regarding UX discussion tool?20:24
anteayajcoufal: ask away, we will do our best20:24
*** sarob has quit IRC20:24
ekarlsojcoufal: UX discusion tool how?20:24
jcoufalright, so20:24
jcoufalcurrently UX community is discussing their issues on Google+20:25
anteayayes, not optimal20:25
jcoufaland we needed to move somewhere else, because it is very unefficient20:25
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Correct elastic-recheck Puppet dependencies
jcoufalyeah, there was voting between Askbot and Discourse20:25
jcoufaland Askbot won the voting20:25
jcoufalbased on previous talks infra team would love to run our own deployment of askbot20:25
anteayawe have had discussion previously about a UX tool20:26
jcoufalbut previously there were some troubles with postgresql20:26
jcoufalso askbot is still heavily dependent on postgresql20:26
jcoufalcurrently one of our developers is trying to setup some puppet scripts to propose new server to infra20:27
jcoufalbut it might take some time20:27
anteayajeblair: what is our latest direction with a UX tool?20:27
jcoufaland it's getting very long20:27
*** dprince has quit IRC20:27
jcoufalso my question is if you could help us getting the deployment run20:27
anteayajim would have the latest info from the infra end of things methinks20:27
fungijcoufal: yes, for the moment askbot runs for us because of a variety of reasons, one of which was that we don't currently have a lot of developers familiar with managing postgresql (but plenty with heavy mysql background)20:27
jeblairjcoufal: hi!  yes, clarkb worked for quite a while on askbot puppet modules before we decided it was getting very complicated and we weren't making progress on our other tasks...20:28
jcoufaljeblair: yeah, but it was few months ago, wasn't it?20:29
jeblairjcoufal: if someone did finish some puppet modules we would be happy to run them (for ux discussion and even for ask.o.o)20:29
jeblairjcoufal: yes20:29
jcoufalwe are trying to get them working20:29
jcoufalit just takes time20:29
jeblairjcoufal: also, i would eventually like to propose storyboard as an alternative, but it's not ready (it's not even ready for bugs yet, so it's definitely not ready for ux discussions)20:30
jeblairjcoufal: but we can talk about that later; will you be at the summit?20:30
anteayawould this be worthy of a design session at the summit?20:30
jcoufalso the question is, if you would tolerate temporary deployment of UX askbot on openshift (until our deployment is ready) and then we can just migrate DB to our deployment20:30
*** ArxCruz has joined #openstack-infra20:30
jcoufaljeblair: anteaya: I'll be there, we just want to move forward sooner20:30
jcoufalthat's why I am talking about temporary openshift solution20:31
*** tvb has joined #openstack-infra20:31
jeblairjcoufal: can you talk a bit about how openshift would help?20:32
jcoufaljeblair: well there is no problem in getting the setup running in normal way or openshift - it's fast. The guy who is helping doesn't have much experience with puppet, so it gets longer for him to 'puppetize' the whole thing20:33
jcoufaldeployment on openshift would be faster20:33
openstackgerritA change was merged to openstack-infra/config: Correct elastic-recheck Puppet dependencies
fungijog0: clarkb: i'll reenable puppet on logstash.o.o in a bit once ^ that propagates, but at this point e-r is started and running there for about the past 15 minutes20:33
clarkbfungi: \o/20:34
clarkbfungi: thank you for helping out with that. Too many distractions on my end to give it an honest effort20:34
jcoufaland UX can discuss there until we have the real deployment ready based on puppet scripts20:34
clarkbfungi: I owe you a beer20:34
jcoufalwhich might take weeks20:34
mordredjeblair: I think that's an excellent policy. I agree though about the bot thing20:35
jog0fungi:  e-r is runnig?20:35
jog0I don't see anything in #openstack-qa20:35
fungiclarkb: you can subtract it from one of the many beers i owe you20:36
fungijog0: fungi@logstash:~$ ps auxww|grep recheck20:36
fungirecheck  18219  0.0  0.8 160240 17936 ?        S    20:18   0:00 /usr/bin/python /usr/local/bin/elastic-recheck /etc/elastic-recheck/elastic-recheck.conf20:36
funginot sure why it's not connecting. i'll see if i can tell20:36
jeblairpretty sure it's there20:36
*** thingee_zzz is now known as thingee20:37
mordredjeblair, jcoufal: honestly, at this point, I think I'm fine with just about any solution to the UX discussion problem, because it seems like all of the 'good' and 'appropriate' solutions are a few months out from being usable20:37
hub_caphey what projects use pecan/wsme ? /me looks at their source for a Trove move20:37
jeblairjcoufal: we don't currently manage anything in openshift; how would you see that working?20:37
*** sarob has joined #openstack-infra20:37
anteayahub_cap: I think ironic and ceilometer20:37
anteayadon't know if there are others20:38
*** davidhadas has quit IRC20:38
mordredif a UX guy spinning an ask up in openshift buys us enough time to talk about the overall problem at the summit and then get a thign in that things can migrate to and we're happy with20:38
hub_capthx anteaya20:38
mordredthen, it's not idea, but it's not as bad as google+ perhaps20:38
mordredas long as we're clear that it's a stop gap measure and not meant to be the long-term solution to the problem (I don't want someone getting attached to their openshift thing on the side) - of course, as always, my opinions are often wrong20:39
jeblairmordred, jcoufal: so the proposal is for a privately managed askbot on openshift until puppet-managed askbot is ready?20:39
*** davidhadas has joined #openstack-infra20:39
*** tvb has quit IRC20:39
mordredjeblair: I think that's what I'm hearing.20:39
fungijog0: currently we're creating a /var/log/elastic-recheck directory, but the bot doesn't seem to have an open fd in it20:40
jcoufaljeblair, mordred: Yes, mostly yes. We would create an account, deploy askbot there, set everything up and take care about it until we have real deployment.20:40
mordredjeblair: I'd amend that to be "until puppet-managed askbot or storyboard, whichever comes first"20:40
jeblairmordred: ++20:40
mordredjcoufal: ++20:40
jeblairjcoufal, mordred: that seems reasonable to me.  and i'm glad you're working on this btw -- regardless, it sounds like we'll end up with askbot run in the public infra which is something we really want20:41
jcoufaljeblair, mordred: the only thing what I would like to ask is, if could be pointed to that deployment?20:41
mordredjcoufal: and then, at the summit, we also would like to talk to you about our plans with storyboard so that we can make sure the long-term vision has you ultimately happy20:41
jeblairjcoufal: and then if storyboard can do a better job later on, even better.20:41
clarkbfungi: it may be opening the fd then closing it when daemonization happens20:43
clarkbfungi: geard has/had this problem20:43
mtreinishmordred: quick question, what's the recommended way for stubbing things out in unit tests? I remember a discussion about this from a while ago, but I can't remember the answer.20:44
jeblairfungi: did anyone configure the bot to log?20:44
jeblairfungi: it would be a separate config file, pointed to by the main config file20:44
jcoufaljeblair, mordred: will be happy to discuss that20:44
anteayamtreinish: I think mock is the preferred framework since it works with py320:44
mtreinishanteaya: ok thanks20:45
*** fbo is now known as fbo_away20:45
jcoufaljeblair, mordred: what do you think about pointing the domain ( to the askbot deployment, would it be possible?20:45
fungiclarkb: after a while of select looping, strace says
jeblairjcoufal: we don't generally point dns entries at hosts not managed by the project; i don't feel very comfortable doing that.20:45
fungijeblair: ahh, checking20:45
jeblairfungi: there is no "log_config" in the config file20:46
jeblairjcoufal: can you live without that until the puppet-managed askbot is ready?20:47
jog0fungi: I actually have never run the full bot in daemon mode20:47
jog0but the logs are pretty verbose20:47
jeblairjog0: i noticed.  i fixed a number of things.20:47
jeblairit does work now20:47
*** sarob has quit IRC20:47
jog0jeblair:  thanks20:48
*** sarob has joined #openstack-infra20:48
jcoufaljeblair: I see the reason. My concern is about moving people temporarily somewhere, make them used to that and then move them to another place20:48
jcoufaljeblair: that's I wanted to make the transition smooth (even the deployment would be temporal, because user won't see any change when changing to our deployment)20:49
*** sarob has quit IRC20:52
jeblairfungi, jog0: i'll propose a log config in a minute20:52
fungijeblair: yeah, i have it restarted with logging enabled now20:52
fungipatch coming20:52
jeblairfungi: oh, ok.20:53
fungibut it does seem to be connecting20:53
mtreinishfungi: where do the logs go?20:53
fungiand identifying successfully with the right nick20:53
jeblairfungi: yes, i was talking to it in -qa earlier.20:53
fungimtreinish: they will go into /var/log/elastic-recheck/ once the patch goes in20:53
fungijeblair: thought so. i saw some tests from you in the debug log when i was reviewing it20:54
*** mordred has quit IRC20:57
*** mordred has joined #openstack-infra20:57
*** dizquierdo has joined #openstack-infra20:58
mtreinishjeblair: I'm going to submit a summit topic for elastic-recheck, should I put it in qa or infra?20:58
jog0sdague: ^20:59
mtreinishapparently, there was a discussion about it earlier that I missed20:59
clarkbmtreinish: I think we decided earlier today that either would work20:59
jeblairmtreinish: +1 infra20:59
jog0infra works for me too20:59
mordredjcoufal: so, I hear that - the problem is that if we give you the dns pointer, and then something breaks, people will come ask us to fix it (this happens a lot) and we have to say "sorry, we don't actually run that"20:59
jeblairmtreinish: unless you think due to subject matter it should be in qa (it's a crossover project, that could easily be the case)20:59
mtreinishjeblair: ok that works for me I'll put it in infra20:59
mtreinishjeblair: no, I think most of the discussion about it are more infra related21:00
jeblairmtreinish: cool21:00
jog0jeblair: as long as sdague shows up ...21:00
mordredjcoufal: I betcha swapping a url won't be too bad for folks once we have it up for real, and if it is, I'll buy everyone a beer21:00
jcoufalmordred: I was just wondering that it might be possible to point dns entries outside the project, because does the same thing21:00
jcoufalthe difference would be, that it would be temporary21:00
jcoufalbut I understand your concerns21:01
*** emagana has joined #openstack-infra21:01
jeblairwe're trying to avoid repeating that :(21:01
jcoufaljeblair: I know :(21:02
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Turn on logging for elastic-recheck
matty_dubsI was going to be helping jcoufal set up AskBot. So it looks like the best path is to just set up a server via puppet?21:02
*** dkliban has quit IRC21:02
mtreinishfungi: ^^^ whitespace!21:02
jcoufalmordred, jeblair: can matty_dubs reach you and ask few questions around puppetizing?21:02
fungimtreinish: grrr21:02
mordredmatty_dubs: yup! we're here all the time!21:03
fungimtreinish: clearly i forgot my 'git diff --check master' mantra21:03
jeblairmatty_dubs: have you seen these docs?
jeblairmatty_dubs: also, you may find helpful21:04
*** rfolco has quit IRC21:04
matty_dubsjeblair: Ooh, I had seen the first, but not the latter21:05
matty_dubsDo you know off-hand why it was abandoned? Was it just a lack of time/resources, or as there a blocker?21:06
*** tvb has joined #openstack-infra21:06
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Turn on logging for elastic-recheck
fungimtreinish: ^ this house is clean21:07
jeblairmatty_dubs: we tried to use mysql, which sent us very far down a dead-end, basically, and clark had to move onto other things (it was expedient to just outsource to the askbot folks at that point)21:07
anteayasodabrew: available?21:08
sodabrewhey, yep available now21:08
mtreinishfungi: what about that tab change wasn't there before :)21:08
clarkbmatty_dubs: jeblair: right, the mysql support wasn't there. Also neither was the i8ln support21:08
sodabrewso, you're at the issue of the dashboard job worker appears not to be processing?21:08
matty_dubsjeblair: And the outsourced solution doesn't support multi-tenancy, right?21:08
fungimtreinish: it's documented in the commit message though ;)21:08
clarkbso two of the things we wanted to get working were non starters21:08
anteaya593 pending tasks right now21:08
anteayaI have 6 workers working21:08
matty_dubsIOW, is the right path for me to try this vs. us looking at hosting this with them, too?21:08
sodabrewand you're starting the job workers by rake task?21:09
mtreinishfungi: fair enough, you prempted my nit-picking21:09
jeblairmatty_dubs: it's not working out that well; we'd like to insource it.21:09
anteayasudo -u puppet-dashboard bundle exec rake jobs:work21:09
sodabrewaha! I have it. add RAILS_ENV=production21:09
anteayathey all have their own screen window21:09
jeblairmatty_dubs: so yes, if you get this working, we'll run it here for ux and probably try to move ask.o.o to it as well.21:09
*** amotoki_ has joined #openstack-infra21:09
anteayasodabrew: I specified that in root.rb21:09
*** thingee is now known as thingee_zzz21:09
anteayathe entire deployment is in proctuction21:10
matty_dubsjeblair: Ah, okay. FWIW, when I tried to set AskBot up locally, the MySQL stuff just flat out didn't work. I could install it fine, but AskBot refused to actually use it.21:10
anteayayeah, I didn't want to have to add that to every command21:10
matty_dubsIs MySQL a requirement, or could we use Postgres?21:10
sodabreware the job workers claiming to do anything?21:10
anteayaoh yes21:10
anteayabusy buys21:10
clarkbmatty_dubs: :( that is disappointing because that was the state in january21:11
jeblairmordred, clarkb: want to weigh in on matty_dubs question?21:11
jeblairmy feeling is that if it needs postgres, then we should run postgres; but we also want to use trove, which is mysql-only, right?21:11
clarkbI personally don't have strong opinions about mysql vs postgres because I am not a mysql expert and have to think hard for either21:11
mordredwe want to move all of our databases to be run by cloud databases / trove21:11
mordredwhich are mysql only21:11
clarkbbut we are surrounded by mysql expertise and cloud dbs are mysql21:11
mordredalso, none of us know anything about admin-ing postgres21:11
mordredat all21:12
clarkbmordred: well I know enough to run it on our slaves21:12
fungithe postgres need had to do with its search indexing mostly, right? or was that something else?21:12
clarkbmordred: which isn't much21:12
clarkbfungi: index for serach was a big part of it21:12
clarkbsounds like it may not work at all now on mysql though21:12
mordredfungi: yes. it was for search, and because ask is using $library and $library has very snarky bearded and lame comments about mysql not being a real db21:12
fungiyay bitrot21:12
jeblairit used to involve a lot of vacuuming.21:12
matty_dubsYeah, they say that. My problem was just that whatever they used to abstract the database calls just flat-out didn't work with MySQL.21:12
anteayasodabrew: do you want me to give you access to the server, you can poke around?21:12
*** mrodden has joined #openstack-infra21:12
matty_dubsIt was supposed to, but didn't.21:12
sodabrewwell that's weird. you're sure that you have only dashboard 2.0 running both the web interface and the job workers?21:13
* mordred wants to cry21:13
anteayasodabrew: can you pm me your public key?21:13
clarkbmordred: its ok21:13
sodabrewsure, @anteaya will do21:13
anteayasodabrew: I used your fork21:13
anteayathat is all the puppet I have on that server21:13
*** mrodden1 has quit IRC21:14
matty_dubsSo I can poke around some more and see if it has an easy fix.21:14
dstufftmordred: I make snarky comments about mysql not being a real db ;P21:14
*** amotoki has quit IRC21:14
fungijog0: so is it doing the things it needs to then? since it seems to be making it into irc... what else is missing?21:14
matty_dubsHehe, I tend to favor Postgres anyway, but I don't have any real DBA experience either way21:15
mordreddstufft: I know - but you at least do it with guts in a channel with former MySQL Inc folks21:15
*** thomasm has quit IRC21:15
mordredthat I can at least respect21:15
clarkbfungi: it should be yelling at us on every failure21:15
dstufftmordred: :D21:15
*** amotoki_ is now known as amotoki21:15
clarkbfungi: either with a bug that it detected or info that it couldn't correlate a bu21:15
*** MarkAtwood has joined #openstack-infra21:15
jeblairclarkb, mordred: so can we say to matty_dubs "mysql would be preferred due to local expertise and our ability to better integrate it into our future directions in deployment; but failing that, we are okay with the puppet module running postgres on the host (with the caveat that none of us know anything about that)?"21:15
fungiclarkb: and it detects these by scraping something?21:15
dstufftmordred: I never was afraid to be snarky to peoples faces, (well "faces"), not sure if that's a good or a bad thing aobut me :)21:15
clarkbfungi: elasticserach21:16
openstackgerritMonty Taylor proposed a change to openstack/requirements: Align our with ourselves
fungiahh, right21:16
clarkbjeblair: ++21:16
dstufftmordred: FWIW I know a few postgresql people21:16
dstufftif you end up needing help21:16
fungiclarkb: firewall rules?21:16
clarkbfungi: maybe, it is running on the host where the elasticsearch proxy is located21:16
fungiclarkb: oh, nevermind it hits a public endpoint21:16
*** reed has joined #openstack-infra21:16
jog0fungi: what room is it in?21:16
clarkbfungi: but you should eb able to hit it locally21:16
fungijog0: -qa21:16
jeblairfungi: accept ssh host keys?21:16
jeblairfungi: gimme a minute to finish up my patch to your logging patch21:17
jeblairand let's put it into place21:17
jog0fungi: a bug just came through21:17
jog0so should trigger21:17
openstackgerritMonty Taylor proposed a change to openstack/requirements: Remove pin from WebTest
fungijeblair: good point--i don't see any known_hosts management in the earlier puppetry21:17
mordredI thnk the most salient point is that we'd like to stop running databases on hosts21:17
mordredand use cloud databases21:18
anteayaif I want to add a keypair to a running server, do I have to reboot the server?21:18
mordredhub_cap: is RAX growing a postgres aaS any time soon?21:18
mordredvipul: what about HP?21:18
clarkbdstufft: I just ask selenamarie any time the slave postgres servers derp and I am stumped21:18
fungijeblair: clarkb: though would ssh host key issues prevent it from barking in irc? does it use that for more than just leaving comments in gerrit?21:18
clarkbdstufft: but she may not like it if I do that every time a production service derps21:18
anteayaI'll try rebooting21:18
hub_capbut mordred i talked to josh berkus and he says he will be building it21:18
mordredhub_cap: in trove?21:18
hub_caphad lunch w/ him last wk21:18
mordredjosh knows his stuff21:18
jeblairfungi: it wouldn't know what to barck about21:18
openstackgerritChris Hoge proposed a change to openstack-infra/config: Added puppet-vswitch project to Stackforge
mordredso - in any case.21:19
hub_capwe talked about the pain points and how to start21:19
fungijeblair: so it does query something out of gerrit then, not just scrape elasticsearch for info?21:19
mordredI do not like the idea of growing a special snowflake21:19
clarkbfungi: yeah it uses the gerrit event stream too so needs ssh21:19
mordredhow about this:21:19
fungik, addressing that manually for now21:19
clarkbfungi: one comments from jenkins it checks elasticsearch21:19
mordredgo ahead and set it up in puppet to do postgres - BUT21:19
clarkbwe can actually change it to feed off of the zmq stream now that we host it21:19
anteayanope, looks like I have to rebuild the server to add a keypair21:19
mordreddo it such that there would be a postgres host and an ask host21:20
anteayasorry sodabrew this might take a few minutes21:20
sodabrewno problem21:20
mordredthat way we could potentially have just one single postgres machine (or set of machines)21:20
*** pblaho has quit IRC21:20
jeblairmatty_dubs: see mordred's suggestion ^21:20
vipulmordred: Postgres isn't a hgih priority yet for HP either21:20
mordredand the _service_ on the ask host we can kind of manage puppet-wise the same as if it was a cloud database21:20
mordredwhich is to say the hostname and username and database name and password will all be input params21:21
mordredthat way, once a cloud grows a postgres aaS21:21
mordredwe can migrate the data and delete our postgres host21:21
mordredand once the postgres host is up and running, we can migrate ask.o.o into the folks potentially as well21:21
* mordred shudders21:22
dstufftmordred: herou has a cloud pg21:22
dstufftif that's your thing21:22
mordreddstufft: it's not21:22
mordreddstufft: we only use openstack tech to run openstack21:22
dstufftmordred: ok!21:22
mordreddstufft: dogfood tastes good!21:22
clarkbwe only* use ... * except in cases were we don't21:22
mordredclarkb: SSSHHHHH21:22
mordredclarkb: you're taking away my rigteousness21:22
mordredclarkb: you're taking away my righteousness21:22
clarkbmordred: sorry, but I think it is worth mentioning that some stuff isn't possible unfortunately21:23
anteayarats I still don't have your key attached to the vm21:23
mordredAlex_Gaynor, clarkb, dhellmann, fungi: could I get some thoughts on: ?21:23
fungiclarkb: i see yet a new wrinkle here... since ~recheck is /var/run/elastic-recheck, the known hosts file will get blown away along with the rest of the homedir on every reboot21:23
mordredclarkb: yes. indeed21:23
mordredclarkb: but we try real hard21:23
clarkbfungi: gah21:23
clarkbfungi: we can put it in /etc/elastic-recheck instead21:23
*** davidhadas_ has joined #openstack-infra21:23
*** slong has joined #openstack-infra21:24
jeblairwhy is ~recheck there?21:24
dstufftmordred:  I love me some dogfood, I also love me some pragmatism21:24
jeblairfungi, clarkb: how about we put it in /home/recheck?21:25
clarkbjeblair: that works too21:25
clarkbjeblair: I picked an existing dir that was needed for other things21:25
fungijeblair: i am great with that suggestion21:25
clarkband chose poorly21:25
Alex_Gaynormordred: I think there's a seperate review somewhere which pins it to >=2.0 or something, have you coordinated with whoever wrote that?21:25
mordredAlex_Gaynor: oh, is there? lemme to look21:25
*** slong has quit IRC21:25
mordredAlex_Gaynor: thanks21:25
*** slong has joined #openstack-infra21:25
mordredAlex_Gaynor: you are very right21:25
*** weshay has quit IRC21:26
*** davidhadas has quit IRC21:26
mordredAlex_Gaynor: clarkb, fungi <--21:26
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Turn on logging for elastic-recheck
anteayasodabrew rebuilding, will take me about 15 minutes21:26
jeblairfungi: ^ updated the log config file based on what i was using locally21:26
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: log info on job deletion
*** mrodden has quit IRC21:28
*** sodabrew has quit IRC21:29
fungijeblair: how does recheckwatchbot differ from elastic-recheck in that? what are the logger keys exactly?21:29
*** mrodden has joined #openstack-infra21:30
mordredjeblair, fungi: we're about to test whether or not the new propose-requirements-changes patch works in the wild...21:30
fungimordred: great21:30
openstackgerritSergey Lukjanov proposed a change to openstack/requirements: os/requirements should be in sync with itself
clarkbmordred: propose-requirements-changes patch? is this with pypi-mirror?21:32
mordredclarkb: no, this is the translations like "propose requirements sync patches to all projects when requirements changes" job21:34
clarkboh that one21:34
jgriffithmordred: jeblair Any of you know what might have caused this to start happening again: Unknown distribution option: 'install_requires'21:34
jeblairfungi: recheckwatchbot is the current key in use.  i like your idea that it should change though, so i left that in.21:35
jgriffithmordred: jeblair unit tests can't run on Cinder except in the gate, assuming our mirror is saving us from something bad here?21:35
fungijeblair: ahh, got it. the key is defined in python where logging is being called21:37
*** davidhadas_ has quit IRC21:38
*** dkranz has quit IRC21:39
*** davidhadas has joined #openstack-infra21:39
clarkbjgriffith: can you paste the output of a `tox --version && tox` for cinder?21:40
*** mriedem has quit IRC21:40
*** jcoufal has quit IRC21:41
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Use separate script for reviews for bugs
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Add milestone option to reviews_for_bugs
*** dizquierdo has left #openstack-infra21:41
*** dkliban has joined #openstack-infra21:42
sdagueAlex_Gaynor: do you really +A something before we had jenkins results? :)21:46
russellbfor something that trivial i might do it too, heh21:47
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Move elastic-recheck shell account out of /var/run
sdagueI would submit that changing requirements never ends up being trivial :)21:47
sdagueman, that's a query I need to build, how many "trivial" changes +Aed and exploded in the gate21:48
*** jcoufal has joined #openstack-infra21:48
*** julim has quit IRC21:49
*** tvb has quit IRC21:51
*** tvb has joined #openstack-infra21:52
fungiclarkb: jog0: okay, so aside from recheck@logstash.o.o accepting the ssh host key on (which has been done for a while), what else could we check?21:53
clarkbfungi: I'm not sure, jog0 will probably have ideas21:55
anteayaI may have figured it out - I created a dashboard user 'dashboard'@'localhost' in the mysql db when I was the ubuntu user - and then followed some instructions to create a puppet-dashboard user and switch the repo to /opt/puppet-dashboard - I wonder if that would affect the ability of dashboard@localhost to connect to the db?21:56
*** pcm_ has quit IRC21:56
jeblairfungi: i'll take a look21:56
pleia2anteaya: by default mysql doesn't care what system user you are (postgres can)21:57
anteayapleia2: hmmm, okay so that isn't it then21:57
jeblairfungi: did you manually apply logging and restart?21:57
fungijeblair: yes, i actually applied it through my puppet development environment, but also restarted the elastic-recheck service after that as well21:58
jeblairfungi: hrm, i don't see any paramiko logs21:58
jeblairfungi: i'm going to try some interactive debugging on host21:58
fungii still have puppet agent stopped on that server because i've been testing patches that way while proposing them for review, just to make sure there are no other omissions21:58
*** sarob has joined #openstack-infra21:59
*** hashar has quit IRC21:59
fungijeblair: the current state should reflect your last patchset on the logging change as well as on top of that21:59
jeblairthis is much easier to test if i can log in as the account22:00
fungigah, it's actually not running22:00
jeblairi'm going to chsh to bash22:00
jeblairfungi:  i stopped it22:00
fungioh, phew22:00
jog0fungi: first time i ran elastic-recheck I had to make sure review.o.o was added as a host22:00
jeblairthat's what i was trying to say when i said i was going to try interactive debugging22:00
fungii was pretty sure i had made sure it survived my last restart, but freaked out for a sec ;) carry on!22:00
* fungi needs to start grilling dinner. i have a very annoyed girlfriend who was promised i'd start cooking about 1.5 hours ago22:01
*** sarob has quit IRC22:01
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Make elastic-recheck's shell bash
sdaguejeblair: can we land this - ?22:01
fungii'll keep one eye on the computer22:01
*** sarob has joined #openstack-infra22:02
*** hashar has joined #openstack-infra22:03
*** matty_dubs is now known as matty_dubs|gone22:03
*** lcestari_ has quit IRC22:04
*** thomasm has joined #openstack-infra22:04
openstackgerritA change was merged to openstack-infra/reviewstats: Use separate script for reviews for bugs
openstackgerritA change was merged to openstack-infra/reviewstats: Add milestone option to reviews_for_bugs
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Replace queries.json with queries.yaml for elastic-recheck
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: added bug #1224001 to skynet
uvirtbotLaunchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,New]
*** sdake has quit IRC22:07
*** tvb has quit IRC22:07
*** sdake has joined #openstack-infra22:08
*** sdake has quit IRC22:08
*** sdake has joined #openstack-infra22:08
mtreinish^^^ That's a nice effect22:08
uvirtbotmtreinish: Error: "^^" is not a valid command.22:08
*** thomasm has quit IRC22:08
openstackgerritA change was merged to openstack-infra/config: Turn on logging for elastic-recheck
jog0the bots are taking over22:09
*** s1rp has quit IRC22:10
*** s1rp has joined #openstack-infra22:10
*** MoXxXoM has quit IRC22:10
*** datsun180b has quit IRC22:11
*** MoXxXoM has joined #openstack-infra22:11
*** datsun180b has joined #openstack-infra22:11
fungijeblair: oho! how did i not notice that the json symlink into the git repo there was broken. was that all it needed?22:12
anteayathe sodabrew dashboard is up again with 3 workers working22:13
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Increase verbosity in elastic-recheck debug log
clarkbdid I get the symlink wrong/22:13
jeblairfungi: yes, that should be it22:13
jeblairclarkb: no, i changed it from json to yaml this morning22:13
jeblairclarkb: but forgot about the config change22:13
anteayaand I lost him, too slow22:13
*** esker has quit IRC22:14
jeblairjog0, clarkb, fungi, sdague, mtreinish: to the best of my knowlegde, elasticrecheck should be operational now22:14
jeblairit just spoke22:14
anteayatotal of 6 workers working now22:15
russellbis this broken, or do I have to log in now to see anything?
jog0and its not running on my desktop anymore22:15
fungiclarkb: jeblair: the other missing bit is we still need to decide on a good way to manage known_hosts entries for puppeted things. we22:15
fungiwe're still manually accepting those everywhere22:15
clarkbfungi: ya22:15
clarkbreed: can you look at russellb's question?22:15
jeblairfungi: maybe we can brainstorm something involving sshfp records22:16
*** jerryz has quit IRC22:16
jeblairsdague: neutron-pg aprvd22:16
*** rockyg has quit IRC22:18
fungijeblair: sshfp would be lovely, but needs dnssec22:18
jog0does the bot auto pull from git every time there is a change?22:19
clarkbfyi designate intends on supporting dnssec and zone file uploads22:19
*** rockyg has joined #openstack-infra22:19
*** tvb has joined #openstack-infra22:19
clarkbso we could possibly win all the things22:19
fungiclarkb: that's stellar22:19
* fungi is eager to use it22:19
*** sodabrew has joined #openstack-infra22:20
jeblairjog0: when we land all the outstanding changes, then it should be updated within 10-25 minutes after a change22:20
jog0jeblair: nice22:20
jeblairjog0: (the salt work that's going on is intended to reduce that)22:20
*** tizzo has quit IRC22:20
jog0btw for a query file change you don't need to restart the service22:21
jog0so when we pull out the query file ...22:21
*** tizzo has joined #openstack-infra22:21
jgriffithclarkb: sorry... stepped out22:22
jeblairjog0: actually, it should be 0-10 minutes, now that i think about it22:22
jeblairjog0: the way clarkb set it up it's just a symlink into the git checkout, and the git checkout is updated if needed every puppet run, which is on a 10 minute interval22:22
jog0jeblair: nice not bad22:23
*** rnirmal has quit IRC22:23
clarkbjgriffith: hmm you have new tox which comes with new setuptools... not sure what is going on there22:24
*** SergeyLukjanov has quit IRC22:24
clarkbmordred: ^22:25
*** vipul is now known as vipul-away22:26
*** DennyZhang has joined #openstack-infra22:26
*** thedodd has quit IRC22:29
openstackgerritA change was merged to openstack-infra/config: start voting on neutron-pg job
openstackgerritA change was merged to openstack-infra/config: Move elastic-recheck shell account out of /var/run
openstackgerritA change was merged to openstack-infra/config: Make elastic-recheck's shell bash
*** jerryz has joined #openstack-infra22:33
openstackgerritJames E. Blair proposed a change to openstack-infra/elastic-recheck: Remove stray backslash
openstackgerritA change was merged to openstack-infra/elastic-recheck: added bug #1224001 to skynet
uvirtbotLaunchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,New]
reedrussellb, ah! thanks for spotting that22:34
reedrussellb, it is being upgraded22:34
russellbah, k22:34
reedrussellb, anything in particular you're looking for?22:34
russellbnah, it's fine22:34
anteayaso sodabrew is in the puppet-dashboard server22:35
anteayaand we have 45 nodes reporting22:35
anteayaand 71 pending tasks22:35
anteayawith 6 workers running22:35
sodabrew@anteaya So I see the pending tasks, but the reports are in fact being processed in a timely manner. I'm wondering if may have made some error in the update of the delayed_job gem.22:35
anteayasodabrew: that is an interesting thought22:35
anteayaany idea how I could track that down?22:35
*** vipul-away is now known as vipul22:35
reedrussellb, we fixed a bunch of issues with it, improved reports...22:36
sodabrewI'm digging in a bit, looking at what query generates that number22:36
anteayasince pending tasks are going to just keep accumulating until we run jobs:clear22:36
anteayagreat thanks22:36
anteayaI'll keep off the server while you are working, too many cooks and all that22:36
reedrussellb, it should be open now22:37
openstackgerritA change was merged to openstack-infra/config: Replace queries.json with queries.yaml for elastic-recheck
openstackgerritA change was merged to openstack-infra/config: Increase verbosity in elastic-recheck debug log
*** davidhadas has quit IRC22:38
*** davidhadas has joined #openstack-infra22:39
jgriffithclarkb: mordred thanks!  That gave me enough of to work with22:40
russellbreed: cool, in now.  noticed that some project links on this page are an IP address instead of -
jgriffithclarkb: mordred uninstalled setuptools and tox, reinstalled and good to go22:40
clarkbjgriffith: good to know22:40
reedrussellb, yep, one of the reasons why it was closed :)22:40
sodabrew@anteaya The 70 jobs are ones that failed. Looks like there's no rake provided by delayed_job to inspect failures22:40
russellbreed: ha, ok, will be gentle then :)22:40
reedit's being fixed now22:40
*** tizzo has quit IRC22:40
russellbsounds good, thanks!22:40
anteayasodabrew: suggestions on going forward?22:41
anteayawe might have to run jobs:clear on a cron of some kind maybe?22:41
sodabrewthey are always reporting this probelm: Validation failed: Host already has a report for time and kind22:41
anteayahow difficult would it be to offer a rake task for delayed_job to inspect failures?22:42
clarkbanteaya: sodabrew: I think I know what that is. Our slaves share a single cert22:42
anteayayeah, i see that a lot22:42
clarkbthat confuses the dashboard22:42
sodabrewdo they all have the same cert hostname?22:42
clarkbsodabrew: different hostnames same certname22:42
*** OlivierSanchez has quit IRC22:42
*** hemna has quit IRC22:42
*** ojacques has joined #openstack-infra22:42
sodabrewoh, yeah that'll do it i think. generally the certname should match the hostname, right?22:43
sodabrewi think a lot of puppet docs use those two almost interchangeable22:43
*** hemna has joined #openstack-infra22:43
*** amotoki has quit IRC22:43
clarkbsodabrew: right, but there is no guaruntee they will be identical or that they will be unique...22:43
anteayaclarkb: so live with the queue of pending tasks and clear them from time to time or unique certnames?22:44
clarkbwe are trying to move to unique certnames but that is slow going22:44
*** tvb has quit IRC22:44
anteayaso for now, it appears that the sodabrew dashboard is working as well as it is going to22:44
anteayaso run jobs:clear occasionally22:44
anteayaand then when unique certnames kick in, remove the jobs:clear job22:45
anteayaany dissenting arguments?22:46
anteayasodabrew: how many workers should I be running, any thoughts?22:46
anteayaright now I have 6 running22:46
sodabrew@anteaya Meanwhile, I'll make myself a TODO task to have some kind of failed jobs inspector rake task22:47
anteayado I run into issues if I have too many running?22:47
anteayasodabrew: thanks, let me know if I can help or be a guinea pig for that22:47
anteayaI'm going to try starting the server with --port=80 on it22:48
anteayaclarkb would like port 8022:48
*** mrodden has quit IRC22:49
lifelessmordred: btw you're not lurking in openstack-cinder22:49
lifelessmordred: we just saw an interesting thing there - tox can fail if the package can't be built outside the venvs, which means we really shouldn't be building it outside the venvs, rather we shoud make a venv for building it.22:50
anteayasodabrew: it says port is in use or requires root privileges, I was sudo when I ran the command - any thoughts?22:50
*** sodabrew has quit IRC22:51
anteayaclarkb ^22:51
*** sodabrew has joined #openstack-infra22:52
anteayabut I had to sudo su to get that to work22:52
anteayanot sure if that is to your liking or not22:52
*** jcoufal has quit IRC22:53
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Make test_queries save stdout in tox testr
mordredlifeless: yes. that's why I've been putting in the skip-sdist patches to people's tox.ini files22:54
lifelessmordred: ah, so cinder may be missing that?22:55
lifelessjgriffith: ^22:55
mordredlifeless: yup22:55
jgriffithlifeless: mordred :)22:55
*** tvb has joined #openstack-infra22:55
jgriffithlifeless: mordred I'll add22:56
jgriffithunless you're already on it22:56
dstufftmordred: what does skipsdist do22:56
mordredjgriffith: I'm on it22:56
jgriffithmordred: k22:56
jgriffithmordred: thank you22:56
jgriffithlifeless: thanks to you as well22:56
mordreddstufft: it causes tox to run pip install -e . _inside_ of the venv after it's already installed the deps22:56
dstufftso it's an alias for usedevelop22:57
mordredit's in two pieces22:57
*** reed has quit IRC22:57
mordredskipsdist is one part22:57
mordredit skipts the sdist part22:57
*** reed_ has joined #openstack-infra22:57
mordredusedevelop is the other part22:57
mordred(they made me split it)22:57
mordredbut yes22:57
anteayathanks so much for your help sodabrew22:57
anteayawe always have many pupptety type questions22:58
dstufftI think that's a bad idea, there are differences between a package installed with -e and installed normally; Why don't your packages build?22:58
anteayayou are always welcome to hang out and lurk22:58
*** cody-somerville has quit IRC22:58
anteayaand if you ever want to know anything about openstack, please be sure to ask22:58
jgriffiththank you sir22:58
sodabrewContext question: is this the channel for those working on itself?22:58
mordreddstufft: setup_requires22:58
anteayawe are the infrastructure team22:59
mordreddstufft: means that you have to run first to be able to get the deps you need to run setup.py22:59
anteayawe keep gerrit, and zuul and jenkins running22:59
anteayaand all the other little pieces22:59
anteayaincluding puppet and salt22:59
mordreddstufft: they should all build/install sanely outside of that env too22:59
mordreddstufft: and we have tests to double-check that22:59
mordredbut dev laptops are notoriously ... unpredictable23:00
anteayageneral customer service, any questions on testing and helping folks to find the right place for something23:00
sodabrewok, gotta get back to the day job. au revoir23:00
*** sodabrew has left #openstack-infra23:00
dstufftmordred: sounds like tox should do a egg_info before doing a sdist23:00
mordredeven that would require to run and then download its setup_requires23:00
mordredvia easy_install23:00
mordredwhich will helpfully drop an egg into the source dir23:01
mordredwhich in some other contexts will lead to the egg being seen and subsequently not installed into the venv when requested :)23:01
mordreddstufft: there are many many things that _should_ be different here23:01
dstufftmordred: I know that feel23:01
mordreddstufft: yah. it's when people are like "omg, that's terrible, why would you ever?" and you're like "trust me, I know. It is terrible. It's just less terrible than the other choice right now"23:02
mordredand then you hit your face with a large animal23:02
dstufftsomeday this small corner will be better23:03
dstufftmordred: oh yea, pip pep isn't getting a backport23:03
dstufftnot yet anyways23:03
*** DennyZhang has quit IRC23:04
*** hogepodge has quit IRC23:04
*** hogepodge_ has joined #openstack-infra23:04
* mordred cries23:05
dstufftmordred: does openstack support windows in any capacity?23:05
dstufftlike clients or anything23:06
*** tvb has quit IRC23:08
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove stray backslash
*** rcleere has quit IRC23:08
mordreddstufft: sure23:08
mordreddstufft: I believe both rackspace and hp clouds have windows guests23:09
lifelessand the python clients should run on windows just fine23:09
mordredand microsoft have been spending a bunch of effort to make sure openstack supports hyper-v23:09
lifelessand there is murana23:09
mordredwhich thing is murano again?23:09
dstufftmordred: no I mean where you'd install some component of openstack on windows23:09
lifeless'install windows clusters for you'23:09
mordreddstufft: I don't think even msft have put too much effort into that23:09
dstufftmordred: basically I'm going to make the windows installation story for pip (and setuptools) better, so idk if that'll affect you guys or not23:09
lifelessit will23:10
mordredit's pretty linux-centric in design23:10
lifelessfor the clients23:10
mordredfor the clients, sure. although we don't actually have windows in our target support matrix at all23:10
mordredbut I'm _certain_ it will make people happy for the story to be better23:10
lifelessthey are the first against the wall23:10
*** dkliban has quit IRC23:10
mordredI don't grok your new zealand language23:10
lifelessto support windows23:11
lifelessclients first23:11
lifelessI realise it could be misinterpreted :)23:11
*** cody-somerville has joined #openstack-infra23:12
mordredah. they == clients23:12
mordredyes. same with python3 - clients first23:12
mordredI am not expecting large amounts of clamoring for windows gating for the clients in the near term23:12
*** sarob has quit IRC23:12
mordredwhich makes me glad, because I have no idea how we'd do that - it's sort of against current policy23:13
*** datsun180b has quit IRC23:13
*** sarob has joined #openstack-infra23:13
mordredand I'd rather not have to shoot that down- it would be much nicer if just nobody ever wanted it :)23:14
dstufftlifeless: mordred cool, , well I'm putting the wheels in motion to make a unified (pip + setuptools) msi installer, and I now have a signing key with which to sign it; So there's that at least :/23:14
dstufftmight do the same thing for OSX, though curl is not hard there23:14
mordreddstufft: well, honestly, woot23:16
mordredbecause the windows pushback in the easy_install area is just frustrating23:17
dstufftmordred: we haven't given up on 2.7 btw23:17
dstufftWe're defering it23:17
dstufftso we can get 3.4 in becuase the timeline is shorter on that23:17
*** tvb has joined #openstack-infra23:17
mordredI think that's a good choice23:17
*** sarob has quit IRC23:17
*** mrda has joined #openstack-infra23:17
* anteaya wanders away ... dinner23:22
fungidstufft: i can't wait for bootstrapped pip in python. hopefully in about a decade when most stable server distros are using python >= 3.4 by default, we'll all be better off for it ;)23:27
*** mrda has quit IRC23:27
dstufftfungi: yea :[23:28
fungiso, of course, even more excited about an eventual 2.7 backport. but baby steps i guess23:28
*** tvb has quit IRC23:28
dstufftyea it's a mixed emotions PEP for me23:28
dstufftI got emotionally invested in it, and it got half way accepted23:28
*** tizzo has joined #openstack-infra23:30
*** tvb has joined #openstack-infra23:35
*** melwitt has quit IRC23:35
*** melwitt1 has joined #openstack-infra23:35
fungiwell, still seems like progress, even then23:37
*** hemna is now known as hemnafk23:37
*** UtahDave has joined #openstack-infra23:37
*** davidhadas has quit IRC23:38
*** tvb has quit IRC23:39
*** davidhadas has joined #openstack-infra23:39
*** mrda has joined #openstack-infra23:40
*** hashar has quit IRC23:40
*** sarob has joined #openstack-infra23:41
*** dims has quit IRC23:44
*** thingee_zzz is now known as thingee23:51
*** matsuhashi has joined #openstack-infra23:58
*** dims has joined #openstack-infra23:59
*** nicedice has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!