Wednesday, 2015-04-29

clarkbbut I too do not grok the ruby00:00
*** peristeri has quit IRC00:00
mordredalso - what's a defaultfor declaration?00:00
fungiclarkb: should be safe to undo the fw rule and reenable puppet agent on the frontends, yeah00:01
fungii've been sniffing for that source address and see nothing coming from it now00:01
clarkbmordred: Ithink that goes in your puppet manifest so that puppet knows what a default for a resource is?00:01
clarkbfungi: doing that now00:01
* nibalizer punts to crinkle 00:02
clarkband done00:02
* fungi too is sorely lacking in rupies00:02
fungior rupees00:02
*** zz_dimtruck is now known as dimtruck00:03
clarkbmordred: I was wrong
*** melwitt has quit IRC00:03
nibalizerhrm that docs should probably use osfamily instaed of operatingsystem00:04
clarkbmordred: looks like that goes in your provider and you would basically say defaultfor :somefact => value00:04
clarkbmordred: in this case I think defaultfor :useexim => true00:04
clarkbwhere I just made up that mapping00:04
mordredis useexim a fact?00:05
mordredyeah - I need a fact to match on00:05
*** rmcall has quit IRC00:05
*** camunoz has quit IRC00:05
*** camunoz has joined #openstack-infra00:05
clarkbbut I need to afk now, if I don't do grocery shopping I won't have any food00:06
openstackgerritMonty Taylor proposed openstack-infra/puppet-mailman: Add non-alias-modifying mailman provider
mordredok. there is the remove-the-method thing00:06
mordredI don't see a way to do defaultfor that makes sense00:06
mordredalthough Iagree with teh theory00:06
mordredI'd suggest that "what mail system is in use" should be a fact00:06
clarkbmordred: ya I think that would be the correct way to defaultfor but would need the fact to be implemented00:07
mordredbecause honestly - exim and sendmail and postfix and qmail are different00:07
clarkbso I am good setting the provider for now and getting that in to00:07
*** rmcall has joined #openstack-infra00:09
mordredSpamapS: so - I have a patch up you might be interested in00:09
mordredSpamapS: 17759900:09
*** otter768 has joined #openstack-infra00:10
mordredSpamapS: if you apply that, it's possible to source a bunch of OS_ env vars that correspond to a real cloud account on a real cloud00:10
mordredand then run the functional tests in shade00:10
mordredSpamapS: I have tested the current functinoal tests against an account at rax00:10
*** rmcall has quit IRC00:10
mordredSpamapS: so once you get an account, I suggest trying that00:10
ianwnow the crisis is over, can someone look at (Fix swapiness sysctl set) ... centos7 is unstable now it's doing more than smoke testing and i want to make sure i'm getting consistent tests00:11
mordredSpamapS: you'll want these set:
mordredianw: looking00:11
mordredianw: +A00:12
openstackgerritMerged openstack-infra/project-config: Correct output path for infra-site index
ianwmordred: thanks, i'll work on getting the +e checking back around all the ansible stuff00:13
ianwwe shouldn't be missing typos like that00:13
*** sputnik13 has quit IRC00:13
*** otter768 has quit IRC00:14
*** claudiub has quit IRC00:15
*** salv-orlando has quit IRC00:16
* crinkle wonders what was punted to her00:20
*** rmcall has joined #openstack-infra00:21
nibalizerclarkb: mordred would like one of us to just push a new patchset to although maybe monty did it00:21
*** mtanino_ has quit IRC00:21
*** achanda has joined #openstack-infra00:23
mordredI did the main part00:23
mordredbut there was a comment about defaultfor that I could not figure out a tangible way to implement00:23
mordredsince there are no relevant facts to attach it to00:24
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: Put input variables into ansible inventory
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: Move all the ansible calls into playbooks
mordredclarkb: fixed your comments ^^ also left a response where I mostly agree with you about whether or not eh patches are a good idea00:26
*** armax has joined #openstack-infra00:29
*** bknudson has quit IRC00:31
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Fix swapiness sysctl set
*** david-lyle has quit IRC00:32
ianwmordred: doh, i think i rebased that, appreciate if you could look00:32
*** Somay has quit IRC00:34
nibalizermordred: :kernel => 'Linux' /mike drop00:34
greghaynespoor mik00:35
mordrednibalizer: wow. yeah?00:35
nibalizermordred: why not its a site specific configuration00:35
crinklethat was sort of what i had in mind00:35
crinkleotherwise you'd have to give provider => noaliasmailman to every resource that used that type00:35
crinklei'm not sure if there's a more graceful way to do it00:36
* crinkle can ponder it later00:36
nibalizerclarkb: well could set a default provider in the manifest00:36
*** Somay has joined #openstack-infra00:36
openstackgerritMonty Taylor proposed openstack-infra/puppet-mailman: Add non-alias-modifying mailman provider
mordredlike that?00:36
crinklemordred: nibalizer's suggestion is probably less hacky00:37
*** Somay has quit IRC00:37
* nibalizer will keep providing terrible suggestions00:37
*** jtriley has joined #openstack-infra00:37
nibalizeralso is it :linux or "Linux"? I can't tell from the docs00:38
crinkle:linux i think00:39
* crinkle not sure if it matters00:39
*** tiswanso has joined #openstack-infra00:41
*** jtriley has quit IRC00:42
*** tiswanso_ has joined #openstack-infra00:45
*** tiswanso has quit IRC00:46
*** asettle has joined #openstack-infra00:49
*** Somay has joined #openstack-infra00:49
*** bswartz has quit IRC00:53
*** bswartz has joined #openstack-infra00:54
*** asettle has quit IRC00:54
*** asettle has joined #openstack-infra00:56
*** asettle has quit IRC00:56
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script
*** signed8bit has joined #openstack-infra00:58
*** asettle has joined #openstack-infra00:59
*** tjones1 has joined #openstack-infra01:01
*** zhiwei has joined #openstack-infra01:01
*** xyang1 has quit IRC01:05
*** emagana has joined #openstack-infra01:06
openstackgerritMonty Taylor proposed openstack-infra/shade: Fix functional tests to run against live clouds
mordredjhesketh: if you're bored ^^01:09
*** woodster_ has quit IRC01:10
*** emagana has quit IRC01:10
*** emagana has joined #openstack-infra01:11
jheskethmordred: I'm never bored, but always happy to take a look01:14
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script
*** salv-orlando has joined #openstack-infra01:17
*** spzala has joined #openstack-infra01:20
*** david-lyle has joined #openstack-infra01:20
*** tjones1 has quit IRC01:21
*** salv-orlando has quit IRC01:22
*** yamamoto has quit IRC01:24
*** jamesmcarthur has joined #openstack-infra01:24
*** yamamoto has joined #openstack-infra01:24
*** erikmwilson has quit IRC01:25
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script
ianwbash really needs goto01:25
openstackgerritAngus Salkeld proposed openstack-infra/project-config: Add missing ceilometer services to heat job
mordredianw: that's the sickest thing I've ever heard01:30
greghaynesianw: ive seen hacks that do it01:31
greghaynesIIRC they basically reeval'd the script at that line01:31
greghaynesnot that im encouraging this01:31
ianwmordred: hey, you wrote all this ansible stuff into the scripts :)  "goto cleanup" is the best way to deal with failures01:31
SpamapSany argument that includes bash needs more language features should also acknowledge that kittens _WILL DIE_ as a result of said argument.01:32
mordredianw: heh01:32
ianw <- this is how we should handle errors in the scripts, i think01:32
ianwi'm not 100% how those logs are picked up when the cleanup fn didn't run, but anyway...01:34
openstackgerritMonty Taylor proposed openstack-infra/jeepyb: Add a command to generate configs for hound
mordredok. so, I'm on a plane - which means it's ok for me to waste time ^^01:34
SpamapSianw: technically traps are goto's in bash btw01:36
*** markvoelker has quit IRC01:36
SpamapSclosest thing anyway01:36
SpamapSnot technically.. arguably I suppose01:37
ianwSpamapS: i kind of want to keep away from them, to avoid nesting problems when running arbitrary gate functions others might provide01:37
*** marun has quit IRC01:38
SpamapSianw: wait, what exactly are we bashing here?01:38
*** rmcall has quit IRC01:38
ianwSpamapS: i'm beating in into some sort of workability01:39
*** otter768 has joined #openstack-infra01:41
SpamapSianw: yeah, if you could nest traps easily that would be useful in this case.01:41
SpamapSI once made it work sort of generically01:41
SpamapSthen recoiled in horror and threw the implementation into a boiling vat of sulfuric acid01:42
*** jtriley has joined #openstack-infra01:43
*** jtriley has quit IRC01:48
*** yamahata has quit IRC01:48
*** signed8bit has quit IRC01:48
*** mwagner_lap has quit IRC01:49
*** baoli has joined #openstack-infra01:49
*** dims_ is now known as dims01:52
*** Somay has quit IRC01:52
*** baoli has quit IRC01:53
*** baoli has joined #openstack-infra01:53
*** camunoz has quit IRC01:54
*** Somay has joined #openstack-infra01:55
*** yamamoto has quit IRC01:56
*** bnemec has quit IRC01:56
*** asettle has quit IRC01:57
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script
greghaynesmordred: just released all the dib goodness01:58
*** bnemec has joined #openstack-infra01:59
SpamapStest 'sploded nice first try01:59
SpamapSgreghaynes: may god have mercy on dib's soul :)02:00
greghaynesand its users02:00
*** dannywilson has joined #openstack-infra02:02
SpamapSmordred: wait.. is there no v2 CLI in glanceclient?02:03
*** harlowja is now known as harlowja_away02:03
*** sputnik13 has joined #openstack-infra02:04
greghaynesso, AIUI, v2 means upload to swift, and task-create to import02:04
SpamapSright but task-create where?02:04
SpamapSglanceclient has no task-anything02:04
greghaynesglance task-create02:04
greghaynesmine does02:04
SpamapSubuntu@devstack-trusty:~/shade$ glance --version02:05
*** dimtruck is now known as zz_dimtruck02:05
greghaynesSpamapS: You may need to set export OS_IMAGE_API_VERSION=202:06
SpamapSOh thats just insane02:06
*** kiseok7_ has quit IRC02:06
SpamapSto hide it from the HELP?02:06
greghayneswelcome to the jungle02:06
*** mwagner_lap has joined #openstack-infra02:06
SpamapS"its a secret!!!"02:06
SpamapSI'm upgrading to trunk first02:06
SpamapSjust to see02:06
*** camunoz has joined #openstack-infra02:07
greghaynesim on 0.17.002:07
SpamapSof course now I see my system cryptography upgrading and I'm eh-scared02:07
* greghaynes needs to afk and dinner for a couple02:07
*** HeOS has quit IRC02:09
*** HeOS has joined #openstack-infra02:10
*** baoli has quit IRC02:11
dimsanyone poked at this error?02:11
dims"fatal: unable to access '': GnuTLS recv error (-9): A TLS packet with unexpected length was received."02:11
SpamapSImportError: No module named glanceclient02:11
SpamapSubuntu@devstack-trusty:/opt/stack/python-glanceclient$ ls -l /usr/local/lib/python2.7/dist-packages/glanceclient02:11
SpamapSso.. confused..02:11
lifelessSpamapS: is /usr/local//.... on your PYTHONPATH ?02:12
lifelesspython -c 'import sys; print(sys.path)'02:12
*** baoli has joined #openstack-infra02:12
SpamapSlifeless: yes02:13
*** woodster_ has joined #openstack-infra02:13
*** sputnik13 has quit IRC02:13
mordredgreghaynes: woot!02:13
mordredSpamapS: also - yes re: v2 being hidden. wtf02:13
SpamapSI notice there's no in glanceclient/02:14
mordredSpamapS: that seems broken02:14
SpamapSso confused02:15
SpamapSthey're all deleted in my git tree02:15
SpamapSno idea why02:15
*** sputnik13 has joined #openstack-infra02:15
mordredSpamapS: because you were a naughty bow02:15
mordredor boy02:15
*** sputnik13 has quit IRC02:15
SpamapSdevstack punished me?02:15
SpamapS$ glance help02:16
SpamapS'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'02:16
SpamapSthe fun continues02:16
mordredSpamapS: what did you do???02:16
*** dannywilson has quit IRC02:16
SpamapSI think I summoned a Balrog02:16
SpamapSor Gozer02:16
SpamapSeither way I'm pretty sure I have to duel it for eternity now02:16
*** dannywilson has joined #openstack-infra02:17
*** Somay has quit IRC02:18
SpamapSpicking up system six02:19
*** jamesmcarthur has quit IRC02:19
nibalizerSpamapS: are you a god?02:19
*** zz_dimtruck is now known as dimtruck02:20
SpamapSnibalizer: eleventeen02:20
*** ddieterly has quit IRC02:20
SpamapSwouldn't it make sense for the default path to have /usr/local/lib before /usr/lib ?02:21
*** tjones1 has joined #openstack-infra02:21
lifelessSpamapS: yes02:21
*** dannywilson has quit IRC02:21
lifelessSpamapS: also its a lot better to use a venv if you're replacing system things02:22
SpamapSlifeless: have you fallen to the bottom of this rabbit hole before?02:22
lifelessSpamapS: avoids any confusion about versions etc02:22
SpamapSperhaps devstack is using venvs and I missed that02:22
*** tjones1 has left #openstack-infra02:22
lifelesssuspect not02:22
lifeless/u/l/l is not a venv path02:22
SpamapSwell I was trying to install newer glanceclient so I pip installed it02:22
SpamapSbut I didn't bother to pay attention to the one I replaced02:23
lifelessSpamapS: oh02:23
* SpamapS checks the others02:23
lifelessSpamapS: are you running vivid?02:23
SpamapSno they're not venv installed02:23
SpamapSlifeless: trusty vm02:23
lifelessSpamapS: ok phew, eliminates that problem02:23
SpamapSvivid have some new fun?02:23
*** unicell1 has quit IRC02:24
*** Swami_ has quit IRC02:24
SpamapSwow I have no idea how I got from having a working 0.16.x to non-working thing02:30
SpamapSbut at this point, just venv'd and moving on with life02:30
* SpamapS will burn this vm later :-P02:30
*** david-lyle has quit IRC02:30
*** david-lyle has joined #openstack-infra02:31
*** patrickeast has quit IRC02:32
SpamapSyeah so thats amazing, glance's help adapts to the api version02:32
SpamapSinstead of showing you all commands, and saying something like "v2" "v1"02:32
openstackLaunchpad bug 1419695 in pip "Should default to --user to not fail default pip install usage" [Undecided,New]02:33
SpamapSlifeless: oh... my02:34
lifelessupstream -> ubuntu 'please don't do this'02:34
lifelessubuntu -> upstream 'you're too slow, deal with it'02:34
SpamapSthats just amazing02:35
*** sputnik13 has joined #openstack-infra02:35
SpamapSAs is the fact that the glance task API doesn't allow you to cancel things.02:35
lifelessI strongly suspect that sudo python -m pip install -U pip done twice will confuse the hell outta folk02:35
lifelessfirst one will install --user; second will run the new one and install dist wide02:35
lifelessnext run after that will no longer do --user by default.02:35
*** ayoung has joined #openstack-infra02:36
*** david-lyle has quit IRC02:36
SpamapSat least if you hit it enough times, it behaves02:36
lifelessfirst time will no longer upgrade it for other users02:36
mordredhave I mentioned that I avoid python-pip like the plague?02:37
SpamapSyeah seriously, blacklist that thing02:37
SpamapSwe should mirror btw02:37
lifelessI now wonder if they patch up cpython's ensurepip too02:37
SpamapSin every swift installation, just because02:37
mordredI think there is absolutely no reason to install python-pip02:38
mordredpip should always be installed from get-pip02:39
mordredmostly because of insanity like this02:39
mordredthere has literallly not been a time when distro pacakges of python-pip have not screwed us in some way02:39
*** emagana has quit IRC02:40
SpamapSmaybe we need a pip-downloader package02:40
*** sputnik13 has quit IRC02:40
lifelessSpamapS: what, with ensure-pip in it?02:40
SpamapSlifeless: I actually don't know what ensure-pip is02:40
SpamapShave only ever used get-pip.py02:40
lifelessSpamapS: sorry, I meant get-pip02:41
lifelessensure-pip is get-pip inside the std lib02:41
mordredwget ; sudo python == pretty easy02:41
SpamapSoh ok. I thought maybe there was some magic thing that I wasn't aware of :)02:41
lifelessit is :)02:41
SpamapSah nice02:41
lifelessget-pip isn't infallible though02:41
SpamapSand yes I suspect we can't trust distro ensure pip either02:41
mordredbut ^^02:41
lifelessyou'll eyeballs after you read that bug02:41
*** markvoelker has joined #openstack-infra02:41
mordredthat's awesome02:42
SpamapSmordred: how well tested is shade's v2 upload?02:43
*** ashleighfarnham has joined #openstack-infra02:43
SpamapSmordred: cause it's 'sploding for me.02:43
greghaynesSpamapS: did the OS_IMAGE_API thing work?02:44
SpamapSgreghaynes: yes02:45
SpamapSannoyingly well :-P02:45
greghaynesSpamapS: awesome, that should probably be a bug02:45
greghaynesof "at least document the damn thing"02:45
SpamapSgreghaynes: i dunno02:46
SpamapSthe bug is v2 at all02:46
SpamapSthis thing..02:46
greghaynesyes, I think theres a spec for that already though02:46
*** ashleighfarnham has quit IRC02:47
openstackgerritMerged openstack-dev/pbr: Improve
SpamapSis there an extra service that has to be running to run tasks?02:49
*** Somay has joined #openstack-infra02:50
*** asettle has joined #openstack-infra02:51
*** ildikov has quit IRC02:51
*** gyee has quit IRC02:53
mordredSpamapS: I've used it many many times02:55
lifelessmordred: I'm guessing that SpamapS context is devstack02:55
mordredSpamapS: I've even got an ansible playbook that uses it to upload images to all of the clouds02:56
*** nfedotov has joined #openstack-infra02:56
mordredlifeless: yah02:56
SpamapSmordred: so what I'm seeing here is that the task never creates the image, so it's never seen in the list02:56
mordredI have not run it against devstack02:56
mordredSpamapS:the poll/wait timesout?02:56
SpamapSmordred: it seems to fall through immediately.. still debugging02:57
mordredSpamapS: because if youre not using the shade wait function - you'll need to do glance task-show over and over again02:58
SpamapSbut I now have like 9 'processing' tasks02:58
SpamapSI suspect there's a worker that doesn't run02:58
*** Somay has quit IRC02:58
mordredSpamapS: did you add "wait=True" to the image_create call in shade? (I'm just saying things outloud, btw)02:58
SpamapSmordred: yeah, using wait=True02:58
SpamapSmordred: I'm more concerned that the tasks are just sitting there02:59
mordredyay for tseting!02:59
SpamapSand there are so many lovely layers of indirection in glance02:59
SpamapSno idea who actually reads and updates tasks02:59
mordredmagical fairies02:59
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add a hound source code indexer
mordrednibalizer: ^^03:00
mordrednibalizer: I did not split it into a puppet-hound module because I coyuld't figure out a sane split point, since things need to know about each other03:01
*** Somay has joined #openstack-infra03:01
mordrednibalizer: also - don't actually  hack on that - I'm just doing stupid things while I'm on the plane with laggy internet03:02
mordredbut, it turns out, hound is insanely easy to set up03:02
SpamapSok so..03:03
SpamapSthere's this thing in setup.cfg that I don't undersatnd03:03
SpamapSentry_points has 'glance.flows'03:03
SpamapSand 'glance.flows.import'03:03
SpamapSI only understand entry_points for console_scripts03:03
mordredso - that's defining a plugin point called glance.flows03:04
mordredand is also saying that glance is going to install a plugin into that namespace03:04
*** ildikov has joined #openstack-infra03:04
mordredSpamapS: so : glance/async/taskflow_executor.py03:05
SpamapSthat is so helping me03:05
SpamapSit realy is03:05
mordredglance/async/            return driver.DriverManager('glance.flows', task.type,03:05
SpamapSmordred: yeah I'm trying to figure out if and where and how that gets installed/run03:05
mordredis where it looks for drivers ... and glance/async/flows/    extensions = named.NamedExtensionManager('glance.flows.import',03:05
mordredis where it loads the import flow03:05
*** miqui has joined #openstack-infra03:05
SpamapSharlowja_away: if you happen to be paying attention, this is _maddeningly complicated_03:05
SpamapSmordred: do we even run v2 in the gate anywhere?03:07
mordredwell, I would hope that we always run it03:08
mordredAIUI, none of the other services use it yet03:08
*** camunoz has quit IRC03:09
SpamapSreading taskflow's manual now03:10
SpamapSmaybe it can explain how those things happen03:10
lifelessI didn't realise v2 was using taskflow03:12
lifelessperhaps it predates it03:12
SpamapSlifeless: I'm not even sure it does use taskflow.03:13
*** ilyashakhat has quit IRC03:14
SpamapSI've been staring at the guts of glance for 45 minutes and I honestly can't tell what acts on the records in the tasks table.03:14
*** salv-orlando has joined #openstack-infra03:14
*** camunoz has joined #openstack-infra03:17
*** Somay has quit IRC03:18
* SpamapS gets more brute force03:18
*** Somay has joined #openstack-infra03:20
*** dimtruck is now known as zz_dimtruck03:21
*** ddieterly has joined #openstack-infra03:21
*** Somay has quit IRC03:24
SpamapSit almost looks like glance-api might do it03:25
*** salv-orlando has quit IRC03:25
SpamapSbut damn if I can read through the path to that03:25
*** tiswanso_ has quit IRC03:26
*** erikmwilson has joined #openstack-infra03:26
*** ddieterly has quit IRC03:26
*** coolsvap|afk is now known as coolsvap03:27
*** yamahata has joined #openstack-infra03:28
*** jklare has quit IRC03:28
*** erikmwilson has quit IRC03:31
*** jklare has joined #openstack-infra03:31
*** david-lyle has joined #openstack-infra03:32
*** Somay has joined #openstack-infra03:32
*** Swami has joined #openstack-infra03:33
*** changbl has joined #openstack-infra03:33
*** camunoz has quit IRC03:34
*** ivar-laz_ has joined #openstack-infra03:34
*** baoli has quit IRC03:34
*** jklare has quit IRC03:35
*** ivar-lazzaro has quit IRC03:38
*** ivar-laz_ has quit IRC03:38
*** jklare_ has joined #openstack-infra03:39
SpamapSok, so glance-api runs the things03:41
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script
SpamapSwhich is like zomg o_O03:41
SpamapSInvalid: Input does not contain 'import_from_format' field03:41
SpamapSand was throwing an exception03:41
*** bswartz has quit IRC03:43
*** bswartz has joined #openstack-infra03:45
*** kiseok7_ has joined #openstack-infra03:46
SpamapSmordred: so I have a suspicion that whatever they use in RAX is nothing like what is in trunk.03:52
SpamapSmordred: the devstack v2 is super picky about things, and reports 0 errors ever. :-(03:53
SpamapSit's.. almost not worth pursuing.03:53
*** Swami has quit IRC03:53
SpamapSanyway I'm spent03:54
*** camunoz has joined #openstack-infra03:54
* SpamapS quits on v203:54
*** Swami has joined #openstack-infra03:56
*** rm_work is now known as rm_work|away04:00
*** ivar-lazzaro has joined #openstack-infra04:01
*** sdake_ has joined #openstack-infra04:04
*** yfried|afk is now known as yfried04:05
*** sdake has quit IRC04:08
*** armax has quit IRC04:09
*** Somay has quit IRC04:10
*** sdake_ has quit IRC04:11
*** btully has quit IRC04:11
*** rm_work|away is now known as rm_work04:13
*** Swami_ has joined #openstack-infra04:15
*** Somay has joined #openstack-infra04:16
*** Swami has quit IRC04:18
openstackgerritMerged openstack-infra/devstack-gate: Fix swapiness sysctl set
*** stevemar has joined #openstack-infra04:19
*** cody-somerville has joined #openstack-infra04:20
*** nfedotov has quit IRC04:22
*** jyuso1 has quit IRC04:23
*** yfried is now known as yfried|afk04:26
*** asettle has quit IRC04:30
*** yfried|afk is now known as yfried04:31
*** asettle has joined #openstack-infra04:34
*** mwagner_lap has quit IRC04:36
*** miqui has quit IRC04:38
*** btully has joined #openstack-infra04:39
*** yfried has quit IRC04:41
*** beisner has joined #openstack-infra04:43
*** ildikov has quit IRC04:44
*** asettle has quit IRC04:47
*** miqui has joined #openstack-infra04:47
*** miqui__ has joined #openstack-infra04:48
beisnerhi.  ping clarkb / fungi04:49
*** mwagner_lap has joined #openstack-infra04:52
*** miqui has quit IRC04:52
*** unicell has joined #openstack-infra04:53
blahdeblahbeisner: I'm not sure if clarkb & fungi are in .us TZs; lifeless?04:53
lifelessblahdeblah: they are04:54
lifelessblahdeblah: clark is west coast, fungi east.04:54
beisnerfyi, i'm in malta atm04:54
lifelessblahdeblah: 'sup ?04:54
beisnerhi lifeless04:54
lifelessoh, beisner pinging. Hai :).04:54
lifelessthere's a europe admin nowadays, IIRC.04:54
lifelessbeisner: hi; whats up ?04:55
beisnerlifeless, fyi the traffic is related to deploy from git source testing04:55
lifelessbeisner: yeah, we did figure that much :)04:56
lifelesssomething about the requests was particularly expensive AIUI04:57
lifelessI don't think we identified what particular git request was so expensive04:57
*** ildikov has joined #openstack-infra04:57
*** camunoz has quit IRC04:57
beisneri can only imagine the demand rising, so i think we'll need to set up a local daily sync04:59
beisnerinstead of a zillion jobs pulling from you04:59
*** unicell1 has joined #openstack-infra05:00
lifelessso FYI05:00
lifelesswe map src IP to just one git backend05:00
lifelessyou're behind NAT, so *all* your load hits just one server05:01
*** unicell has quit IRC05:01
blahdeblahlifeless: What's the best way to get an initial sync happening so that beisner and co. can get going?05:01
blahdeblahlifeless: All the load is coming from one box anyway. :-)05:01
lifelessblahdeblah: pull from github05:01
lifelessblahdeblah: everything is mirrored there05:01
blahdeblahlifeless: Any idea on the lag?05:01
lifelessblahdeblah: seconds05:01
lifelessblahdeblah: its pushed to synchronously with the pushes out to our own mirror farm05:02
lifelessblahdeblah: if something fails it could be out of date but thats rare05:02
bradmsurely it'd be less lag than being blocked :)05:02
lifelessblahdeblah: and certainly for seeding a local mirror to get going with, it will be straight forward05:02
blahdeblahlifeless, beisner: So it sounds like even just an initial sync from github would be enough to get us going without messing with your load too much.05:03
lifelessI imagine if the source tests are disabled atm that we could remove the block anyway05:03
lifeless(But I can't - I don't have ops access)05:03
lifelessblahdeblah: yeah05:03
lifelessall our CI jobs that work from source work from local mirrors on the test nodes05:03
lifelesswe do an incremental pull to bring them up to date at the start of the CI job05:04
beisnerthat's the right thing to do, we just need to set it up.  but we do have a time-sensitive need to deploy one more today, if that is manageable?05:04
lifelessbut our full clone load is very low05:05
*** mrmartin has joined #openstack-infra05:05
*** camunoz has joined #openstack-infra05:05
lifelessbeisner: There's noone active in the channel at the moment with ops access05:05
lifelessbeisner: you'd be a lot better off doing s/ if you've got something time sensitive to do05:05
lifelessbeisner: I'm just looking up the european sysop05:06
lifelessoh there might be two05:06
lifelessI don't remember05:06
lifelessjhesketh: do you have root?05:07
bradmdo we know they didn't remove the block?  fungi said they'd do it soon, so it might be done?05:07
lifelessworth trying05:07
jheskethlifeless: no sorry05:07
lifelessI'm fairly sure SergeyLukjanov does but might be too early for him05:07
*** dhritishikhar has joined #openstack-infra05:07
*** ajmiller__ has joined #openstack-infra05:09
beisnerthanks lifeless - appreciate the help!05:09
lifelesssounds like the rule is removed:05:10
lifeless"12:01 < clarkb> fungi: doing that now05:10
lifelessso if blahdeblah unblocks at your end, we can find out05:10
*** Somay has quit IRC05:11
blahdeblahlifeless: We'll probably end up with exactly the same behaviour if we just crank it up again...05:12
lifelessblahdeblah: beisner is saying 'one job'05:12
lifelessbeisner: how many jobs were running when we hit the wall?05:12
*** ajmiller_ has quit IRC05:12
*** miqui__ has quit IRC05:13
beisnerwhat's the utc time-ish?05:13
blahdeblah20:50 was when it started05:14
lifelessnow? 051305:14
blahdeblahbeisner: See graphs linked in the ticket05:14
*** abregman has joined #openstack-infra05:14
beisnerok so that was actually not automation.  that was one guy prepping for an internal demo.05:14
beisnerbut i've paused our -from-git  automation for the moment.05:15
*** salv-orlando has joined #openstack-infra05:15
lifelessbeisner: ok so don't do that thing again05:16
lifelessbeisner: how much load would we expect from what you want to do now vs what was happening then ?05:16
*** hyakuhei has joined #openstack-infra05:17
*** yamamoto has joined #openstack-infra05:18
beisneractually i'm quite surprised there wasn't an issue earlier.  we've had concurrency of deploy from git tests at 6 for a couple of weeks.   it's interesting that those haven't run in 2 days as there haven't been any internal triggers.  but one guy prepping a demo ... ha.05:19
*** asettle has joined #openstack-infra05:19
beisnerthose tests have run dozens of times in that time frame.05:20
*** asettle has quit IRC05:20
beisnerat any rate, we'll make adjustments to place nicely05:20
lifelessit may warrant closer investigation into why it was expensive05:21
lifelessdifferent protocol perhaps? git vs https?05:21
jamespagelifeless, the code is using git05:21
jamespagemorning all05:22
lifelessjamespage: both the demo and the automated tests?05:22
lifelessgit:// that is ?05:22
blahdeblah\o jamespage05:22
*** dannywilson has joined #openstack-infra05:22
blahdeblahlifeless: Yep - that's the only port open in the firewall05:22
jamespagelifeless, no difference05:22
jamespagelifeless, we can switch pretty easily05:22
jamespagebeisner, lets just hold the ci tests for now - we can fixup and then re-release05:23
jamespagerelease rather05:23
*** ddieterly has joined #openstack-infra05:23
jamespagehey blahdeblah05:23
*** hdd has joined #openstack-infra05:24
beisnerjamespage, ack, automation on that is on hold05:24
greghayneslifeless: clarkb: was pretty convinced there are certain conditions where a git fetch will baloon in memory size, and then causes everything else to backlog once the box goes into swap05:24
lifelessgreghaynes: yes, the question is why05:25
greghaynesyerp, there was some log digging, nothing was found :(05:25
lifelessgreghaynes: I know :) I was here...05:26
blahdeblahgreghaynes: I've seen that quite a lot on git gc of small-ish repos, but never on a fetch.05:26
*** salv-orlando has quit IRC05:26
*** dannywilson has quit IRC05:26
*** erikmwilson has joined #openstack-infra05:27
*** ildikov has quit IRC05:28
beisnerthx again lifeless.  let us know if you'd like to have us 'load test' again.  ;-)05:28
*** ddieterly has quit IRC05:28
* beisner will be offline for a bit05:28
lifelessbeisner: I think it would be ok to try as long as you keep your eyes on the cactii graph and if it starts to spike shut down your thing05:28
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add errexit to setup & cleanup functions
*** erikmwilson has quit IRC05:31
*** yfried has joined #openstack-infra05:34
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Add errexit to setup & cleanup functions
*** yamamoto has quit IRC05:37
ianw^ 25 revisions that took ... hopefully it's in good shape now05:37
*** dhritishikhar has quit IRC05:39
*** yamamoto has joined #openstack-infra05:40
*** HeOS has quit IRC05:43
*** yfried is now known as yfried|afk05:44
*** rlandy has joined #openstack-infra05:44
*** Krinkle is now known as Krinkle|detached05:45
*** mwagner_lap has quit IRC05:46
*** Somay has joined #openstack-infra05:46
*** yfried|afk is now known as yfried05:49
openstackgerritMerged openstack-dev/pbr: Build all the wheels in one pass rather than many.
*** stevemar has quit IRC05:54
*** yamamoto has quit IRC05:54
*** mrunge has joined #openstack-infra05:54
*** yamamoto has joined #openstack-infra05:54
*** Somay has quit IRC05:55
*** lucap has joined #openstack-infra05:59
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** Longgeek has joined #openstack-infra06:03
*** mwagner_lap has joined #openstack-infra06:03
clarkbcouple things, I did remove the block, anyone can trivially test with `telnet 9418` from that source06:03
clarkbit would probably be helpful to know what was going on at the time as git daemon doesnt really log this well06:04
StevenKnc -vz!06:04
*** otter768 has quit IRC06:04
clarkband yes, I dont think its a matter of scale, Instead I think its a particular (set) request that causes things to fall over06:06
clarkbwe do happen to run a pretty giant test farm against this mirror just fine for the most part06:06
clarkbStevenK or nc :)06:07
*** doude has quit IRC06:07
StevenKclarkb: telnet's escape sequence is annoying, I like:06:08
StevenKsteven@undermined:~% nc -vz 941806:08
StevenKConnection to 9418 port [tcp/git] succeeded!06:08
*** Longgeek has quit IRC06:08
*** yamamoto has quit IRC06:08
clarkbI usually suggest telnet because unlike nc it seems to exist everywhere and with the same options06:09
clarkbso annoying when you get nc and its bsd nc06:09
clarkband nothing works06:09
*** pcaruana has joined #openstack-infra06:09
clarkb(also I removed the block hours ago)06:09
*** isviridov_away is now known as isviridov06:10
StevenKclarkb: I forcibly install the openbsd one everywhere I can. I think -vz works with either06:10
clarkbI will read scrollback in the morning if people want to leave more info, but now its time to return to my book then sleep06:12
*** Swami has joined #openstack-infra06:13
*** bboese has quit IRC06:14
*** jlvillal has quit IRC06:14
*** Swami_ has quit IRC06:14
*** vlaza has joined #openstack-infra06:14
*** jlvillal has joined #openstack-infra06:16
*** zul has joined #openstack-infra06:17
*** Somay has joined #openstack-infra06:17
*** yfried is now known as yfried|afk06:18
*** Somay has quit IRC06:19
*** jamespage_ has joined #openstack-infra06:19
*** bboese has joined #openstack-infra06:20
*** jyuso1 has joined #openstack-infra06:22
lifelesscan I get 2x+2 on ?06:22
lifelessclarkb: jhesketh: ^ - its super shallow06:23
jheskethlifeless: looking06:23
*** ddieterly has joined #openstack-infra06:24
*** zul has quit IRC06:24
*** zul has joined #openstack-infra06:25
*** sushilkm has left #openstack-infra06:27
jheskethlifeless: lgtm, however you're overwriting the existing pip venv whereas all the other tests create a new one06:27
jheskethshould you create a new one for consistency, or was this deliberate06:27
lifelessjhesketh: they all overwrite old venvs06:27
lifelessjhesketh: it looks like they are unique, but they aren't: its in a loop, and mkvenv rm -r's the old tree as its first thing06:28
*** tnovacik has joined #openstack-infra06:28
lifelessjhesketh: the uniqueness of the last component is merely a debugging aid, I think its clear enough from logs where this would get to06:28
jheskethright, but in that loop they are all unique06:28
lifelessjhesketh: I am happy enough to change it, or to not.06:28
jheskethyeah I'm not blocking on it, more highlighting that it's inconsistent06:29
lifelessjhesketh: they're distinct, not unique ;)06:29
lifelessjhesketh: if you see what I mean06:29
*** ddieterly has quit IRC06:29
jheskethlifeless: heh, fair enough06:29
*** jamespage_ has quit IRC06:29
lifelessits more oversight from me06:29
lifelessTBH I'd be inclined to change this to use just one venv path06:30
lifelessto reduce page cache impact06:30
lifelesslog a **** Testing SDIST ****06:30
lifelessor whatever before each one06:30
lifelessfuture work06:30
lifelessso -> with that, inclined to ignore the inconsistency here06:30
lifelessso sigh, oslo-incubator is one of the things that will wedge when pbr releases06:31
*** hashar has joined #openstack-infra06:32
*** salv-orlando has joined #openstack-infra06:34
*** yfried|afk is now known as yfried06:38
*** Swami has quit IRC06:38
*** Somay has joined #openstack-infra06:39
*** ildikov has joined #openstack-infra06:40
*** salv-orlando has quit IRC06:42
*** dhritishikhar has joined #openstack-infra06:42
*** nfedotov has joined #openstack-infra06:44
*** Ala has joined #openstack-infra06:45
*** ssam2 has joined #openstack-infra06:47
*** yfried is now known as yfried|afk06:48
*** jcoufal has joined #openstack-infra06:49
ttxlifeless: the trick is "release" is not an atomic event, it takes time to tag and make Launchpad release pages for everything06:49
ttxtakes about 5 hours to target all fixed bugs to relevant milestones06:50
lifelessttx: mainly API performance?06:50
ttxI plan to start release when I get up Thursday (around 8 UTC) and finish around 13 UTC06:50
lifelessttx: yeah, I'll be fast asleep - which is fine06:50
ttxlifeless: well to be fair it's a lot of bugs :)06:51
lifelessttx: I wasn't in any way complaining about the timing of the release06:51
ttxbut yeah, takes about 2 seconds per bug06:51
lifelessttx: can you ping dims when its done?06:51
ttxlifeless: sure06:51
lifelessttx: 2s per - there's something fundamentally wrong there06:51
lifelessttx: either excessive use of multi-project bugs (which are a big source of issues)06:51
ttxmaybe there is an optimization to be made somewhere06:52
lifelessttx: or possibly not enough clobbering of the ORM anti-features in launchpadlib06:52
lifelessttx: one of the things to watch out for is that any use of an attribute that is a relation triggers an API call06:52
lifelessttx: using attribute_id and maintaining an in-process cache can avoid this06:52
ttxSee on openstack-infra/release-tools06:53
lifelessttx: and make a huge perf difference - but we may be doing that already06:53
*** Somay has quit IRC06:53
ttxin retrospect we should probably not do that (consolidate all milestone stuff onto a single release page per project) but since we historically did that, I'll continue as long as we use Lp06:54
lifelessthat should use the _id trick06:54
lifelessbp.milestone will be doing an API call06:54
lifeless(if its not None)06:54
ttxlifeless: care to propose a patch there ? I'll gladly used a sped-up version06:55
*** BharatK_ has joined #openstack-infra06:55
*** BharatK has quit IRC06:55
ttxlifeless: moving BPs is not very long. Bugs is where the hit is though06:55
lifelessthe assignee etc looksups likewise down on line 12206:55
*** Longgeek has joined #openstack-infra06:56
lifeless(since bt.assignee is a ref)06:56
ttxack. If you have example code (or better, can propose a quick patch), I look forward cutting down on that time06:57
ttxalthough it was not painful, just long :)06:57
lifelessI can't do a patch just now06:58
lifelessI've too much in-head to do the context switch sanely06:58
ttxok, I'll look in launchpadlib litterature06:58
ttxI can relate to that06:58
*** camunoz is now known as camunoz_gone06:58
lifelessI may be able to find you a ref06:59
lifelessI don't think it will be a huge win in this case but its worth doing06:59
lifelessits a shame we need to materialise bug06:59
ttxwell in that code we usually don't use the copytask feature, so we geenrally don't use L121-12607:00
ttxwhen I tried to optimize, realized what took time was bt.lp_save()07:00
lifelessso that means that we're mainly hitting bad perf07:00
lifelesscould you for this release do this?07:01
ttxwhich is why I start early, perf also drops when the US gets up07:01
lifelessput a start/end time.time call around the lp_save07:01
*** ajo has quit IRC07:01
lifelessand log to a trace log07:01
ttxlifeless: sure07:01
lifelessthe start time, duration, bug id07:02
lifelessthen we can give that to wgrant with puppy-dog eyes07:02
*** Longgeek has quit IRC07:02
lifelesssetting a milestone is a rare thing in the API07:02
lifelessmost of the optimisation work gets guided by data07:02
lifelesse.g. rare things have to be REALLLY bad to be spotted07:02
lifelessand setting a milestone is -so- rare I think you'd need to be looking at just the right time07:03
lifelessbut we're going to do several hundred/ thousand07:03
lifelessso lots of data captured07:03
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Add errexit to setup & cleanup functions
*** devvesa has joined #openstack-infra07:04
*** devvesa has quit IRC07:04
lifelessoh, I think its not _id, its _link07:04
lifelessto bt.bug_link07:04
lifelessvs bt.bug, which is an ORM like lazy loading object07:04
lifeless(inferred from
*** devvesa has joined #openstack-infra07:04
*** Longgeek has joined #openstack-infra07:06
*** jamespage_ has joined #openstack-infra07:07
*** jtriley has joined #openstack-infra07:09
*** yfried|afk is now known as yfried07:09
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
lifelessfailing that, look at the descriptor on type(BugTask).bug07:11
*** jamespage__ has joined #openstack-infra07:11
*** MaxV has joined #openstack-infra07:13
*** jamespage_ has quit IRC07:13
*** niedbalski_ has joined #openstack-infra07:13
*** jtriley has quit IRC07:14
*** luqas has joined #openstack-infra07:15
lifelessanyone here happen to know/remember where the  updating from zuul of the stuff being tested, and of /opt/git to trunk appens?07:16
AJaegerlifeless, do you mean this script in project-config: jenkins/scripts/gerrit-git-prep.sh07:18
AJaegerlifeless, it'S called in many jobs as builder gerrit-git-prep07:18
lifelessthat looks like it07:19
lifelessso I have this patch07:19
lifelessin which I was hoping to avoid a chunk of IO07:19
*** yfried is now known as yfried|afk07:19
mordredlifeless: chunk07:20
*** mpavone has joined #openstack-infra07:20
lifelessmordred: 1.mumble GB right07:20
lifeless/opt/stack has all the stale refs07:20
*** nmagnezi has joined #openstack-infra07:20
lifelessso I need a variant of that prep script that works in-place I guess07:20
AJaegerlifeless, check the zuul-cloner line in jenkins/jobs/infra.yaml07:21
openstackgerritIlia Meerovich proposed openstack-infra/jenkins-job-builder: Adding unittest to ownership property
AJaegerlifeless, but better talk with clarkb, mordred or fungi about this. I'll disappear now again and they know this far better than I do.07:22
lifelessAJaeger: yeah, I don't know enough either07:22
lifelessmordred: morning, where are you?07:22
*** yfried|afk is now known as yfried07:22
openstackgerritlifeless proposed openstack-dev/pbr: Use /opt/git directly
openstackgerritlifeless proposed openstack-dev/pbr: Stop testing easy_install behaviour
lifelessjhesketh: if youplease07:23
lifelessmordred: and if you please07:23
mordredlifeless: I have just arrived in Paris07:24
mordredlifeless: and I'm waiting on jeblair to clear customs whilst having a coffee07:24
mordredlifeless: looking at the patches07:24
lifelessParis heh heh... for?07:24
*** aarefiev has quit IRC07:24
*** ddieterly has joined #openstack-infra07:25
mordredlifeless: we're meeting with markmc and the folks who used to be enovance about the work they did on software-factory and making infra reconsumable07:25
mordredlifeless: hopefully to outline a path forward to working more closely together and stuff07:26
*** btully has quit IRC07:26
*** arxcruz has joined #openstack-infra07:27
*** erikmwilson has joined #openstack-infra07:28
mordredlifeless: both lgtm07:28
lifelessmordred: have you applied your +2 to them :)07:28
lifelessmordred: thats all I really care aboot07:28
mordredlifeless: yup07:28
*** mwagner_lap has quit IRC07:28
mordredlifeless: all +2d and whatnot07:29
*** ddieterly has quit IRC07:29
lifelessmordred: so why can't I see the +2 on ?07:29
lifelessmordred: yeah, you went up the stack, to the /opt/git patch which is subtley broken and I need some -infra braintime to fix :)07:30
StevenKI'm going to guess "Because gertty"07:30
*** zul has quit IRC07:32
*** zul has joined #openstack-infra07:32
*** erikmwilson has quit IRC07:33
*** Hal has joined #openstack-infra07:38
*** Hal is now known as Guest6066607:38
*** nfedotov has quit IRC07:38
*** kaisers has quit IRC07:39
*** scheuran has joined #openstack-infra07:39
*** dizquierdo has joined #openstack-infra07:39
*** aarefiev has joined #openstack-infra07:40
*** yfried has quit IRC07:41
*** kaisers has joined #openstack-infra07:41
*** Guest60666 has quit IRC07:42
hasharmordred: jeblair: oh do you guys come to Paris and don't ping me beforehand :D07:42
hasharenjoy your stay!07:42
*** jlanoux has joined #openstack-infra07:43
*** ildikov has quit IRC07:43
lifelessmordred: ping?, sadface -
*** yfried has joined #openstack-infra07:44
*** dguerri is now known as _dguerri07:44
*** ildikov has joined #openstack-infra07:44
*** kaisers has quit IRC07:45
*** mwagner_lap has joined #openstack-infra07:46
hasharlifeless: hello! I started using diskimage-builder at Wikimedia , is there any specific irc channel I could join for that utility ?07:46
lifelesshashar: #tripleo or here07:47
lifelesshashar: or #openstack-dev07:47
hasharah tripleo is probably a good one07:47
*** kaisers has joined #openstack-infra07:48
*** achanda has quit IRC07:48
hasharI am going to send some patches against a bunch of documentation related patches since I am actually reading/using the doc :D07:49
*** ajo has joined #openstack-infra07:51
*** _dguerri is now known as dguerri07:52
*** sergsh has joined #openstack-infra07:53
*** achanda has joined #openstack-infra07:56
*** jeblair has quit IRC07:56
*** jeblair has joined #openstack-infra07:56
*** sergsh has quit IRC07:58
*** jistr has joined #openstack-infra07:58
*** imcsk8|afk is now known as imcsk807:59
*** Somay has joined #openstack-infra08:00
*** jamespage__ has quit IRC08:00
*** derekh has joined #openstack-infra08:01
*** dizquierdo has quit IRC08:02
*** _nadya_ has joined #openstack-infra08:04
*** hyakuhei has quit IRC08:04
*** otter768 has joined #openstack-infra08:05
*** Longgeek has quit IRC08:05
*** imcsk8 is now known as imcsk8|zZz08:06
*** salv-orlando has joined #openstack-infra08:08
*** HeOS has joined #openstack-infra08:08
*** zul has quit IRC08:09
*** otter768 has quit IRC08:10
*** dims has quit IRC08:10
*** notnownikki has joined #openstack-infra08:12
*** sergsh has joined #openstack-infra08:12
*** Somay_ has joined #openstack-infra08:24
*** Hal has joined #openstack-infra08:24
*** Somay has quit IRC08:24
*** Hal is now known as Guest9230408:24
*** zul has joined #openstack-infra08:25
*** yamahata has quit IRC08:25
*** ddieterly has joined #openstack-infra08:26
*** ddieterly has quit IRC08:30
mescanefhello, please add me ( to the gerrit groups of a project regarding the :-)08:32
*** achanda has quit IRC08:33
*** jeblair has quit IRC08:33
*** achanda has joined #openstack-infra08:33
mescaneffungi: ^^ :-)08:35
*** achanda has quit IRC08:35
*** zz_johnthetubagu is now known as johnthetubaguy08:36
DuncanTHi. Who are the review team for the requirements repo? Any chance of getting some eyes on please?08:40
*** woodster_ has quit IRC08:40
*** dmellado has quit IRC08:41
*** johnthetubaguy is now known as zz_johnthetubagu08:44
*** zz_johnthetubagu is now known as johnthetubaguy08:45
*** Mithrandir has quit IRC08:46
openstackgerritSpencer Krum proposed openstack-infra/puppet-openstackci: Add developer guidelines
AJaegerDuncanT, you can check this yourself, got to your change, click on "openstack/requirements" and then "Access"08:50
*** dmellado has joined #openstack-infra08:51
DuncanTAJaeger: That's a handy tip, thanks!08:51
*** jeblair has joined #openstack-infra08:51
*** dizquierdo has joined #openstack-infra08:52
*** mwagner_lap has quit IRC08:55
*** Mithrandir has joined #openstack-infra08:55
*** dhritishikhar has quit IRC08:56
*** jtriley has joined #openstack-infra08:58
*** deepakcs has joined #openstack-infra08:59
*** dhritishikhar has joined #openstack-infra09:01
*** jtriley has quit IRC09:02
*** dhritishikhar has quit IRC09:03
*** dhritishikhar has joined #openstack-infra09:03
*** e0ne has joined #openstack-infra09:06
*** Longgeek has joined #openstack-infra09:06
*** e0ne is now known as e0ne_09:06
*** dtantsur is now known as dtantsur|brb09:08
*** Somay_ has quit IRC09:08
*** fhubik has joined #openstack-infra09:10
*** mwagner_lap has joined #openstack-infra09:11
*** Longgeek has quit IRC09:12
*** Longgeek has joined #openstack-infra09:13
*** MaxV has quit IRC09:13
*** jyuso1 has quit IRC09:14
*** jyuso1 has joined #openstack-infra09:14
*** e0ne_ is now known as e0ne09:14
SergeyLukjanovlifeless, hey, what's up?09:14
lifelessSergeyLukjanov: nothing :)09:15
lifelessSergeyLukjanov: we were just talking about whether some stuff had or had not happened and who could do it if it hadn't09:15
SergeyLukjanovlifeless, ok :)09:15
*** MaxV has joined #openstack-infra09:16
*** jyuso has joined #openstack-infra09:17
*** jyuso1 has quit IRC09:17
notnownikkilifeless, hi, did i get the right hp email for you? I sent some stuff about pip dependency handling09:22
*** jyuso has quit IRC09:24
*** ddieterly has joined #openstack-infra09:26
*** zul has quit IRC09:27
*** erikmwilson has joined #openstack-infra09:28
*** ddieterly has quit IRC09:31
*** tnovacik has quit IRC09:31
*** afazekas has quit IRC09:32
*** afazekas has joined #openstack-infra09:32
lifelessnotnownikki: I don't know. What email address did you use? BTW,for open source stuff, my address is best09:33
*** erikmwilson has quit IRC09:33
notnownikkilifeless, rbtcollins@, but I'll fwd to your other email09:34
*** jamespage_ has joined #openstack-infra09:34
*** fhubik is now known as fhubik_afk09:34
*** fhubik_afk is now known as fhubik09:35
lifelessnotnownikki: ah, found it09:36
*** jamespage_ has quit IRC09:39
*** aswadr has joined #openstack-infra09:44
lifelessnotnownikki: I'll reply to you shorlty09:45
*** markmcclain has quit IRC09:49
*** jamespage_ has joined #openstack-infra09:50
*** zul has joined #openstack-infra09:51
*** dims has joined #openstack-infra09:51
*** reed has quit IRC09:56
*** yfried is now known as yfried|afk09:58
*** markmcclain has joined #openstack-infra09:59
*** sergsh has quit IRC10:02
*** jeblair has quit IRC10:03
*** e0ne is now known as e0ne_10:03
*** jeblair has joined #openstack-infra10:03
*** sergsh has joined #openstack-infra10:03
*** e0ne_ is now known as e0ne10:05
*** jamespage_ has quit IRC10:05
*** jamespage_ has joined #openstack-infra10:05
*** otter768 has joined #openstack-infra10:06
*** mtreinish has quit IRC10:07
*** jamespage_ has quit IRC10:08
*** doude has joined #openstack-infra10:09
*** tnovacik has joined #openstack-infra10:10
*** otter768 has quit IRC10:10
*** shardy_ has joined #openstack-infra10:13
*** shardy has quit IRC10:15
*** ociuhandu has joined #openstack-infra10:15
lifelessnotnownikki: replied10:17
lifelessnotnownikki: you did have the right HP address btw10:17
*** dtantsur|brb is now known as dtantsur10:18
notnownikkilifeless, thank you! looks like I have a lot of reading to do... :)10:18
*** shardy_ has quit IRC10:19
*** yfried|afk is now known as yfried10:19
*** doude has quit IRC10:19
*** zul has quit IRC10:19
*** shardy has joined #openstack-infra10:19
lifelessright, waaaay past EOD for me, gl.10:21
*** hashar is now known as hasharLunch10:26
*** ddieterly has joined #openstack-infra10:27
*** wznoinsk has quit IRC10:27
*** doude has joined #openstack-infra10:27
*** reed has joined #openstack-infra10:28
*** yfried is now known as yfried|afk10:31
*** ociuhandu has quit IRC10:31
ekarlsowhat's wrong with something when: py27 develop-inst just hangs ?10:31
openstackgerrityolanda.robla proposed openstack-infra/system-config: Bump stankevich-python version
*** ddieterly has quit IRC10:31
*** ociuhandu has joined #openstack-infra10:34
*** yfried|afk is now known as yfried10:34
*** sputnik13 has joined #openstack-infra10:35
*** mwagner_lap has quit IRC10:41
*** dhritishikhar has quit IRC10:42
*** dhritishikhar has joined #openstack-infra10:43
*** dhritishikhar has quit IRC10:44
*** mmedvede has quit IRC10:45
*** dhritishikhar has joined #openstack-infra10:45
*** jtriley has joined #openstack-infra10:47
*** jtriley has quit IRC10:51
*** mmedvede has joined #openstack-infra10:53
*** _nadya_ has quit IRC10:54
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
openstackgerritSean Dague proposed openstack-infra/project-config: fix git_timed retry logic
*** mwagner_lap has joined #openstack-infra10:59
sdaguefungi / mordred / jeblair ^^^^^10:59
*** dhritishikhar has quit IRC10:59
sdagueI noticed another git clone fail this morning, then realized we had the code wrong yesterday10:59
*** wznoinsk has joined #openstack-infra10:59
openstackgerritSean Dague proposed openstack-infra/project-config: fix git_timed retry logic
AJaegersdague, so, we have to handle it differently than ?11:01
AJaegerOr should we change that one as well?11:02
sdagueso, honestly, that function does a very specific thing that ianw needed, and not the thing we care about11:02
AJaegerAh, wrong copy & paste ;)11:02
AJaegerJust a nit - I would write "exited" but I'm not giving a comment for that. Just in case you want to update again...11:03
sdaguecommit message?11:03
*** zhiwei has quit IRC11:03
sdagueok, sure will fix11:04
AJaegerAlso, we now have two echos there using the git arguments, what about compressing the two lines11:04
openstackgerritSean Dague proposed openstack-infra/project-config: fix git_timed retry logic
AJaegerJust nits...11:04
sdagueyeh, so it was an attempt to be a tactical patch, I have a whole function rewrite patch as well11:04
*** yfried is now known as yfried|afk11:05
sdagueI think double output lines are fine11:05
sdaguefor now11:05
*** yfried|afk is now known as yfried11:06
*** maishsk has joined #openstack-infra11:10
maishskGood afternoon.11:10
maishskWho can help me out with a mailman problem?11:12
*** ociuhandu has quit IRC11:13
*** dizquierdo has quit IRC11:14
*** kashyap has joined #openstack-infra11:14
openstackgerritMerged openstack-infra/project-config: Add keystoneauth library and testing infrastructure
openstackgerritMerged openstack-infra/project-config: Use zaqar's devstack plugin
*** dims has quit IRC11:18
*** dhritishikhar has joined #openstack-infra11:18
openstackgerritMerged openstack-infra/project-config: Let kolla-release create branches
*** fhubik is now known as fhubik_afk11:19
*** maishsk has quit IRC11:20
*** maishsk has joined #openstack-infra11:21
openstackgerritMerged openstack-infra/project-config: add project fuel-plugin-glance-nfs
*** doug-fish has joined #openstack-infra11:22
*** dhritishikhar_ has joined #openstack-infra11:23
AJaegersdague, please put in your review queue as well...11:25
*** dhritishikhar has quit IRC11:26
*** ddieterly has joined #openstack-infra11:28
*** erikmwilson has joined #openstack-infra11:29
*** jamespage_ has joined #openstack-infra11:30
sergmelikyanHi folks, we are trying to release stackforge/yaql following new library release process but we don't have rights to create stable branches :(11:30
sergmelikyan - this patch corrects ACLs for stackforge/yaql11:30
*** e0ne is now known as e0ne_11:30
*** sputnik13 has quit IRC11:30
sergmelikyanPlease, take a look11:30
*** HeOS has quit IRC11:30
*** dhritishikhar has joined #openstack-infra11:31
*** dhritishikhar_ has quit IRC11:31
*** sputnik13 has joined #openstack-infra11:31
*** HeOS has joined #openstack-infra11:31
*** sputnik13 has quit IRC11:32
*** ddieterly has quit IRC11:33
*** coolsvap is now known as coolsvap|afk11:33
*** jamespage_ has quit IRC11:33
*** erikmwilson has quit IRC11:34
kashyapI find it strange - 'check-grenade-dsvm' job seems to succeed on _some_ CI machines, but not others (failing to invoke a certain `` command.11:35
kashyapE.g -
kashyapLooking at logstash, I see just about 11 hits in the last 48 hours.11:35
*** HeOS has quit IRC11:36
*** dizquierdo has joined #openstack-infra11:37
openstackgerritMerged openstack-infra/irc-meetings: Merge State Management and Oslo Meetings
openstackgerritMerged openstack-infra/irc-meetings: Add OpenStackClient Team Meeting
openstackgerritMerged openstack-infra/irc-meetings: Add Cue team meeting
openstackgerritMerged openstack-infra/irc-meetings: Add Chef Cookbook meeting.
openstackgerritMerged openstack-infra/irc-meetings: Add magnetoDB Team Meeting
openstackgerritMerged openstack-infra/irc-meetings: Add DefCore Progress Update Meeting.
openstackgerritMerged openstack-infra/irc-meetings: Add Community Office hours.
openstackgerritMerged openstack-infra/irc-meetings: Alternate Sahara meeting
openstackgerritMerged openstack-infra/irc-meetings: Update Neutron VPNaaS Meeting
openstackgerritMerged openstack-infra/irc-meetings: Glace meeting no longer alternates
openstackgerritMerged openstack-infra/irc-meetings: Move Puppet Meeting
openstackgerritMerged openstack-infra/irc-meetings: Place novanet to Neutron migration meeting on hold
openstackgerritMerged openstack-infra/irc-meetings: Update Telco WG Meeting
*** cdent has joined #openstack-infra11:41
openstackgerritMerged openstack-infra/irc-meetings: Rename OpenStack Security Group (OSSG) Meeting
openstackgerritMerged openstack-infra/irc-meetings: Update nova API Meeting Chair
openstackgerritMerged openstack-infra/irc-meetings: Update the Meeting chairs.
*** nmagnezi has quit IRC11:42
*** blahdeblah has left #openstack-infra11:44
*** marcusvrn1 has joined #openstack-infra11:45
*** mmedvede has quit IRC11:46
sdaguekashyap: is this an existing Elastic Recheck tracked bug?11:46
kashyapsdague: I don't find one,11:46
kashyapI'm trying to add a signature.11:46
sdaguewhat query are you using11:46
kashyapsdague: For logstash? If so -11:47
sdagueyes for logstash11:47
kashyapsdague: There we go --
kashyapsdague: Incidentally,  I was reading your thread on -dev list11:47
*** marcusvrn has quit IRC11:47
kashyapabout "trimming down Tempest smoke tag"11:48
kashyapwhen I noticed the failure in question is also tagged w/ 'smoke'11:48
sdagueoh, so we removed the generic SSHTimeout signature because it's really not useful enough to track anything11:49
kashyapI see, I recall seeing that signature11:49
kashyapIn this specific context,11:49
sdagueyou also need to remove the identifying ip address11:49
kashyapInvocationError: '/bin/bash tools/ (?!.*\\[.*\\bslow\\b.*\\])((smoke)|(^tempest\\.scenario)) --concurrency=4'11:49
sdaguekashyap: right, that just means a test failed11:49
*** _nadya_ has joined #openstack-infra11:50
*** prad is now known as prad-afk11:50
kashyapI think I can use regex in logstash queries?11:50
*** baoli has joined #openstack-infra11:50
sdaguejust segment it into 2 message matches11:51
sdaguethat's the break down by job view11:51
sdagueonly 4 grenade jobs hit it11:51
sdaguesorry 511:52
kashyapAh, I see11:52
openstackgerritMerged openstack-infra/project-config: Install kpartx for DIB tests
kashyapsdague: Any action I can take to fix this? Because, this doesn't seem deterministic11:52
sdaguedig into one of the failures and try to figure out why the networking did not come up for that guest11:53
*** baoli has quit IRC11:53
kashyapThanks, I'll check.11:53
*** zul has joined #openstack-infra11:53
*** baoli has joined #openstack-infra11:54
sdaguethese are never fun to debug, it requires a lot of tracing through the logs to figure out what was all going on when these failed11:54
openstackgerritMerged openstack-infra/project-config: Make oslo.messaging py3 job voting
*** mmedvede has joined #openstack-infra11:54
openstackgerritMerged openstack-infra/project-config: Create networking-zvm project on StackForge
openstackgerritMerged openstack-infra/project-config: fix ec2-api rally job
openstackgerritMerged openstack-infra/project-config: Enable gate jobs for remaining StackTach libraries
*** HeOS has joined #openstack-infra11:55
*** woodster_ has joined #openstack-infra11:56
*** fhubik_afk is now known as fhubik11:56
kashyapsdague: Yeah, can imagine. I was wondering if I can do all of it via some CLI, as browsing through plenty of logs in a browser just slows the workflow11:56
*** dims has joined #openstack-infra11:56
* kashyap fires up gertty to see if can find anything there11:57
sdagueyou'd have to download all the logs, there are no tools for that at this point11:57
openstackgerritMerged openstack-infra/project-config: project-config: Merge lint jobs (2/2)
kashyapsdague: This one from James Blair and co --
sdagueright, but I'm confused why you think that will help here11:59
*** scheuran has quit IRC12:04
*** dprince has joined #openstack-infra12:04
*** bauzas is now known as bauzas_notme12:05
openstackgerritJulia Kreger proposed openstack-infra/shade: Enhance error message in update_patch
*** bauzas_notme is now known as bauzas12:06
openstackgerritJulia Kreger proposed openstack-infra/shade: Update recent Ironic exceptions
openstackgerritJulia Kreger proposed openstack-infra/shade: Convert node_set_provision_state to task
*** openstackgerrit has quit IRC12:07
*** otter768 has joined #openstack-infra12:07
*** openstackgerrit has joined #openstack-infra12:07
*** mrmartin has quit IRC12:08
*** hasharLunch is now known as hashar12:08
*** _nadya_ has quit IRC12:10
*** otter768 has quit IRC12:12
openstackgerritMerged openstack-infra/project-config: Fix mistyped requrieChangeId in Gerrit ACLs
*** maishsk has quit IRC12:16
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
*** maishsk_ has joined #openstack-infra12:17
*** dhritishikhar has quit IRC12:17
*** dhritishikhar has joined #openstack-infra12:17
*** samueldmq has joined #openstack-infra12:20
*** bswartz has quit IRC12:20
*** abregman_ has joined #openstack-infra12:21
*** rfolco has joined #openstack-infra12:22
*** kgiusti has joined #openstack-infra12:23
*** abregman_ has quit IRC12:23
*** abregman_ has joined #openstack-infra12:24
*** abregman_ has quit IRC12:24
*** abregman_ has joined #openstack-infra12:24
*** ivar-lazzaro has quit IRC12:25
*** e0ne_ is now known as e0ne12:26
*** ivar-lazzaro has joined #openstack-infra12:28
*** ddieterly has joined #openstack-infra12:29
*** abregman_ has quit IRC12:29
*** dkranz has joined #openstack-infra12:32
*** ddieterly has quit IRC12:33
*** gordc has joined #openstack-infra12:33
*** erikmwilson has joined #openstack-infra12:34
*** erikmwilson has quit IRC12:35
*** erikmwilson_ is now known as erikmwilson12:35
*** jtriley has joined #openstack-infra12:35
openstackgerritMerged openstack-infra/devstack-gate: Explicitly say when job times out
*** claudiub has joined #openstack-infra12:36
*** kgiusti has left #openstack-infra12:39
*** jtriley has quit IRC12:40
*** EmilienM is now known as EmilienM|afk12:41
fungiwell, you _can_ override the browser callout for gertty's url handling to run an arbitrary program (i have it fire up elinks, but you could make it use wget to mirror the subtree from the url to a local directory or something pretty easily i'm betting)12:41
*** lifeless has quit IRC12:43
*** chlong has joined #openstack-infra12:44
*** MaxV has quit IRC12:44
*** fhubik has quit IRC12:44
*** kgiusti has joined #openstack-infra12:45
kashyapsdague: Was afk for lunch, no I didn't mean it (gertty) has specific options to do that (logs, etc). Was only trying to see if it has anything related to logs.12:45
AJaegerfungi, is available now - and zuul, openstackid, storyboard are missing content. What's the best way to push content there?12:46
openstackgerritMerged openstack-infra/project-config: fix git_timed retry logic
kashyapfungi: Yeah, that sounds interesting, maybe I'll file a bug or something in storyboard for it :)12:46
sdaguekashyap: or just build the tool you want as a cli12:47
fungiAJaeger: in theory, assuming their publish jobs are working, i'll just approve some trivial changes to those and they should pop right in12:48
AJaegerfungi, let's test the theory when you're not in fire fighting ;)12:49
AJaegerfungi, speaking about approving patches - the translation "compression" works fine for manuals, I think we can take the next step, a review is appreciated:
sdaguefungi: so will get picked up in the image builds today?12:50
fungikashyap: i invoke it like `BROWSER=w3m gertty` but you could substitute w3m with the name of whatever script does the downloading you want12:50
*** MaxV has joined #openstack-infra12:50
fungisdague: in theory, yes12:50
kashyapsdague: Yes, if only I can priortize like a ninja12:50
*** mpaolino has joined #openstack-infra12:51
kashyapfungi: Oh, cool. I already use `elinks` for mutt.12:51
sdaguekashyap: so, honestly, when tracing bugs like this you are going to be jumping back and forth between probably 4 - 6 log files. So I find tabs in browsers works fairly well12:51
kashyapfungi: Any reason you prefer w3m over elinks?12:51
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
kashyapsdague: Yep, I see what you're saying - it's not that I don't appreciate the ability to easily switch back n forth in the browser :-)12:52
maishsk_fungi: can you guys help me find out why I am not receiving all the emails from openstack-dev?12:52
*** maishsk_ is now known as maishsk12:53
fungikashyap: it was an example. actually playing around right now with lynx vs w3m vs links vs elinks, lynx seems to be the best and handling our job logs and making them readable on an 80x24 character terminal12:54
*** amitgandhinz has joined #openstack-infra12:54
*** dhritishikhar has quit IRC12:54
*** dhritishikhar has joined #openstack-infra12:54
fungimaishsk: do you have an example of a message you didn't receive?12:54
fungimaishsk: or is it all of them?12:54
*** e0ne is now known as e0ne_12:54
kashyapfungi: Oh, really, thanks for the test w/ lynx12:55
maishskAnything in this thread
maishskabsolutely not a single mail from this thread12:55
*** ibiris is now known as ibiris_away12:55
fungikashyap: also, minor bug (i'll probably get around to submitting a patch soon if nobody else beats me to it), after exiting the browser you need to hit ctrl-l to refresh the gertty display12:56
*** asahlin_afk is now known as asahlin12:56
kashyapfungi: From time to time the sqlite databse Gertty uses goes corrupt or something? If I start gertty after a few weeks,12:57
kashyapIt almost always errors out trying to sync. I haven't dug into it deeply though12:57
fungikashyap: it doesn't handle git repos being renamed in gerrit. my post-maintenance notices have included example sqlite commands for updating the gertty database12:58
fungikashyap: generally that only affects it if you've subscribed to a repo that then gets renamed12:58
kashyapfungi: Oh, notices to the -dev list?12:59
fungior possibly managed to review a change in one of those repos even if not subscribed12:59
fungikashyap: yeah, i can find you an example after i'm done looking into listserv logs for maishsk12:59
*** shardy_ has joined #openstack-infra12:59
*** yfried has quit IRC12:59
kashyapfungi: Sure, no rush. Thanks for looking.13:00
*** sandywalsh has joined #openstack-infra13:00
*** shardy has quit IRC13:01
*** ir2ivps has quit IRC13:01
fungimaishsk: is it possible you've set your openstack-dev subscription to not include messages with the [all] (cross-project discussions) tag?13:01
*** Alexandra_ has joined #openstack-infra13:02
maishskfungi: How would I check that?13:02
fungimaishsk: log in at (do a password reminder first if you need)13:03
*** annegentle has joined #openstack-infra13:03
maishskfungi: checking...13:04
fungimaishsk: then for the "Which topic categories would you like to subscribe to?" option, make sure "Cross-project coordination" is included13:04
fungithat subscribes you to messages with [all] in the subject line13:04
maishskah... Crap - you are right!!13:04
maishskfungi: Thank you.13:05
fungimaishsk: any time!13:05
*** shardy_ has quit IRC13:05
*** julim has joined #openstack-infra13:05
*** spzala has quit IRC13:05
jistrhi folks, could someone from infra-core please take a look at this change when you have some time? it has +1 from both PTLs
*** shardy has joined #openstack-infra13:05
*** e0ne_ is now known as e0ne13:06
maishskfungi: but wouldn't selecting 'yes' on Do you want to receive messages that do not match any topic filter?  cover that?13:06
*** julim has quit IRC13:06
*** dhritishikhar_ has joined #openstack-infra13:06
*** bknudson has joined #openstack-infra13:06
*** radez_g0n3 is now known as radez13:08
fungimaishsk: no, that's for messages which aren't covered by any of the matching rules13:08
maishskfungi: OK - that explains it!13:08
*** ZZelle has quit IRC13:08
*** julim has joined #openstack-infra13:08
fungimaishsk: so for example if i send a message to the list with no [tag] in the subject line at all, or an unrecognized tag like [xyzzy]13:08
*** ZZelle has joined #openstack-infra13:09
fungikashyap: see the notes/example for renaming congress in the gertty db at the end of
*** ibiris_away is now known as ibiris13:09
* kashyap clicks13:09
fungikashyap: also i think if you look in gertty's error log you should see what database keys it's having trouble with13:10
kashyapfungi: Yeah, I was indeed looking at a database error13:10
*** dhritishikhar has quit IRC13:10
kashyapfungi: I'm afraid, guess now I'm too late to catch the exact error as I deleted the db and started a resync again :(13:11
*** ociuhandu has joined #openstack-infra13:12
*** bhunter71 has joined #openstack-infra13:13
*** deepakcs has quit IRC13:14
*** julim has quit IRC13:18
*** abregman_ has joined #openstack-infra13:18
*** abregman has quit IRC13:20
*** tiswanso has joined #openstack-infra13:22
*** bhunter71 has quit IRC13:22
*** mattfarina has joined #openstack-infra13:23
kashyapfungi: And, btw - thanks to the infra team for Gertty - it really made using Gerrit palatable in my workflow :)13:23
*** mattfarina has quit IRC13:24
fungikashyap: i can't take responsibility for that. it's a personal project of jeblair's (to whom i too am thankful for this awesome piece of software)13:24
*** amitgandhinz has quit IRC13:25
kashyapfungi: Yep, that's what I meant, thanks to jeblair too!13:25
fungiit's technically not an infra project, just happens to be written by our illustrious ptl13:25
fungiall praise be unto him13:25
*** peristeri has joined #openstack-infra13:25
*** vhoward has joined #openstack-infra13:26
*** abregman_ has quit IRC13:29
kashyapfungi: Any plans to make it an "official" (maybe it'll help test/catch more bugs) infra project13:30
fungikashyap: i'm not opposed to the idea, but it's up to jeblair since it's his project (and i think he enjoys not being encumbered by our typical bureaucracy when he's hacking on something for fun like that)13:31
kashyapfungi: Sure. And, I totally understand his hesitation for fear of beaurocracy.13:32
kashyaps/fear/burden? :-)13:32
fungiclarkb: when you're around, what's a good way to troubleshoot boot issues with dib nodepool images in hpcloud? i think we're dib'ing unbootable devstack-precise images (they fail nodepool's ssh test and get deleted immediately)13:32
*** ir2ivps has joined #openstack-infra13:33
fungiclarkb: does the novaclient console support work for cases like that?13:33
fungiif so, i'll try to dig into it later today13:34
*** jtriley has joined #openstack-infra13:34
*** EmilienM|afk is now known as EmilienM13:37
*** dustins has joined #openstack-infra13:37
*** e0ne is now known as e0ne_13:38
*** lifeless has joined #openstack-infra13:38
claudiubfungi: hi, I would require your assistance. I'm having trouble pushing a new version of networking-hyperv to gerrit13:39
*** e0ne_ is now known as e0ne13:39
*** ddieterly has joined #openstack-infra13:39
*** MaxV has quit IRC13:39
fungiclaudiub: you mean pushing a networking-hyperv change to review?13:39
fungiclaudiub: what's the error?13:40
claudiubfungi: no, pushing a tag13:40
claudiubfungi: for a new release, 1.0.313:40
fungioh! yes. following hopefully?13:40
claudiububuntu@ubuntu:/opt/stack/hyperv/networking-hyperv$ git push gerrit 1.0.313:40
claudiub ! [remote rejected] 1.0.3 -> 1.0.3 (prohibited by Gerrit)13:40
fungiyou used git tag -s right (so that it's signed)?13:41
claudiubdid a checkout master, git pull, git tag -s 1.0.313:41
*** esker has quit IRC13:41
claudiuband i did a git tag --verify13:41
claudiububuntu@ubuntu:/opt/stack/hyperv/networking-hyperv$ git tag --verify 1.0.3 object e625ea751f6eadd65c88d1a0eb2eb25eeea3d651 type commit tag 1.0.3 tagger Claudiu Belu <> 1430314358 -070013:41
*** pabelanger has joined #openstack-infra13:42
fungithen chances are either the acl doesn't have a section to permit tagging, or you're not a member of the group currently authorized by that acl. checking now...13:42
claudiubnetworking-hyperv Release 1.0.3 gpg: Signature made Wed 29 Apr 2015 06:32:38 AM PDT using RSA key ID 8450D996 gpg: Good signature from "Claudiu Belu <>"13:42
fungithe acl itself looks fine13:43
claudiubthen I wonder what could be wrong..13:43
fungiclaudiub: get the current member of,members to add you to that group13:44
mescanefhello, please add me ( to the gerrit groups of a project regarding the :-)13:44
*** abregman_ has joined #openstack-infra13:44
mescaneffungi: ^^ pls :-)13:44
claudiubfungi: ah, i see... I talk to alexpilotti then. thanks. :D13:44
fungimescanef: i think i did already--have you checked?13:44
fungisaw your request in the channel history when i woke up and did it then13:45
mescaneffungi: oh yeah, thanks :-)13:45
fungino problem--have fun!13:45
*** Alexandra_ has quit IRC13:46
*** pabelanger has quit IRC13:46
*** flaviosr has joined #openstack-infra13:47
claudiubfungi: added to the list you've linked, pushing the new version worked, thanks! :D13:47
*** prad has joined #openstack-infra13:48
fungiclaudiub: you're welcome!13:49
*** mriedem_away has quit IRC13:51
*** marcusvrn1 has quit IRC13:54
*** mtreinish has joined #openstack-infra13:54
*** hdd has quit IRC13:54
*** marcusvrn has joined #openstack-infra13:55
*** asselin has joined #openstack-infra13:56
*** liam_ has joined #openstack-infra13:58
sdaguefungi: did we lose gerrit bot?13:58
*** BobH1 has joined #openstack-infra13:58
fungiwe may have lost gerrit13:58
*** liam_ is now known as Guest3894913:58
fungiyeah, probably have a dead stream-events queue for the last 30 minutes judging from the most recent change age in the check pipeline13:59
fungiconfirming and collecting some more debug details now before restarting gerrit _again_13:59
*** bswartz has joined #openstack-infra14:00
*** dustins_ has joined #openstack-infra14:00
*** yfried has joined #openstack-infra14:00
fungimost recent event zuul saw was 13:29:2414:01
*** BobH has quit IRC14:01
*** BobH1 has quit IRC14:01
*** zz_jgrimm is now known as jgrimm14:01
*** sslypushenko has quit IRC14:02
*** BobH has joined #openstack-infra14:02
*** sslypushenko has joined #openstack-infra14:02
*** dalgaaf has joined #openstack-infra14:02
*** mriedem has joined #openstack-infra14:03
*** dustins has quit IRC14:03
*** dkranz has quit IRC14:04
*** maishsk has quit IRC14:04
*** marun has joined #openstack-infra14:04
fungi#status notice gerrit has been restarted to clear a stuck events queue. any change events between 13:29-14:05 utc should be rechecked or have their approval votes reapplied to trigger jobs14:04
openstackstatusfungi: sending notice14:04
*** alexpilotti has joined #openstack-infra14:05
*** mtreinish has quit IRC14:05
-openstackstatus- NOTICE: gerrit has been restarted to clear a stuck events queue. any change events between 13:29-14:05 utc should be rechecked or have their approval votes reapplied to trigger jobs14:05
fungiso... in GOOD news, i have corresponding jstack dumps, connection list and queue list (including jeblair's debug details showing corresponding gerrit accounts for each task now)14:06
fungii'll see if i can make heads or tails of any of it14:06
openstackstatusfungi: finished sending notice14:06
openstackgerritRussell Bryant proposed openstack/requirements: global-requirements: Add ovs
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Decrease PO files for python projects
fungilooks like gerritbot is happy again14:08
*** otter768 has joined #openstack-infra14:08
*** btully has joined #openstack-infra14:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Decrease PO files for imports (horizon+friends)
fungi15 hung stream events tasks for netapp-ci14:09
*** sigmavirus24_awa is now known as sigmavirus2414:09
fungi8 for neutronryu14:10
*** nfedotov has joined #openstack-infra14:10
fungithe others are 5 or fewer14:10
*** mase_x200 has joined #openstack-infra14:11
*** pabelanger has joined #openstack-infra14:11
fungi17 connections from netapp-ci, 12 from neutronryu, 11 from our ci system (zuul and all its mergers)14:12
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Allow Magnum to conform to global requirements process
*** otter768 has quit IRC14:12
*** annegentle has quit IRC14:12
*** amitgandhinz has joined #openstack-infra14:14
*** yamahata has joined #openstack-infra14:14
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Allow Magnum to conform to global requirements process
sdagueso, that seems excessive14:16
sdagueany idea why so many connections?14:16
fungiwell, looking at the connections list, netapp-ci connects from at least half a dozen different addresses in widely varying netblocks14:16
fungiso my guess is that one account is shared by a lot of different ci systems14:16
*** abregman_ has quit IRC14:17
*** annegentle has joined #openstack-infra14:17
sdagueok, are they really reporting that many different ways? maybe thingee or someone could help illuminate there14:17
fungithere's not necessarily any correlation between the accounts connecting and listening to the event stream and the hung state it gets into14:17
sdaguethat's true14:17
fricklerfungi: where does the statusbot get its channel list from? I've seen the gerrit notice on some openstack channels, but not on all of them14:17
*** isviridov is now known as isviridov_away14:18
fungifrickler: inconveniently,
*** flaviosr has quit IRC14:19
fricklerfungi: o.k., will do a patch for that, thx14:19
*** erikmwil_ has joined #openstack-infra14:20
*** erikmwilson is now known as Guest8463614:20
*** erikmwil_ is now known as erikmwilson14:20
thingeesdague: they have about 6 drivers. Not sure how they have things configured, but maybe those have their own CI/stream? Also they may have internal ones.14:20
*** rossella_s has quit IRC14:20
*** erikmwilson_ has joined #openstack-infra14:21
*** rossella_s has joined #openstack-infra14:21
thingeefungi: ^14:22
clarkbfungi: usually I would debug by booting a test node then checking the `nova console-log` output for it, you can get an interactive console too if that helps but unsure if we can actually login unless it has dropped to root shell14:22
thingeesdague: I'm still getting ready to head into the office, but I can talk to folks listed here:
pabelangerSo, wanted to officially offer my support to -infra now that I'm full time employed.  For those unware, I took a position with Red Hat last week with the mandate to help support upstream infrastructure when ever possible.  Plan to be at the summit in Vancouver also, so I can finally put a face to handles.14:22
clarkbfungi: then if that doesn't shed any light you can grab the image used and boot locally against kvm, I used greghaynes, to make that pretty easy14:22
pabelangereither way, just tossing that out there.14:23
*** jtriley has quit IRC14:23
AJaegerpabelanger, great!14:23
sdagueclarkb: hey, you are awake. Want to quick review - - which should make it easier to share ER links14:24
pabelangerBut, did have a thought / question last night.  I wanted to see what people thought about maybe adding grafana into -infra, so people have the ability to generate dashboards based on the nodepool / zuul / jenkins. Tried searching around for info, but didn't see too much chatter about it14:24
*** ajmiller__ has quit IRC14:25
pabelangerIt wouldn't replace statsd / granite but sit a top of it14:25
anteayapabelanger: welcome and yay!!14:25
AJaegerclarkb, fungi : Is there an easy way to get the post jobs for documentation run with higher priority?14:25
clarkband now that I have caught up I see gerrit was restarted14:25
pabelangerI used it a while ago for some call center stats, and found it was super flexible for rendering data14:26
clarkbAJaeger: we can change the post job priority back to being on the same level as check14:26
pabelangerAJaeger, anteaya: Thanks14:26
*** asrangne has joined #openstack-infra14:26
AJaegerWE currently have to wait 3 hours for publishing content - and that blocks us, since we need to prepare for Kilo and test some things14:26
sdagueAJaeger: I don't understand why it's waiting14:26
anteayapabelanger: as for graphana I think someone may have mentioned it recently but I have forgotten who and the context14:26
sdaguewe have the nodes14:26
openstackgerritMerged openstack-infra/elastic-recheck: allow hash urls to work before page loads
clarkbsdague: beacuse check wins, so we have to make check completely satisfied before any jobs go to post14:27
sdagueclarkb: sure14:27
sdaguebut we've got an extra 400 nodes14:27
clarkbsdague: I think its a case of where we don't predict enough extra nodes to build so as new patchsets come in they "steal" from the nodes allocated to post14:27
anteayapabelanger: my best suggestion is stick around in channel and learn the answers to common questions so you too can hand them out14:27
*** esker has joined #openstack-infra14:27
clarkbwe build what we need then what we need gets larger14:27
*** zz_dimtruck is now known as dimtruck14:27
sdagueclarkb: so the fix there is just to up the extra nodes?14:28
pabelangeranteaya, Ya, plan is for me to do local test and see what I can consume from statsd upstream.  Not sure any authenticate is required, but could be wrong14:28
*** aswadr has quit IRC14:28
sdaguepost competing with check puts us into other weird hangs14:28
*** tiswanso has quit IRC14:28
pabelangeranteaya, Agreed, I know some people I talked to, trying to support RHT efforts upstream were interested in the ability to generate dashboard to easily view data14:28
anteayapabelanger: okay, for any new features I defer to jeblair who is currently traveling i believe14:28
*** wenlock has joined #openstack-infra14:28
clarkbI think the real fix is to change post's priority then the numbers we calculate will be more accurate for post14:28
pabelanger /me nods14:29
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Give post queue normal priority
AJaegerclarkb, sdague: Thanks for the info, here's a patch ^14:29
anteayapabelanger: if you want to know what is important to infra our meeting logs are helpful:
anteayapabelanger: any priority effort is something that we have identified as important14:29
*** tiswanso has joined #openstack-infra14:29
sdagueclarkb: that seems wrong to me, because we do specifically want post to be handled after check14:29
sdagueand check win when there is contention14:30
anteayapabelanger: my suggestion would be help with any priority effort, and the propose what you want to the queue14:30
clarkbsdague: but nodepool has no concept of queue priority14:30
AJaegerclarkb, for the other suggestion: I don't know how to change that14:30
jaypipespcrews: looking at Very cool. I've been wanting to look at for a while now. rannsaka is a good way to do that :) BTW, are you familiar with Rally's spawning internals? /me wonders if might be a good replacement spawn/collect engine for it.14:30
*** abregman_ has joined #openstack-infra14:30
anteayapabelanger: then when a priority effort is finished, there are space for others which are drawn from the queue14:30
clarkbsdague: so while we could say always build 5 extra nodes (or some arbitrary number) that isn't going to directly address the issue, it may help make it better but only if we don't exceed that number with inoming check patchsets14:30
sdagueclarkb: so if we're actually slamming that hard, I think that's fine14:31
*** julim has joined #openstack-infra14:31
sdaguebut looking at the ready pool count, I don't understand how things could be hanging out for 3 hours to get docs nodes14:31
*** wenlock_ has joined #openstack-infra14:31
sdaguewe've *always* had available nodes14:31
*** dkranz has joined #openstack-infra14:32
fungii'm guessing we have too many of the "wrong" node types in a ready state right now because of an earlier poor prediction14:32
fungichecking now14:32
clarkbya I am looking too14:32
clarkba good chunk are tripleo nodes14:32
fungipabelanger: also thanks and welcome!14:32
*** tiswanso has quit IRC14:32
clarkbthe others look like potentially "leaked" nodes from hpcloud14:33
clarkboh there are some from rax too so likely not a cloud specific issue14:33
clarkb is alive and ready14:34
clarkbhas been for almost 2 days14:34
sdagueAJaeger: it might also be optimizing to not have 4 distinct post jobs for the manuals, so that you need 4 nodes per change14:34
fungiwe have 66 bare-trusty nodes in a ready state14:34
*** Krinkle|detached is now known as Krinkle14:34
fungidefinitely abnormal14:35
AJaegersdague, let me see what I can do there...14:35
clarkb is another case of the above14:35
clarkbfungi: so they are in jenkins just fine, I wonder if they do not have gearman connection?14:35
fungiif a lot of those aren't picking up jobs for some reason, then that would explain why changes are sitting waiting for a longer than usual time (because nodepool doesn't realize it needs to make more)14:35
clarkbfungi: I am going to try taking 2328080 offline in jenkins and bring it back to see if it gets used14:35
fungiclarkb: that would be my guess14:36
sdaguefungi: so... could the gerrit event stream hangs contribute to the orphans?14:36
fungisdague: i don't believe so, no14:36
fungiclarkb: though it's just one gearman connection to zuul from each jenkins master right?14:36
*** tiswanso_ has joined #openstack-infra14:37
clarkbfungi: no its a connection for each slave14:37
fungiahh, okay14:37
AJaegersdague, publishlang and publishdocs are both "long" running jobs. Merging them is possible but then it's sequential...14:37
AJaeger(long means around 20-30 mins)14:37
sdagueAJaeger: what kind of node do they run on ?14:37
sdaguethose seem to be the ones not getting allocated nodes14:38
clarkbfungi: I have found the gearman worker thread for 2328080 and it seems to be alive and working (running epoll)14:38
sdaguefungi / clarkb: what's node build time now adays?14:38
*** wznoinsk has quit IRC14:39
AJaegersdague, should be "bare-precise || bare-trusty"14:39
fungisdague: depends a lot on local conditions within each provider/region14:39
sdagueAJaeger: under load you are also going to pay node build time cost, so it's only cheaper to make them parallel if the whole system is idel14:39
AJaegersdague, so, worth to create a job that runs for an hour?14:39
fungiAJaeger: i believe we're starved on bare-trusty jobs right now. nodepool thinks there are a lot ready but most of those aren't actually picking up tasks14:40
clarkbbut so far everything says they should pick up tasks just fine14:40
clarkbI don't know that I will be able to map 2328080 to a specific tcp connection but I wonder if the remote end disagrees that there is something to epoll on14:41
AJaegerthis situation is there for some time already, this is not something suddenly changing today. It's just hurting us more today ;)14:41
sdagueAJaeger: it might be, also your other jobs that run in about 50s that run in post would be good to consolidate14:41
*** dims has quit IRC14:42
AJaegersdague, those are "proposal" nodes14:42
*** dims has joined #openstack-infra14:42
AJaeger(please double check that I'm right with the node types)14:42
clarkbnetstat connection counts look sane at least14:43
*** wznoinsk has joined #openstack-infra14:43
AJaegerthe upstream-translation job is used by several repos, the propose one is unique to openstack-manuals.14:43
AJaegerLet me check the descriptiosn...14:43
*** jamesmcarthur has joined #openstack-infra14:43
clarkbthough jenkins06 claims 89 established connectinos and zuul just 87, but thats not a big enough discepancy to account for these nodes14:44
*** mtreinish has joined #openstack-infra14:45
clarkbfungi: I have confirmed via jenkins logs that 2328080 did do a gearman reconnect as part of my disable enabling it14:45
AJaegerSo, consolidation of these jobs is possible - fungi, clarkb, sdague: I wait for your advise whether it's worth it14:46
sdagueAJaeger: well, I think it will on average be faster14:46
*** mriedem has quit IRC14:46
clarkbfungi: so far everything looks good other than this node not running jobs14:46
fungialso those ready bare-precise nodes are pretty evenly spread across all jenkins masters (except the one which is disabled) so it seems to be not a localized problem with one jenkins master14:47
clarkb(we should however consider logging the connection details in the gearman plugin since right now there is no way to figure that out from what I can tell)14:47
*** bhunter71 has joined #openstack-infra14:47
clarkbI know I have a connection but don't know which is which14:47
clarkbI want my next step to be tcpdump, maybe I can grep on the snarfed output and see if that shows me what the connection is14:48
fungithere's a possibility, yeah14:48
openstackgerrityolanda.robla proposed openstack-infra/system-config: Move server outside of openstack_project::git
*** alexpilotti has quit IRC14:49
*** marcusvrn_ has joined #openstack-infra14:49
*** tonytan4ever has joined #openstack-infra14:49
*** mtreinish has quit IRC14:49
fungiyou can just dump all gearman traffic for the master in question to a pcap file and then use tcpdump to read that file and pare it down to the right connection with an additional pcap filter expression after you've identified the right one by matching on some content in the packets14:50
fungithough you'll probably have to toggle the slave status again to trigger another registration14:50
*** katyafervent is now known as katyafervent_awa14:52
openstackgerritJiri Stransky proposed openstack-infra/project-config: Add tripleo-core to puppet-tripleo
*** BharatK_ has quit IRC14:52
*** mriedem has joined #openstack-infra14:52
*** claudiub has quit IRC14:52
clarkb`sudo tcpdump -i any -A port 4730 | grep 2328080` on jenkins06 has not found anything even after toggling that node's state14:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Merge some openstack-manuals post jobs
AJaegersdague, fungi, clarkb: here's what sdague proposed ^14:53
openstackgerritGary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Merge some openstack-manuals post jobs
fungiAJaeger: sdague: that probably doesn't make any difference at the moment. those run on the worker, which is static and doesn't deplete our worker pool14:54
sdagueAJaeger: ok, it's still probably worth letting clarkb and fungi figure out why we aren't making bare nodes14:54
*** tteggel_ has quit IRC14:54
sdaguefungi: ok, I didn't know about proposal node being static14:54
AJaegerfungi, ah - ok, I abandon....14:55
openstackgerritValeriy Ponomaryov proposed openstack/requirements: Add python-manilaclient to global requirements.
AJaegersdague, I didn't remember either...14:55
openstackgerritValeriy Ponomaryov proposed openstack/requirements: Bump ddt to min version 0.7.0
*** katyafervent_awa is now known as katyafervent14:55
fungialso it doesn't look like we had any trouble running those. they've all completed. it's stuff running on bare-trusty nodepool workers which are currently blocked14:55
*** emagana has joined #openstack-infra14:55
*** tteggel has joined #openstack-infra14:56
sdaguefungi: yeh, I just didn't realize they weren't sucking up nodes14:56
clarkboh right nodes don't register with any more specific info that their label iirc, so the plan to grep off of the contents in the gearman packets is probably a bad plan14:57
*** unicell has joined #openstack-infra14:57
*** mase_x200 has quit IRC14:57
*** unicell1 has quit IRC14:57
*** imcsk8|zZz is now known as imcsk814:57
fungiother than possibly timing of the registration datagram?14:58
AJaegerfungi, clarkb, sdague: Thanks for looking into this! I'm cycling home soon and will log back in later this evening14:58
*** lucap has quit IRC14:58
fungiclarkb: if you can narrow down the registration message to the timeframe where you toggled the status back to active14:58
*** cody-somerville has quit IRC14:58
*** jtriley has joined #openstack-infra14:58
fungichances are nodepool's not adding lots of bare-trusty workers right this moment14:59
openstackgerritGal Sagie proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow
*** patrickeast has joined #openstack-infra15:00
openstackgerritJens Rosenboom proposed openstack-infra/system-config: Add two missing channels to statusbot
clarkbya pulling up gearman protocol doc now to see what I should be looking for15:00
*** gsagie has joined #openstack-infra15:00
gsagieHello all, any review for will be appreciated15:00
openstackgerritBrant Knudson proposed openstack/requirements: Document discover used for python26
*** packet has joined #openstack-infra15:02
*** nelsnelson has joined #openstack-infra15:02
*** gary-smith has joined #openstack-infra15:04
*** BharatK_ has joined #openstack-infra15:04
*** gsagie has left #openstack-infra15:04
*** peristeri has quit IRC15:04
openstackgerritSirushti Murugesan proposed openstack-infra/project-config: Add grenade jobs to Heat
openstackgerritMerged openstack-infra/storyboard: Refactored event worker plugin structure into plugin directory.
openstackgerritMerged openstack-infra/storyboard: Every worker now gets its own session.
openstackgerritMerged openstack-infra/storyboard: Pre-resolve author record for events.
*** alexpilotti has joined #openstack-infra15:11
pabelangerwhat flavor of OS is -infra defaulting to now? I still see 12.04 in launch-node script, but also know 14.04 has been used15:13
ryanpetrello_It's pretty well decided that OpenStack will *not* be supporting Python 3.2, yes?15:13
*** claudiub has joined #openstack-infra15:13
ryanpetrello_considering removing the support from pecan because I doubt anybody is actually using it15:13
*** wznoinsk_ has joined #openstack-infra15:14
fungiryanpetrello_: yep. or 3.3 for that matter15:15
fungiryanpetrello_: our current py3k testing upstream is all with 3.415:15
ryanpetrello_okay, good to know15:15
ryanpetrello_thanks :)15:15
fungipabelanger: we're starting to put new things on trusty and slowly move services from precise to trusty as opportunity allows15:15
*** dprince has quit IRC15:16
pabelangerfungi, thanks15:16
*** miqui has joined #openstack-infra15:17
fungimost recent thing to move was gerrit15:17
*** rlandy has quit IRC15:17
*** wznoinsk__ has joined #openstack-infra15:18
fungiand people are contributing patches for, e.g., apache 2.4 configuration support so that more things will hopefully work on trusty when we get around to rebuilding a particular machine15:18
*** vlaza has left #openstack-infra15:20
*** heyongli has quit IRC15:21
pabelangerfungi, okay, sold. Going to trying building new server using trusty and see how it rolls15:22
*** heyongli has joined #openstack-infra15:22
*** wznoinsk has quit IRC15:23
*** eharney has quit IRC15:23
*** wznoinsk__ has quit IRC15:24
*** wznoinsk_ is now known as wznoinsk15:24
openstackgerritLouis Taylor proposed openstack-infra/project-config: Add functional test job for python-glanceclient
openstackgerritAlexander Tivelkov proposed openstack-infra/project-config: Added branch creation to yaql ACL
fungipabelanger: lots of people are running zuul on trusty, for example. we just haven't gotten around to rebuilding ours yet15:28
fungiso it's assumed it probably works, but i don't have first hand experience making ti work15:28
pabelangerfungi, cool.  Ya, last zuul stand-up was still precise.15:28
pabelangerSome how I suspect next one will be centos 7 :)15:28
fungithat could be interesting15:29
fungii hear the latest opensuse release is really nifty too15:29
pabelangerYa, plan to start working on zuul RPM shortly.  Get it up into zuul-packaging15:29
*** cody-somerville has joined #openstack-infra15:31
*** cody-somerville has quit IRC15:31
*** cody-somerville has joined #openstack-infra15:31
*** hdd has joined #openstack-infra15:31
*** dangers_away is now known as dangers15:32
fungipabelanger: awesome--have you seen the packaging discussion on the infra ml? StevenK and you should definitely collaborate on cross-distro ideas there15:32
*** peristeri has joined #openstack-infra15:33
*** Guest38949 has quit IRC15:33
*** liam_ has joined #openstack-infra15:33
*** liam_ has quit IRC15:33
pabelangerfungi, Yes, exciting times. Looking forward to it and eager to help15:34
*** sdake has joined #openstack-infra15:34
*** frobware_ has joined #openstack-infra15:37
*** mtanino has joined #openstack-infra15:38
*** bswartz has quit IRC15:39
*** sabeen has joined #openstack-infra15:40
clarkbfungi: I think I have narrows it down, port 55192 on jenkins06 appears to the source side of the gearman connection for 232808015:40
*** ildikov has quit IRC15:41
*** stevemar has joined #openstack-infra15:42
*** jamesmcarthur has quit IRC15:43
openstackgerritTerry Howe proposed openstack-infra/project-config: Add OpenStack Python SDK gate
*** julim has quit IRC15:43
clarkbfungi: I am running a tcpdump against that port only and not seeing the typical work request traffic that I would expect15:44
clarkbfungi: however the can do function registration appears to have succeeded properly15:44
*** mtreinish has joined #openstack-infra15:44
*** dizquierdo has quit IRC15:45
clarkbiirc gearman workers are supposed to be chatty about requesting work15:45
*** armax has joined #openstack-infra15:45
clarkbalso wireshark totally groks gearman which is awesome15:45
*** jcoufal has quit IRC15:45
clarkbyup reading the protocol doc the worker should be sending GRAB_JOB requests15:46
*** sdake_ has joined #openstack-infra15:47
openstackgerritTerry Howe proposed openstack-infra/project-config: Add OpenStack Python SDK gate
*** hashar has quit IRC15:48
stevemarbumping keystoneclient for requirements... would be awesome >.>15:49
*** dtantsur is now known as dtantsur|afk15:50
*** jamesmcarthur has joined #openstack-infra15:51
*** sdake has quit IRC15:51
clarkbI am beginning to suspect a gearman plugin bug, possibly we don't change our state on reconnection if we had entered PRE_SLEEP? there are probably a bunch of ways the state machine could break here15:52
*** eglynn has joined #openstack-infra15:52
*** rm_work is now known as rm_work|away15:53
*** annegentle has quit IRC15:53
*** asrangne has quit IRC15:53
claudiubfungi: quick question: if I pushed a new version / tag to gerrit (it appeared on the stackforge repo), shouldn't it be automatically pushed to pypi as well? Or does it take a while?15:54
*** dannywilson has joined #openstack-infra15:54
*** dannywilson has quit IRC15:54
*** bswartz has joined #openstack-infra15:54
clarkbclaudiub: what project was the tag made on?15:55
*** dannywilson has joined #openstack-infra15:55
*** Sukhdev has joined #openstack-infra15:55
claudiubclarkb: openstackci is added as an admin user on the networking-hyperv pypi project15:56
*** ssam2 has quit IRC15:56
clarkbclaudiub: networking-hyperv does not have the pypi publishing jobs so will not publish to pypi on tags15:56
claudiubclarkb: I see. So, that should have been added to the project-config commit, right?15:58
*** sergsh has quit IRC15:58
claudiubclarkb: I see, thanks for the info! :D I will create a commit to add those jobs.15:58
clarkbfungi: still no grab_job packets15:59
clarkbfungi: so I don't think this is merely my impatience, it appears to be "dead"15:59
fungithat's weird16:01
clarkb"or any packets at all"16:01
fungireceiving you mean?16:02
fungiat the jenkins end or the zuul end (or are you sniffing at both endpoints)?16:02
fungiwell, when you're done exploring available avenues, i can clear all the stale nodes which have been in a ready state for more than an hour16:02
clarkbfungi: I am only sniffing jenkins side16:02
clarkbfungi: because we should see jenkins send GRAB_JOB packets indicating it wants to run a job16:02
clarkbfungi: so I think the state machine in jenkins is borked16:02
openstackgerritTristan Cacqueray proposed openstack-infra/puppet-jenkins: Adds jenkins default file
*** dprince has joined #openstack-infra16:03
fungibut just for those nodes?16:03
clarkbfungi: ya, possibly because there is a race with some server response aginst the current state of the jenkins gearman state machines16:03
*** zz_ja is now known as zz_zz_ja16:03
*** abregman_ has quit IRC16:04
*** jamesmcarthur has quit IRC16:04
clarkbfungi: so most of the time it just works, but at other times it does not. My hunch is something like if jenkins slave has to reconnect to gearman when it is in a sleep state then it doesn't undo the sleep state and the server never sends a noop to the slave to wake it because after reconnecting it shouldn't be in a sleep state any longer16:04
clarkbreading the code we try to handle this, but I am not sure that we actually do16:05
*** jlanoux has quit IRC16:05
*** sarob has joined #openstack-infra16:06
*** hemnafk is now known as hemna16:06
openstackgerritEmilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from
*** ryanpetrello_ has quit IRC16:08
*** ildikov has joined #openstack-infra16:08
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for cinder bug 1450110
openstackbug 1450110 in tempest "boto tests randomly fail in resource_cleanup with "Failed to delete volume"" [Undecided,New]
*** ryanpetrello_ has joined #openstack-infra16:08
*** devvesa has quit IRC16:09
*** otter768 has joined #openstack-infra16:09
*** mriedem is now known as mriedem_lunch16:09
*** eharney has joined #openstack-infra16:09
*** akshai has joined #openstack-infra16:09
*** Ala has quit IRC16:09
clarkbpleia2: morning16:09
clarkbfungi: I am going to chew through more of this gearman-plugin state machine after breakfast, but need brain food first16:10
fungii am similarly lunching16:10
*** jistr has quit IRC16:11
gary-smithclarkb, fungi: if you need a diversion while you eat, could you review ?  :-)16:11
*** jamesmcarthur has joined #openstack-infra16:12
bswartzgary-smith: +116:13
bswartzthe sooner that change happens the sooner we can proceed with setting up tarballs and packaging16:13
gary-smithclarkb: thanks for the review!16:13
*** otter768 has quit IRC16:13
*** ryanpetrello_ has quit IRC16:14
*** zul has quit IRC16:15
*** whoops has joined #openstack-infra16:15
*** ryanpetrello_ has joined #openstack-infra16:15
*** mpaolino has quit IRC16:15
*** julim has joined #openstack-infra16:17
*** Swami has joined #openstack-infra16:18
*** unicell has quit IRC16:19
openstackgerritMerged openstack-infra/elastic-recheck: Add query for cinder bug 1450110
openstackbug 1450110 in Cinder "boto tests randomly fail in resource_cleanup with "Failed to delete volume"" [Undecided,New]
*** chlong has quit IRC16:19
*** ociuhandu has quit IRC16:20
*** alexpilotti has quit IRC16:20
*** pcaruana has quit IRC16:22
*** _nadya_ has joined #openstack-infra16:22
*** _nadya_ has quit IRC16:23
*** nfedotov has left #openstack-infra16:24
*** rmcall has joined #openstack-infra16:25
*** gyee has joined #openstack-infra16:25
zaroyolanda: parallel update is looking good.16:26
yolandai finally was able to setup my jenkins master and test live there16:26
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
zaroyolanda: a few more tests but i still may -1 for 'workers' param being in 'test' command16:28
openstackgerritGary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack
*** btully has quit IRC16:28
clarkbzaro please see scrollback I think we have a bug in gearman plugin where slaves never send a grab job request after getting into a bad state16:28
*** mpaolino has joined #openstack-infra16:28
yolandazaro, ok , initially i just wanted to do a clean rebase, but i'm fine taking ownership of the change and amending things16:28
yolandai can check with David as well16:29
clarkbzaro my current theory is that if we are sleeping then some event happens that we should stop sleeping but we dont16:29
openstackgerritTristan Cacqueray proposed openstack-infra/system-config: Use jenkins default file from puppet-jenkins
* zaro looing thru scrollbak16:30
openstackgerritVadim Rovachev proposed openstack-infra/project-config: Enable ceilometer functional tests
*** arxcruz has quit IRC16:31
*** ivar-lazzaro has quit IRC16:32
openstackgerritTristan Cacqueray proposed openstack-infra/puppet-jenkins: Adds jenkins default file
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
*** HeOS has quit IRC16:34
krtaylorpatrickeast, asselin, when you get a chance, please weigh in on
*** mattfarina has joined #openstack-infra16:34
*** mattfarina has quit IRC16:34
patrickeastkrtaylor: done16:35
clarkbzaro reading the worker implementation we send ourselves a noop but only in the registerfunctions() method but we seem to use setFunctions() more16:36
clarkbwe can problablu simplify that code so it is less confusing16:36
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: provide a more targetted location setting
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options
yolandazaro, amended prior to your -1 :)16:40
clarkbfungi did you catch sb from last night? the canonical thibg was apparebtly a demo?16:40
clarkbbut I think it supports the theory that its not a ton of stuff16:40
yolandaclarkb, fungi, i was starting with openstack_project::git usage, and i see that there is no upstream module to define git and haproxy there, and it involves some functionality. What do you think of some puppet-git-ha for example ? i see  a puppet-cgit, but nothing for ha16:41
clarkbjust very persistent16:41
*** _nadya_ has joined #openstack-infra16:42
clarkbyolanda its really just an haproxy config does that need a new module?16:42
yolandaclarkb, ideally i'd liked a puppet-git, with ::cgit and ::haproxy classes , everything together16:43
*** jtriley has quit IRC16:43
yolandacurrently that openstack_project::git installs packages16:43
yolandaadds config files16:43
*** achanda has joined #openstack-infra16:43
fungiclarkb: no, i missed that. a demo huh?16:44
yolandasets up haproxy16:44
yolandaso it's embedded functionality there, not just config16:44
*** mriedem_lunch is now known as mriedem16:44
*** dhritishikhar_ has quit IRC16:44
*** dhritishikhar_ has joined #openstack-infra16:45
yolandaclarkb, i was tempted to add a class to puppet-cgit, but the cgit name seems to be isolating that module just for backend16:45
fungiyolanda: clarkb: i'm not opposed to adding a similar haproxy load balancing class to our puppet-cgit module16:45
yolandafungi, that module is yours16:45
*** armax has quit IRC16:46
fungiwhat module is mine?16:46
yolandafungi, the naming was throwing me back , as cgit refers to that backend, but if you are ok with that, i'm happy to propose a change16:46
yolandahas your name there16:46
fungii'm not following. where is my name?16:47
*** asselin has quit IRC16:47
yolandaah, just saw that in some commit, i may be wrong16:47
fungii have a few commits in it, but am far from the most prominent contributor to the repo16:48
clarkbya if cgit wants an haproxy frontend manifest that makes sense16:48
*** armax has joined #openstack-infra16:48
yolandaok, happy to propose the change then16:48
fungianyway, i can see an argument for having a cgit::lb class or something16:48
fungiwhich happens to implement load balancing via haproxy as an opinionated implementation16:49
yolandafungi, but that won't be part of the manifest, but a class16:49
yolandayou can use or not16:49
fungihence a separate class16:49
fungientirely optional16:50
yolandayep, that's the idea16:50
*** Guest92304 has quit IRC16:50
*** unicell has joined #openstack-infra16:50
*** unicell1 has joined #openstack-infra16:51
*** unicell has quit IRC16:51
pleia2ok, confirmed translations tooling session will be in i18n rather than our track (& noted in etherpad)16:51
openstackgerritGary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack
fungithat's excellent news. implies they're taking the effort seriously16:53
*** annegentle has joined #openstack-infra16:54
fungiwhich i assumed, but it's great to have it confirmed like that16:54
*** marcusvrn has quit IRC16:55
*** marcusvrn_ is now known as marcusvrn16:55
zaroclarkb: do  slaves pick up new jobs after enable/re-enable gearman plugin?16:56
fungizaro: the effected slaves do not, no16:56
clarkbzaro I havent tried that, only offline/online the slave16:56
*** marcusvrn1 has joined #openstack-infra16:56
fungioh, of the plugin16:57
*** Shrews has quit IRC16:57
fungii misread, thought you were asking about toggling the slave status, yeah16:57
*** gilliard has quit IRC16:57
fungii wonder if briefly disabling and enabling the plugin is particularly disruptive16:57
*** gilliard has joined #openstack-infra16:58
fungibut also worth noting we're currently seeing this behavior across all of our jenkins masters16:58
fungiso it doesn't seem to be a particularly rare situation16:58
clarkbbut not all slaves16:58
zarowhen online/offline the fucntions are correctly registered? or do they not get registered at all with the slave?16:58
clarkbzaro they are registered16:58
*** derekh has quit IRC16:59
clarkbzaro but the slave never sends a grab job which makes me thibk they are sleeping16:59
*** sdake_ has quit IRC16:59
*** akshai has quit IRC16:59
fungidid you confirm that the registration message is arriving at zuul's end of the socket?16:59
*** btully has joined #openstack-infra16:59
*** sdake has joined #openstack-infra16:59
clarkbfungi zuul sends acks17:00
*** Shrews has joined #openstack-infra17:00
clarkbI did not tcpdump from that end though17:00
fungiahh, perfect. so definitely not a dead socket17:00
zaromight be miscommunication between workers and server?17:00
fungidoesn't sound like miscommunication unless it's at layer 717:00
*** patrickeast has quit IRC17:00
zaroi mean gearman server.  we only have 1 gearman server correct?17:01
*** ashleighfarnham has joined #openstack-infra17:01
fungior, well, above layer 4 at any rate (the rest of the osi layers are sort of irrelevant distinctions in my opinion)17:01
*** annegentle has quit IRC17:01
fungiwe only have one gearman server, the one running as a subprocess of the zuul scheduler17:01
clarkbzaro correct just the one zuul starts17:01
clarkbI think its a state issue on the plugin side17:01
clarkbI am failry sure we do not clear the sleep state properly when we offline online17:02
zaroodd, why only happens with certain slaves?17:02
*** bswartz has quit IRC17:02
clarkbzaro probably a race in the state transitions17:02
clarkbeg if you sleep, the reconnect before getting a noop17:02
*** sdake has quit IRC17:02
clarkbor if server sends noop but you dont process it quickly enough17:03
*** sdake has joined #openstack-infra17:03
*** ivar-lazzaro has joined #openstack-infra17:03
*** signed8bit has joined #openstack-infra17:03
*** annegentle has joined #openstack-infra17:03
zaroif that's the case wouldn't it occur randomly?  not happen to the same slave all the time?17:04
*** dkranz has quit IRC17:04
clarkbit is random17:04
clarkbbut once a slave gets in that state it cant get out17:04
*** mpavone has quit IRC17:04
fungiwell, there is _one_ way out... sweet, sweet death17:04
*** ashleighfarnham has quit IRC17:04
fungivia deletion17:04
clarkb(this is my theory at least especially after seeing that registerfunctions and setfunctions are different)17:05
*** ashleighfarnham has joined #openstack-infra17:05
clarkbfungi we should make sure they find manny calavera on their way to a better place17:05
*** bswartz has joined #openstack-infra17:06
*** unicell1 has quit IRC17:06
*** tonytan4ever has quit IRC17:06
*** unicell has joined #openstack-infra17:06
*** sdake has quit IRC17:07
zaroa jenkins slave disconnect/delete should kill the worker thread i believe so not sure why workers would get stuck in that state.17:07
SpamapSam I crazy or did mordred mention there's a spec to kill glance v2 already?17:07
clarkbzaro: I did not do that, I only offlined it then onlined it17:07
*** ldnunes has joined #openstack-infra17:07
fungidisconnect/reconnect might be worth trying17:07
clarkbzaro: I am sure that if I delete the slave then add it back again that would fix it17:07
fungijust as a confirmation17:08
zaroclarkb: mind giving that a try?17:08
clarkbsure I can do that with 232808017:08
fungii'm guessing that offline/online doesn't recycle the thread17:08
clarkbactually 2328080 is gone17:08
clarkband my tcpdump shows it actually did something17:08
fungioh, did it finally wake back up?17:09
openstackgerritMerged openstack-infra/elastic-recheck: provide a more targetted location setting
clarkbfungi: well it was deleted at least, not sure if it ran a job, looking in nodepool logs now17:09
greghaynesSpamapS: Theres a spec to make glance support image-upload again IIRC17:09
fungithe bare-trusty ready node distribution on the whole still looks the same as it did17:09
SpamapSgreghaynes: ah17:09
fungibut yeah, could have finally hit the ready node delete time17:09
clarkbERROR nodepool.NodePool: SSH Check failed for node id: 232808017:09
clarkbso gearman packets we saw there were it going away, it did not run a job17:10
fungifun times17:10
* clarkb finds a different guinea pig node17:10
*** yamahata has quit IRC17:11
clarkbI am going to offline online it, then see if it grabs some jobs, then do the disconnect connect17:11
*** dhritishikhar_ has quit IRC17:12
clarkbdisconnect, connect fixed it17:14
clarkboffline online did not17:14
*** ivar-lazzaro has quit IRC17:14
*** akshai has joined #openstack-infra17:15
clarkbI think that confirms its something funny with the slave state somewhere17:15
*** dustins_ has quit IRC17:15
*** ivar-lazzaro has joined #openstack-infra17:15
*** eglynn has quit IRC17:16
*** ivar-lazzaro has quit IRC17:16
openstackgerritDavide Guerri proposed openstack-infra/shade: WiP: Add keystone services/endpoints methods
zaroclarkb: i concur.17:16
*** ivar-lazzaro has joined #openstack-infra17:17
clarkbI think I may have found something interesting in the logs17:17
*** dguerri is now known as _dguerri17:18
* clarkb is putting a paste together17:18
*** _dguerri is now known as dguerri17:18
fungizaro: also, when you have a moment, i have a jstack dump and connections/queue output (including the corresponding account names for the tasks) from this morning's stream-events hang17:19
fungisome assistance analyzing it for any interesting details would be appreciated17:19
*** marcusvrn has quit IRC17:19
*** melwitt has joined #openstack-infra17:19
fungias far as i can tell, the thread dump doesn't have any sensitive keys/passwords/tokens in it, so should be safe for me to just stick in a paste17:20
*** andrey-mp has joined #openstack-infra17:20
*** sputnik13 has joined #openstack-infra17:21
zarofungi: ok, can take a look.17:22
zarofungi: did you restart gerrit with
zaroohh i see it's there17:22
zaroi assume no problem with the update?17:23
fungizaro: well, clarkb's change you reviewed yesterday was the upshot of us debugging the problem17:23
*** Apsu has quit IRC17:23
fungias you guessed, we had more than one bouncy castle lib when unpacking the .10 war17:23
clarkbfungi: zaro
clarkbfungi: zaro that shows the node come up, do a grab job, then nothing. I need to read the code but I think that confirms the node got a directive to go to sleep since a job was not available17:24
*** sdake has joined #openstack-infra17:24
andrey-mpdoes anybody know can Ubuntu be used in functional tests? e.g. - can I run vm with ubuntu in gating?17:24
clarkbandrey-mp: are you asking if you can run a nested ubuntu VM on our test infrastructure?17:25
fungiandrey-mp: by default we test on ubuntu 14.04 lts17:25
*** patrickeast has joined #openstack-infra17:25
andrey-mpfungi: i mean - can i run ubuntu instead cirros?17:25
fungiahh, nesting. that would get crazy slow trying to boot ubuntu under qemy inside a cloud instance17:25
fungier, qemu17:25
pleia2openstack gets weird when you run it under qemu, too slow17:26
*** tonytan4ever has joined #openstack-infra17:26
clarkbandrey-mp: you can, but it will be terribly slow17:26
*** pelix has quit IRC17:26
SpamapSandrey-mp: It will take many many minutes to boot17:26
fungithere were some heat functional tests i think that booted fedora under qemu and that was pretty painful. no idea if they still run that17:27
SpamapSand your SSH key will take many many more minutes to generate. ;)17:27
andrey-mpfungi: yeah, I understand it :) but is it possible? if i should load this image into cloud - can i download it from in gating?17:27
fungidownloading random things in a job is fragile17:27
andrey-mpok, sounds good :)17:27
fungispeaking of which...
fungitelling tox to clone tempest seems like a bad idea17:28
clarkbzaro: fungi hrm reading the code I think that the slave really did receive a job assignment, it just doesn't appear to have done anything with it17:28
*** dkranz has joined #openstack-infra17:28
andrey-mpmaybe it can be downloaded from some inside resource?17:28
fungiclarkb: that might explain some of the changes we see spending way longer than they should in some pipelines. i bet zuul gives up on the assignment and puts it back to queued after a while17:29
andrey-mpi understand about fedora - i will see to heat17:29
fungiandrey-mp: devstack has a list of files it can use, and nodepool reads that and downloads them to a local cache when generating our worker images. it would probably need to be a fairly small ubuntu image and you would need to convince devstack reviewers that there's a benefit to adding it17:30
*** dprince has quit IRC17:30
*** wenlock_ has quit IRC17:30
*** dprince has joined #openstack-infra17:30
*** wenlock_ has joined #openstack-infra17:31
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test v1 and v2
clarkbfungi: ya I think if the worker doesn't acknowledge back that it is taking the job the server offers it to another worker17:31
fungiandrey-mp: i also recommend starting an 8gb nova instance with no hardware acceleration for nested virtualization and seeing how well booting that image under qemu does or doesn't work17:31
andrey-mpfungi: thank you! we will discuss it inside our team first17:31
*** yamahata has joined #openstack-infra17:32
fungiandrey-mp: that should give you some indication of whether it will be usable at all for what you're considering17:32
krtaylortimrc, ping, re: your question and neg on  was your question answered? we'd really like to move that along17:33
clarkbzaro: so, I definitely think it is something worker side, but I am not convinced its related to sleep anymore17:33
*** emagana has quit IRC17:33
timrckrtaylor, Ah sorry.  I was super distracted last week.  Let me go look.17:34
clarkbthe logs are revealing much more info, we may actually have this solved shortly :)17:34
*** Apsu has joined #openstack-infra17:34
zaroclarkb: i believe  if the worker receives the job assignment it will execute the job regardless of whether the sever acks or not17:34
krtaylortimrc, thanks!17:35
zaroclarkb: but you are saying the job doesn't get executed?17:35
clarkbzaro: ya there is nothing in the logs after that point about that executor17:35
*** AJaeger__ has joined #openstack-infra17:36
clarkbzaro: if you compare it to the job that did end up running after I did a disconnect/connect the job that did run goes on to log its executing a function and what the params are and all that17:36
clarkbzaro: the job assign uniq from before the slave livelocked did none of that17:36
*** mpaolino has quit IRC17:38
clarkbzaro: we never hit
*** andrey-mp has quit IRC17:39
clarkbzaro: but don't appear to otherwise register any failure17:39
clarkbzaro: which makes me think the work() loop is blocked on something17:39
clarkbthere is also a 2 hour gap between it getting the job assignment and it going into the periodic register functions behavior17:42
*** shardy_ has joined #openstack-infra17:44
fungiclarkb: interesting example, there's a neutron change at the top of the gate right now. all its jobs but one have finished, and that one's only started a few minutes ago17:45
*** shardy has quit IRC17:46
zaroclarkb: does it do anything in work()?17:46
fungioddly, the nova change immediately below it is in the same situation, and even over the same job17:46
*** jamesmcarthur has quit IRC17:47
fungii wonder if there's some separate problem with the neutron tempest full job17:47
fungimaybe tanking the worker under certain situations and resulting in restarts17:47
fungihrm, no further down the pipeline there's a swift change in that situation over the non-neutron tempest full job17:48
clarkbzaro: its doing enough to send the grab job request17:49
openstackgerritMerged openstack-infra/elastic-recheck: Expand fingerprint for git fetch error
clarkbzaro: beause we get the job assignment back, I think I see where the potential race is in work()17:49
clarkbzaro: will have a bunch of links with some execution order that may result in us hanging perpetually17:50
*** shardy_ has quit IRC17:50
*** shardy has joined #openstack-infra17:50
*** notnownikki has quit IRC17:51
*** dkranz has quit IRC17:51
clarkbfirst we send the grab job then we check if the function list is empty at this point if our handleSessionEvent is called and we hit17:52
clarkb we will put a job on the funclist and drain the input buffer then we hit and block until IO is received that is never sent to us17:52
fungiit's possible we're separately having some connectivity issues to workers. i'm seeing if i can stumble across some additional examples17:52
clarkbfungi: connectivity issues in general may be tickling these bugs since it appears to be a race, so possible its related17:53
*** jamesmcarthur has joined #openstack-infra17:53
clarkbfungi: but not the direct cause17:53
fungiright, that's what i'm suspecting. mostly searching for the answer to why we don't see this more often17:53
clarkbzaro: I think we can avoid this case by having stronger locking around those critical sections or we can go lock free witha timeout on the select, check the funclist, reselect if still empty17:54
clarkbI think I prefer going lock free17:54
fungialso, the neutron plugin-specific job i linked to earlier is one place where our git retries aren't going to help us17:54
clarkbfungi: because of the direct clone in tox.ini ya?17:55
*** Sukhdev has quit IRC17:55
*** SumitNaiksatam has joined #openstack-infra17:56
*** hdd has quit IRC17:57
*** e0ne has quit IRC17:57
fungiand not just trivially. it's got pip git cloning neutron and tempest both17:57
*** hdd has joined #openstack-infra17:58
AJaeger__fungi, clarkb: Is this really still debugging our docs problem? Wow - and THANKS!17:58
clarkbAJaeger__: I think I have found the underlying issue :) at the very least it is a race that could cause this problem. Don't have enough logging to confirm it was the exact cuase here17:58
AJaeger__clarkb: Good, those are tricky.17:59
*** hyakuhei has joined #openstack-infra17:59
*** harlowja_away is now known as harlowja18:00
*** hyakuhei has left #openstack-infra18:00
fungiin the jenkins plugin which communicates with zuul to handle job assignment to workers18:00
*** woodster_ has quit IRC18:00
*** emagana has joined #openstack-infra18:01
*** marcusvrn has joined #openstack-infra18:01
*** dhellman_ has joined #openstack-infra18:03
harlowjaSpamapS whats up, something with glance taskflow executor stuff? i didn't make that part of glance, so maybe the glance folks know?18:03
harlowjahopefully u figured it out18:04
SpamapSharlowja: I finally figured it out, but it was _maddening_ to find the entry point for the executor.18:04
*** marcusvrn1 has quit IRC18:04
harlowjasounds like a glance issue?18:04
SpamapSharlowja: also I didnt want to believe it was just glance-api18:04
SpamapSlike, all that trouble, to have the app servers do all the work themselves?18:04
*** dkranz has joined #openstack-infra18:04
harlowjagotta start somewhere afaik18:05
*** davidlenwell has quit IRC18:05
asilenkovhello! I can't signup to who could help?18:05
*** dannywilson has quit IRC18:05
*** davidlenwell has joined #openstack-infra18:05
*** dhellmann has quit IRC18:05
*** _nadya_ has quit IRC18:05
harlowjaSpamapS i assume u are talking about how the workflow runs on glance-api ?18:05
harlowja*runs on the glance-api 'server'18:05
fungiasilenkov: did you follow the steps at in the sequence documented there?18:05
*** ubuntu2 has quit IRC18:06
AJaeger__clarkb, fungi: IS this only hitting the post queue or is this just the one where we noticed it?18:06
fungiAJaeger__: it's more obvious to changes in the post pipeline because it only gets the dregs after release, gate, check et cetera have their pick of workers18:06
harlowjaSpamapS afaik the idea was that when the glance folks wanted they can move it to run elsewhere, but if u have an inflexible framework that can't do that in the first place, moving to one that can start to make that possible (while maintaining the status quo) is easier to integrate than moving to one that can't do that (therefore making a new glance release that much more work for operators...)18:07
fungiso teh starvation becomes more obvious there18:07
asilenkovfungi: yes18:07
*** rm_work|away is now known as rm_work18:07
harlowjaSpamapS but i'm not sure of the current status of all that stuff, the glance folks know best18:07
asilenkovfatal: ICLA contributor agreement requires current contact information.18:07
asilenkovPlease review your contact information:18:07
SpamapSharlowja: So, I don't blame taskflow as much as I blame people landing code with 0 documentation.18:07
asilenkovI can't store it - server error18:07
AJaeger__ah, understood. thanks - will continue listening in. If there's anything I can help, please tell me.18:07
harlowjaSpamapS undersetandable18:07
fungiasilenkov: see the step there in the documentation about signing up as a foundation individual member18:07
fungiasilenkov: did you do that step?18:07
harlowjaSpamapS just saying what i know (which isn't all that much with regards to glance + taskflow integration)18:07
harlowjatis all :-P18:08
*** ubuntu2 has joined #openstack-infra18:08
SpamapSharlowja: and it's something to look out for too.18:08
SpamapSharlowja: the entry points are not as obvious because of the nice disconnected nature of flows.18:08
fungiasilenkov: second paragraph in that section where it says "join The OpenStack Foundation"18:08
harlowjaSpamapS understood18:08
clarkbzaro: I am working on a non locking patch18:08
*** imcsk8 has quit IRC18:08
SpamapSharlowja: the place where executors are entered should be really clear about it.18:09
SpamapSAlso, the words are all double-meaning in glance18:09
harlowjathats a glance thing though :-p18:09
SpamapSimport and python import, task and task flow...18:09
* harlowja didn't write any of that glance + taskflow code (others afaik did) lol18:09
*** otter768 has joined #openstack-infra18:09
*** claudiub has quit IRC18:10
zaroclarkb: i was still trying to grok but i do agree that non-blocking would be better way to go18:10
SpamapSharlowja: my point is, the way they're using your library is confusing and that _does_ end up with frustration, even if it's not your fault.18:11
SpamapSharlowja: It's a PR problem. :-/18:12
harlowjaSpamapS ha, sounds like u should go talk to the glance folks :-P18:12
SpamapSharlowja: I've been trying to all morning. :)18:12
*** rossella_s has quit IRC18:12
SpamapSharlowja: appears none of the task stuff is tested in the gate18:12
harlowjathat su18:13
SpamapSharlowja: which probably explains why it doesn't work at all for me :)18:13
harlowjahopefully works for someone :-/18:13
clarkbzaro: this is more complicated than I expected18:13
fungiharlowja: unlikely. if it's not tested, it's broken ;)18:13
fungimight have worked for someone at one time18:14
harlowjafungi sure, but hopefully someone somewhere has ran it, lol18:14
harlowjabut i know what u mean18:14
clarkbzaro: since I think there are cases where we will want to not block there when the funclist is empty18:14
*** otter768 has quit IRC18:14
*** dustins has joined #openstack-infra18:15
zaroanybody seeing network issues from hpcloud?18:15
*** harlowja has quit IRC18:16
*** harlowja has joined #openstack-infra18:16
zarogetting an error resovling from my hpcloud V<18:16
clarkbzaro: hrm maybe its not complicated since the handleSessionEvent function will send presleep directly when it gets a NO_JOB and will send a grab job when it receives a NOOP18:16
asilenkovfungi: well ty it works now, sorry )18:16
fungiasilenkov: no worries, glad you got it figured out18:17
*** harlowja has quit IRC18:17
asilenkovfungi: we should update our internal mirantis documentation about this stuff, I'll fill a bug.18:17
fungiasilenkov: sure, you can also feel free to link to our infra manual--that's official documentation18:18
fungiwe do try very hard to keep it current and correct18:18
*** harlowja has joined #openstack-infra18:20
*** sabeen3 has joined #openstack-infra18:20
fungiclarkb: here we go... see end of
fungilooks like we're getting more than a few of those18:22
*** sabeen has quit IRC18:22
clarkband during tempest testing too, so tempest is likely at fault18:23
fungifirst one i've caught in the act so maybe it's just the nova v21 job18:23
fungithough evidence suggests it's other tempest jobs too18:23
*** cdent has quit IRC18:25
*** maishsk has joined #openstack-infra18:25
*** rossella_s has joined #openstack-infra18:25
*** bswartz has quit IRC18:26
* zaro afk, bb in a hrs.18:27
*** bswartz has joined #openstack-infra18:28
*** markvoelker has quit IRC18:28
*** harlowja has quit IRC18:29
*** sdake has quit IRC18:29
*** harlowja has joined #openstack-infra18:29
*** dguerri is now known as _dguerri18:30
clarkbhrm untangling this is difficult, I am very afraid of adding new (dead|live)locks18:30
*** e0ne has joined #openstack-infra18:31
*** e0ne is now known as e0ne_18:31
*** abregman has joined #openstack-infra18:32
*** e0ne_ is now known as e0ne18:32
*** e0ne is now known as e0ne_18:32
*** e0ne_ is now known as e0ne18:32
*** spredzy is now known as spredzy|afk18:32
*** e0ne has quit IRC18:32
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check
*** _dguerri is now known as dguerri18:34
*** abregman has quit IRC18:34
*** arxcruz has joined #openstack-infra18:35
ctlaugh_krtaylor: In the meeting earlier today, you mentioned sending out some e-mails that no one had commented on.  I don't believe I am subscribed to that mailing list -- could you point me to which one that is?18:36
*** annegentle has quit IRC18:36
*** emagana has quit IRC18:36
*** sdake has joined #openstack-infra18:37
*** dhp_ has quit IRC18:38
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow
*** marun has quit IRC18:39
clarkbfungi: did you delete the nodes that had been ready and idle? I don't need them but can tell nodepool to delete them if you haven't18:39
*** arxcruz has quit IRC18:39
fungiclarkb: i hadn't. didn't want to yank any out from under you, but will do it now18:40
clarkbya I think I grok the race(s) here, basically we can "driveIO" from multiple places so can't assume that blocking waiting for new IO before driving further is going to return18:41
fungiall cleaned up now, though i wouldn't be surprised to see us accumulate more (perhaps fairly rapidly if the network problems keep up)18:41
clarkbit also explains why we would register out of band but not actually run a job, because the loop that handles the work isn't reset in that case18:41
clarkbfungi: ya we should keep an eye on it18:41
*** dguerri has quit IRC18:42
lifelessfungi: - could you re-add your +2 ?18:42
krtaylorctlaugh_, that was the openstack-dev mail list18:43
*** emagana has joined #openstack-infra18:43
clarkbits kind of a mess because the java gearman lib doesn't really give us a good async model. Instead we are sharing a Selector to select on which is a bit :( imo18:46
*** maishsk has quit IRC18:47
clarkbif we weren't doing that it would be as simple as matching up Selector.wakeup() and calls18:47
*** e0ne has joined #openstack-infra18:48
*** rmcall has quit IRC18:48
openstackgerritLouis Taylor proposed openstack-infra/project-config: Add functional test job for python-glanceclient
*** rmcall has joined #openstack-infra18:49
*** annegentle has joined #openstack-infra18:49
*** annegentle has quit IRC18:52
*** annegentle has joined #openstack-infra18:53
clarkbhrm handleSessionEvent is synchronous with driveSessionIO18:58
*** erlon has joined #openstack-infra18:58
*** tsg_ has joined #openstack-infra18:59
clarkbexcept maybe not because we have two different handleSessionEvents and one does not appear to override the other19:00
*** emagana has quit IRC19:01
*** mrmartin has joined #openstack-infra19:02
*** ivar-laz_ has joined #openstack-infra19:02
*** ivar-laz_ has quit IRC19:02
*** mrunge has quit IRC19:03
*** dustins has quit IRC19:03
*** ivar-laz_ has joined #openstack-infra19:03
fungithe post pipeline backlog dropped like a stone after i cleaned up those stale workers gumming up the ready pool19:03
*** emagana has joined #openstack-infra19:04
fungialso my fishing expedition has turned up no new restarted jobs, so i think whatever the cause was has subsided19:04
*** ivar-lazzaro has quit IRC19:04
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update secgroup API for new get/list/search API
*** dhellmann has joined #openstack-infra19:06
*** jtriley has joined #openstack-infra19:06
*** jaypipes has quit IRC19:06
openstackgerritMerged openstack/requirements: Update keystoneclient requirement
*** rmcall has quit IRC19:07
*** rmcall has joined #openstack-infra19:07
clarkbthe more I read the less I am convinced this is the issue, if select blocked indefinitely it would certinly explain this problem I just can't figure out why it would block like that19:08
*** cody-somerville has quit IRC19:09
*** dhellman_ has quit IRC19:10
*** teran has joined #openstack-infra19:12
clarkbI think its the function registration19:14
clarkbit happens in the log and does a select wakeup19:14
clarkbwhich means if there is >1 select happening only that one will wake19:14
*** marcusvrn has quit IRC19:15
clarkbya ioAvailable is shared with the session object which could select on it independently19:16
*** ivar-laz_ has quit IRC19:16
teranHi folks, I have a user Denis Meltsaykin with two accounts in, could you please help?19:17
*** ivar-lazzaro has joined #openstack-infra19:17
clarkbyup the gearman session runs selectNow() on the selector which clears and wakeups19:17
*** salv-orl_ has joined #openstack-infra19:18
fungiteran: sure, just a sec and i'll see what i can do19:18
*** marcusvrn has joined #openstack-infra19:18
*** unicell has quit IRC19:18
*** salv-orlando has quit IRC19:22
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check
fungiteran: there shouldn't be any duplicate as of 8 days ago. i think i remember cleaning that one up myself when someone mentioned it in here19:22
fungiteran: is it still causing an issue?19:22
*** jaypipes has joined #openstack-infra19:23
teranfungi: I see one of accounts just without email, but there are two still. I'll poke him check but now I asking since there was no reply from previous mentioning him(or I simply missed it):19:24
*** AJaeger__ has quit IRC19:25
teranfungi: anyway thanks :)19:25
*** dustins has joined #openstack-infra19:25
fungiteran: i can also blank out his full name in the database, but that will cause any comments left by that account to appear as "anonymous"19:26
fungiso i usually don't do that19:26
teranfungi: no necessary for now, I think, let's give Denis a chance to try authentication :)19:26
*** cody-somerville has joined #openstack-infra19:28
*** akshai has quit IRC19:28
*** miqui has quit IRC19:29
*** rlandy has joined #openstack-infra19:29
*** woodster_ has joined #openstack-infra19:31
*** dizquierdo has joined #openstack-infra19:35
*** unicell has joined #openstack-infra19:43
lifelessfungi: thanks19:44
*** david-lyle has quit IRC19:44
*** spzala has joined #openstack-infra19:44
lifelessso I need some brainpan time from someone familiar with the zuul application stuff19:44
*** sarob has quit IRC19:44
*** david-lyle has joined #openstack-infra19:44
*** sarob has joined #openstack-infra19:45
openstackgerritlifeless proposed openstack-infra/project-config: Fixup the PBR_PIP_VERSION more.
openstackgerritlifeless proposed openstack-infra/project-config: Don't setup devstack for pbr integration tests.
fungilifeless: the goal is to update the git clones in /opt without copying them to another directory tree first?19:46
lifelessfungi: yes19:47
lifelessfungi: well there are two goals19:47
lifelessfungi: they're tangled together19:47
lifelessfungi: one is to stop running devstack19:47
lifelessfungi: since its entirely wasted in this job19:47
lifelessfungi: that seems to require switching to using /opt/git.19:47
fungilifeless: /opt/stack and /opt/git are on the same filesystem... what was the i/o involved in copying a directory tree there?19:48
lifelessfungi: the second goal is avoiding the copy of a couple GB of data around to no point19:48
lifelessfungi: I'm fairly sure its not hardlinking them19:48
bknudsonis it possible for us to do in our libs? e.g., for argparse since it's only needed for 2.619:48
lifelessbknudson: no, for two reasons19:48
fungilifeless: yeah, if it clones from file:// rather than just the pathname then it won't hardlink19:49
*** rmcall has quit IRC19:49
lifelessbknudson: firsly, its needed in 2.7+ when things Require('argparse'), because the one in the stdlib isn't registered as a package19:49
openstackgerritAndrew Smith proposed openstack-infra/jenkins-job-builder: Add support for git sparse checkout
*** HeOS has joined #openstack-infra19:49
lifelessbknudson: secondly, we don't have proper support for *either* extras, or [better] environment markers in pbr yet. We're working up the stack to do that.19:49
bknudsonlifeless: I think that's exactly what we're running into.19:49
lifelessbknudson: but excluding argparse would be wrong anyway19:50
lifelessbknudson: (because of 1))19:50
lifelessfungi: however I don't know how to best sensibly reuse the 'apply the zuul refs to the trees' in situ on /opt/git19:50
lifelessbknudson: what are you running into ?19:51
*** dannywilson has joined #openstack-infra19:51
bknudsonlifeless: so we're building rpms... the rpms say they require python-argparse, and python-2.7 package says it provides python-argparse19:51
bknudsonbut then pkg_resources says there's no argparse19:52
*** dannywilson has quit IRC19:52
lifelessbknudson: sounds like you need a real python-argparse package with just the egg metadata in it19:52
clarkblifeless: you want to use zuul-cloner, look in openstack-infra/system-config/tools/ iirc19:52
*** dannywilson has joined #openstack-infra19:52
*** dprince has quit IRC19:52
clarkbalso zuul-cloner should hardlink iirc19:52
fungilifeless: i'm looking now to see whether zuul-cloner will let you update and operate on an existing git repository in your workspace without helpfully blowing away and re-cloning it for you19:53
bknudsonlifeless: yes, and there is a real python-argparse rpm, but yum isn't installing it automatically probably because python says it provides it.19:53
clarkbfungi: I don't think it will, but it will hardlink19:53
clarkbfungi: so effectively the same thing19:53
lifelessclarkb: jenkins/scripts/ ?19:53
fungihardlinks might be "good enough" in this case19:53
lifelessit has git clone file:///19:53
clarkblifeless: no, openstack-infra/system-config/tools/ uses zuul-cloner, this has nothing to do with g-g-p19:53
fungilifeless: talking about changing up the job to call the zuul-cloner utility from openstack-infra/zuul19:53
*** ivar-lazzaro has quit IRC19:54
fungilifeless: for example infra's puppet-apply integration tests use it19:54
*** jamesmcarthur has quit IRC19:54
fungiit's basically a rewrite of devstack-gate's git cloning logic in python19:54
*** ivar-lazzaro has joined #openstack-infra19:54
lifelessbknudson: so, yum's python 2.7 advertises something falsely. How would changing the install-requires on openstack projects fix the issue?19:55
lifelessbknudson: whatever is calling pkg_resources.requires('argparse') would still do so, no?19:55
bknudsonlifeless: yes, that python 2.7 advertisement does seem wrong.19:55
fungithis goes back to the "we don't test specific versions of python, we test distributions of python" discussion19:57
*** ajmiller has joined #openstack-infra19:58
clarkbfungi: the more I stare at this code the less convinced I become of my fix and the bug19:59
clarkbI am going to take a break and find lunch19:59
*** ajmiller_ has joined #openstack-infra19:59
*** doude has quit IRC20:01
*** doude has joined #openstack-infra20:01
clarkbit does definitely appear to be an interaction between sending a grab job requests and having setFunctions() be called at that same time due to a configuration update20:01
clarkbboth affect the selector, one by running selectNow() and the other by doing a wakeup() then later we do a vanilla blocking select() that seems to select forever20:02
lifelessfungi: so the second angle- stopping bringing up devstack20:03
lifelessfungi: would be a win in and of itself20:03
lifelessfungi: thats about 10m of runtime in the job that we entirely fail to care about20:03
clarkbits actually almost twice that so even more win20:03
clarkblifeless: you can either just run d-g with an overridden gate_hook, or have zuul-cloner do the cloning for you20:04
fungiyeah, the comment saying 10 minutes has... uh... bitrotted a little20:04
fungiclarkb: devstack is started before the gate_hook though, right?20:04
lifelessok so - I have a patch to projcet-config and one to pbr both aiming at that20:05
* fungi refreshes recollections20:05
lifelessbut I'm out of my depth :020:05
fungiahh, no clarkb's right20:05
lifelessso - if either of you could take mercy on me and be very very specific, or perhaps push up something, that would rock. There's no super urgency on it, though cutting fat out would be very nice20:06 won't be run if you supply a different gate_hook20:06
*** kgiusti has quit IRC20:06
fungilifeless: yeah, just a sec. i'll see what can be done there20:06
clarkbwhich we actually already seem to do20:07
lifelessthere's a 5m gap here:
*** eharney has quit IRC20:07
lifelessand then a 10m gap here20:07
clarkbso devstack is not run20:07
*** megm has joined #openstack-infra20:08
clarkblifeless: the first gap is the clone and fetch things gap20:08
*** esker has quit IRC20:08
lifelessI don't have whats going on in those internalised in any way20:08
lifelessclarkb: yeah, I thought so :)20:08
*** Rockyg has joined #openstack-infra20:08
fungilifeless: the first gap is
clarkblifeless: the setup_workspace log file is the log for that ya ^20:09
clarkband it appears it does run devstack? thats weird20:10
fungithe second gap seems to be installing things
*** ivar-lazzaro has quit IRC20:10
*** otter768 has joined #openstack-infra20:10
lifelessso about 30s to update just nova20:11
*** ivar-lazzaro has joined #openstack-infra20:11
fungioh, yeah, and running devstack services20:11
clarkbOH! this runs devstack20:11
clarkbI think you can just delete that line20:11
openstackgerritMerged openstack-infra/tripleo-ci: Set root disk == node disk for puppet jobs
clarkband only run the integration script20:11
*** emagana has quit IRC20:11
lifeless ?20:11
lifelessclarkb: ^20:11
clarkbyup +220:11
lifelessclarkb: it has a dependency too20:11
fungii have to admit i don't know the difference between #egg=pipexport and #egg=pip20:13
fungioh! #egg=pipexport was a typo20:13
clarkbya, it was appended20:13
fungifrom one of the many other export lines20:13
fungigot it20:13
clarkbok really lunch time now, I think that solves a large chunk of the time waste here, using zuul-cloner is going to be a bit more involved20:13
*** emagana has joined #openstack-infra20:14
openstackgerritArmando Migliaccio proposed openstack-infra/devstack-gate: Follow up with some comments to clarify OVS commands
*** otter768 has quit IRC20:15
*** jtriley has quit IRC20:15
*** annegentle has quit IRC20:15
megmAfter lunch, I'm having trouble getting set up with gerrit if someone can help.20:16
fungimegm: did you follow the steps at in the sequence documented there?20:17
megmYes -- Anne has been helping me but I'm getting errors she hasn't seen after I run  ssh-keygen -f ~/.ssh/known_hosts -R []:2941820:18
*** dguerri has joined #openstack-infra20:18
megm# Host []:29418 found: line 1 type RSA20:19
megmline 3 invalid key: ssh-rsa...20:19
megm/home/megm/.ssh/known_hosts is not a valid known_hosts file.20:19
megmNot replacing existing known_hosts file because of errors20:19
*** jtriley has joined #openstack-infra20:19
*** tonytan4ever has quit IRC20:21
fungimegm: that's odd. sounds like your .ssh/known_hosts file is corrupt somehow. can you try renaming it? mv .ssh/known_hosts .ssh/known_hosts.old20:22
openstackgerritMerged openstack-infra/project-config: Fixup the PBR_PIP_VERSION more.
fungimegm: then see if trying again prompts you properly to accept the host key for review.openstack.org20:22
openstackgerritMerged openstack-infra/project-config: Don't setup devstack for pbr integration tests.
fungimegm: oh! wait20:22
*** marcusvrn has quit IRC20:23
fungilet me test what you have there just to be sure20:23
openstackgerritJulia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through
openstackgerritJulia Kreger proposed openstack-infra/shade: Change ironic maintenance method to align with power method
*** annegentle has joined #openstack-infra20:23
*** _nadya_ has joined #openstack-infra20:23
megmhi, annegentle...20:24
annegentlemegm: hi120:24
annegentleer that should be !20:24
megmI'm pretty good at interpreting typos ;-)20:24
annegentlemegm: heh20:24
fungimegm: yeah, looks like that should delete the host key for if it finds it in the known_hosts file, but for some reason it's having a hard time reading your known_hosts file20:24
annegentlefungi: thanks for the help, I wasn't sure how to troubleshoot further without risking the system20:25
fungimegm: i've also confirmed that (at least for me) the error message is different if the known_hosts file doesn't exist at all20:25
annegentlemegm: what does ls -la ~/.ssh/ look like20:25
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check
*** e0ne has quit IRC20:26
*** marun has joined #openstack-infra20:26
fungimegm: so, yeah, chances are just moving that known_hosts file out of the way (since it seems to be corrupt) will let you proceed to testing ssh20:27
megmOkay, let me try20:28
megmssh-keygen -f ~/.ssh/known_hosts -R []:29418 -- do I need to manually recreate the known_hosts file?20:30
fungimegm: no, you should just be able to skip that step now20:30
*** jamesmcarthur has joined #openstack-infra20:31
fungimegm: that command is for removing an existing entry for from your known_hosts file if it exists20:31
megmhere is ls -lai of .ssh:  13109276 drwx------  2 megm megm  4096 Apr 29 13:28 .20:31
megm11403265 drwxr-xr-x 50 megm megm  4096 Apr 29 10:42 ..20:31
megm13109261 -rw-------  1 megm megm  1766 Jan 20  2014 id_rsa20:31
megm13109264 -rw-r--r--  1 megm megm   404 Jan 20  2014 id_rsa.pub20:31
megm13109265 -rw-r--r--  1 megm megm  1288 Mar  3 18:54 know_hosts.old20:31
megm13109280 -rw-rw-r--  1 megm megm  2200 Jan 20  2014 meg.notes20:31
megm13109285 -rw-rw-r--  1 megm megm 10240 Feb  7  2014 ssh.tar20:31
fungimegm: but the next time you use ssh it will make a new one20:31
megmAh, so on to ssh -p etc20:31
*** ivar-lazzaro has quit IRC20:32
*** ivar-lazzaro has joined #openstack-infra20:32
*** ajmiller_ has quit IRC20:33
*** ajmiller has quit IRC20:34
megmOkay, progress...  ssh -p seemed to work -- it spit out a long list of stackforge/* filenames.  And git remote -v gives gerrit  ssh:// (fetch)20:34
megmgerrit  ssh:// (push)20:34
megmorigin (fetch)20:34
megmorigin (push)20:34
megmBut git review -s fails:  Problem running 'git remote update gerrit'20:34
megmFetching gerrit20:34
megmfatal: Project not found: openstack/doc-specs20:34
megmfatal: Could not read from remote repository.20:34
megmPlease make sure you have the correct access rights20:34
megmand the repository exists.20:34
megmerror: Could not fetch gerrit20:34
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions
*** e0ne has joined #openstack-infra20:36
fungimegm: looks like that repository doesn't exist20:37
fungiaha, i bet it's a typo in the .gitreview file in docs-specs20:37
funginope, it's correct there20:38
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: add for testing web content
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: actually actually make window location work
megmMaybe because my user name isn't in there?  I screwed up a long time ago -- I'm StackScribe in git but SstackScribe in gerrit...20:38
*** mrmartin has quit IRC20:39
fungimegm: did it really say "Project not found: openstack/doc-specs" or did it say "Project not found: openstack/docs-specs" instead?20:39
fungioh, your remote list seems to confirm20:40
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: Adding publisher support to join-trigger plugin
fungisomehow you've got an origin of docs-specs at github but doc-specs in gerrit20:40
megmI just copied the text from my tty window...20:41
*** eharney has joined #openstack-infra20:41
fungimegm: try 'git remote remove gerrit' followed by 'git review -s'20:41
*** teran has quit IRC20:42
fungiit looks like maybe your gerrit remote was added manually or something, and mistyped20:42
megmThat's odd -- I have no idea how I did that.20:42
*** teran has joined #openstack-infra20:42
fungibut git review -s should add the correct remote based on the contents of the .gitreview file in the repository20:42
megmI think that worked!  Horray!  Let me try a git review -d to confirm20:42
megmI AM IN!!!!!  Thank you so very much!20:43
*** radez is now known as radez_g0n320:44
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check
annegentlemegm: woohoo!20:44
*** e0ne has quit IRC20:44
fungimegm: you're welcome--have fun!20:45
megmWords cannot express how happy this makes me!  Now I can actually dig in and do some writing ;-)20:45
*** jcoufal has joined #openstack-infra20:45
*** mtreinish_ has joined #openstack-infra20:45
fungiyou're in like kevin flynn20:45
*** teran has quit IRC20:46
*** dustins has quit IRC20:48
openstackgerritMerged openstack-dev/pbr: Test pip install -e of projects.
openstackgerritMerged openstack-dev/pbr: Stop testing easy_install behaviour
*** dizquierdo_ has joined #openstack-infra20:51
*** mtreinish has quit IRC20:51
*** mtreinish_ is now known as mtreinish20:51
*** ajo has quit IRC20:51
*** dizquierdo has quit IRC20:51
*** bswartz has quit IRC20:54
fungilifeless: here's a testing conundrum for you... is it safe/sane to monkeypatch sys.exit?20:54
*** belmoreira has joined #openstack-infra20:55
fungii'm trying to add a version to optparse in bindep and write a test to exercise it, but optparse seems to directly call sys.exit(0) on its own, unsolicited, when passed --version20:55
*** amitgandhinz has quit IRC20:56
fungidoesn't actually seem to be configurable20:56
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: actually actually make window location work
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: add for testing web content
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: Adding publisher support to join-trigger plugin
*** dannywilson has quit IRC20:57
*** dannywilson has joined #openstack-infra20:58
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check
fungioh, maybe i can assertRaises on SystemExit as a solution20:59
sdaguefungi: you can catch SystemExit20:59
*** amitgandhinz has joined #openstack-infra20:59
*** tiswanso_ has quit IRC21:00
fungiin this case i have to make lambda: main() too i think21:00
*** tonytan4ever has joined #openstack-infra21:01
fungicloser, but it's still not getting the output into the fixture logger21:02
*** pabelanger has quit IRC21:02
lifelessfungi: hi21:02
* fungi is embarrassingly a total n00b at testing21:02
lifelessfungi: there's a way to override the beaviour of the parser21:02
lifelessfungi: its usually recommended here.21:03
fungifor optparse?21:03
openstackgerritGary W. Smith proposed openstack/requirements: Add python-manilaclient to global requirements.
*** jcoufal has quit IRC21:03
fungipatching optparse.OptionParser or something else?21:03
lifelessso for testing the general pattern is to inject your dependency21:04
*** _nadya_ has quit IRC21:04
lifelessso you'd make main accept a parser factory21:04
*** esker has joined #openstack-infra21:04
lifelessor decouple things in some way - there's a few things you could do21:04
lifelessbut the least instrusive would be to monkey patch optparse.OptionParser.exit / .error21:04
*** cody-somerville has quit IRC21:05
fungigot it. so far the other tests here are patching sys.argv and then calling main() which engages optparse. i'll study up--thanks!21:05
*** marun has quit IRC21:08
*** peristeri has quit IRC21:09
*** jtriley has quit IRC21:12
*** Sukhdev has joined #openstack-infra21:14
*** ldnunes has quit IRC21:14
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions
*** mriedem has quit IRC21:17
*** samueldmq has quit IRC21:21
*** dkranz has quit IRC21:22
gary-smithfungi, clarkb: so for pestering, but could you re-review ? Just trying to get this done21:22
*** achanda has quit IRC21:22
fungigary-smith: my +2 is still there from ~4 hours ago21:24
fungiAJaeger: ^21:24
fungihe seems to have addressed your comment21:24
gary-smithfungi: apologies, read that wrong21:24
fungiwell, my irc nick isn't immediately obvious from my gerrit full name21:25
gary-smithfungi: thanks for re-re-reviewing it then :-)21:25
fungino apologies needed21:25
*** ddieterly has quit IRC21:25
gary-smithyeah, I noticed that infra team nicks are a bit, well, colorful :-)21:25
*** shardy has quit IRC21:26
*** ddieterly has joined #openstack-infra21:26
fungiit's a funhouse 'round here21:26
gary-smithso is 'fungi' a reference to being a fun-guy, or is it like ringworm?21:28
fungiwhatever you prefer21:28
*** Longgeek has quit IRC21:29
fungitruth be known, it's a reference to
*** marun has joined #openstack-infra21:29
fungibut don't tell anyone21:29
gary-smithwill do. er, won't do21:30
*** ddieterly has quit IRC21:30
*** Rockyg has quit IRC21:30
*** julim has quit IRC21:30
*** cody-somerville has joined #openstack-infra21:31
*** marun has quit IRC21:35
*** stevemar has quit IRC21:36
*** achanda has joined #openstack-infra21:36
*** harlowja is now known as harlowja_away21:36
*** teran has joined #openstack-infra21:36
*** harlowja_away is now known as harlowja21:37
*** esker has quit IRC21:39
*** mattfarina has joined #openstack-infra21:40
*** unicell has quit IRC21:40
*** unicell has joined #openstack-infra21:40
*** esker has joined #openstack-infra21:40
*** mattfarina has quit IRC21:40
jesusaurushow do logs make it from the nodepool node to elasticsearch? im vaguely aware of a python script that reads files and writes events over tcp to logstash but I don't have a firm grasp on how that system works21:41
fungijesusaurus: if that doesn't answer your questions, then we should expand it21:42
redrobothello infra folk!21:42
redrobotdoes anyone have pointers as to why this CR is not merging?
jesusaurusfungi: oh, perfect, a magic link, i like those21:43
*** SumitNaiksatam has quit IRC21:43
redrobotit appears the gate job is not triggering? :(21:43
*** marun has joined #openstack-infra21:44
fungiredrobot: top of
redrobotfungi ack, thanks!21:44
fungilooks like it was approved between 13:29-14:0521:45
fungiso someone needs to (if necessary remove and then re-)add a workflow +1 to it21:45
fungiredrobot: ^21:45
redrobotfungi I see... I thought "reverify" would take care of it?21:46
funginope. reverify is like recheck. they're synonymous21:46
*** jtriley has joined #openstack-infra21:47
fungithey cause the check pipeline jobs to be rerun, however because of the way gerrit works when zuul tries to leave a new +1 over top of the old +1, gerrit streams that event out as not updating the vote, so gating doesn't get triggered21:48
fungiit's something we'd like to fix21:48
openstackgerritlifeless proposed openstack-dev/pbr: Issue #1450210: Preversioning rc tag handling
redrobotfungi interesting.  that's good to know.  thanks again!21:48
lifelessfungi: ^ thats an rc bug vis-a-vis releasing 0.1121:49
fungigenerally only becomes an issue when zuul misses an event from the stream in the first place though, which we'd like to fix more (the current bugs we're experiencing with gerrit causing us to have to restart it to get the event stream working again)21:49
fungilifeless: good catch21:50
lifelessfungi: been tracking down oddness with alan pevec for a bit21:51
lifelessfungi: finally got a case that was happening with master21:51
*** annegentle has quit IRC21:51
*** bnemec has quit IRC21:52
lifelessclarkb: ^ if you're back21:54
*** tonytan4ever has quit IRC21:54
*** bhunter71 has quit IRC21:54
lifelessmordred: if you're up?
*** shivharis has joined #openstack-infra21:55
*** esker has quit IRC21:55
shivharisanteaya: hi21:56
zaroclarkb: so have you decided what you want to do with gearman plugin?21:58
zaroare we monitoring it?21:58
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: OpenstackId resource server
fungi+6671, -021:59
clarkbzaro: not actively, fungi cleared out the bad nodes. Its easy to check if we get in that state again. I have mostly been staring at the code all day unraveling what it is doing21:59
clarkbzaro: I think there may be layers of bugs here, but basically we have to driveIO and depend on the select() for some of that but then will submit tasks and jobs behind work()s back22:00
*** tsg_ has quit IRC22:00
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Migration of resource server
*** tsg_ has joined #openstack-infra22:01
*** bnemec has joined #openstack-infra22:02
*** melwitt has quit IRC22:02
*** melwitt has joined #openstack-infra22:02
clarkbzaro: I think the intent of the taskmap in the worker implementation was to keep track of this but then work() never uses it?22:03
*** hdd has quit IRC22:03
clarkbzaro: we basically add and remove tasks from that but we never check it22:03
*** Swami has quit IRC22:04
*** patrickeast_ has joined #openstack-infra22:04
*** prad has quit IRC22:04
*** patrickeast has quit IRC22:05
*** patrickeast_ is now known as patrickeast22:05
openstackgerritDavide Guerri proposed openstack-infra/shade: Add keystone services/endpoints methods
openstackgerritDavide Guerri proposed openstack-infra/shade: Rename get_endpoint() to get_session_endpoint()
*** rlandy has quit IRC22:09
clarkbzaro: I am not sure we properly sleep22:10
clarkbzaro: instead we go into a sleep state then immediately send a grab job then will go into a sleep state again22:10
clarkbhrm no thats not quite right either22:11
*** otter768 has joined #openstack-infra22:11
*** shivharis has quit IRC22:12
*** shivharis has joined #openstack-infra22:12
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check
clarkbzaro: handleSessionEvent definitely doesn't have a way to communicate to work() that it sent grab job on receiving a NOOP from what I can tell22:12
zaroclarkb: looking at it i'm not sure why there's a taskmap.  it's definately not getting used.22:14
clarkbzaro: I think its meant to keep track of this22:14
clarkbits just not doing that22:14
clarkbso basically we don't need the grab job sent boolean, instead we should check the taskmap22:15
lifelessclarkb: thanks, wht did you think?22:15
clarkblifeless: sorry got completely distracted by java and I don't know why, opened the change in my browser then squirrel!22:15
clarkbwill actually look now22:15
*** otter768 has quit IRC22:16
clarkbzaro: also we should only have a single session right? not sure why we need a map, i think we just need a reference to the current task22:17
zaroclarkb: you mean taskmap object is meant for communicating with work()?22:17
clarkbzaro: ya, its tracking what the outstanding task is22:17
clarkbzaro: so that work() knows if it needs to drive IO or send another grab job etc22:17
lifelessclarkb: java the drink or java the language?22:18
*** belmoreira has quit IRC22:18
*** dguerri is now known as _dguerri22:18
*** tsg_ has quit IRC22:18
*** nelsnelson has quit IRC22:18
clarkblifeless: java the language22:18
*** salv-orlando has joined #openstack-infra22:19
clarkblifeless: the bug says 2015.1.0.0rc3.dev1 is expected but isn't that an invalid version under pep440?22:20
lifelessits entirely valid22:20
lifelessfixing pbr to generate those was the thing master was blocked on for 6 months22:20
zaroclarkb: yeah, i think multiple tasks per session, but a single session.22:20
clarkbzaro: but its using the same session always as the key22:21
clarkblifeless: ah yup, I am brushing up on 440 now22:21
lifelessclarkb: note that the version generation logic isn't involved in this patch22:21
lifelessclarkb: its simply logic to not emit a lower version than we've already seen22:21
fungiclarkb: also remember that even if you brushed up on it while it was being drafted, it changed in subtle ways several times (including after it was finalized) so refreshing is highly recommended22:22
openstackgerritAlexander Yip proposed openstack-infra/project-config: Add congress.yaml file
*** salv-orl_ has quit IRC22:23
*** amitgandhinz has quit IRC22:25
*** bswartz has joined #openstack-infra22:27
*** esker has joined #openstack-infra22:27
clarkblifeless: so basically distance was calculated from 2015.1.0.0rc2 as 1, but then we use target_version anyways22:27
clarkblifeless: wouldit be simpler to just use the tag returned, make a version of tag + distance?22:27
clarkbassuming tag is not none22:28
lifelessclarkb: no22:28
lifelessclarkb: if you have done a commit22:28
lifelessclarkb: you don't want to say that the version is the version of the tag which is a commit back22:28
lifelessclarkb: distance is 'number of commits since a tag'22:28
clarkbyou want to sa its the tag + distance22:28
*** dimtruck is now known as zz_dimtruck22:29
lifelessclarkb: so there are two cases. preversioning and postversioning22:29
lifelesspostversioning is easy: increment the version number and use distance for the dev count.22:29
lifelesspreversioning is complicated by two things22:29
clarkbbut in either case its tag + distance thenflip the string from dev to post depending22:29
lifelessclarkb: no, we never use post22:29
clarkblifeless: I think we have to with new pip22:30
lifelessclarkb: no, we don't22:30
clarkb0.10 pbr definitely does use post because of this22:30
fungier, for postversioning you don't even increment the version number do you? just the dev count22:30
clarkbfungi: you use postN22:30
lifelessclarkb: it does because folk around at the time chose to hack up 0.10 rather than fix master.22:30
fungier, right post not dev22:30
clarkblifeless: yes because master was in a bad place22:30
lifelessclarkb: which was a mistake, and a bug, and not aligned with pep-440.22:30
clarkblifeless: and 0.10 worked22:30
clarkbit is aligned with 440...22:30
fungifor postversioning, don't increment the version number, just increment the post counter22:30
lifelessclarkb: it really wasn't, it was very fast to change for me. But I don't want to fight over that. Its pointless22:31
lifelessclarkb: no, its not. read the definition of postN. They are public versions.22:31
fungias opposed to preversioning where you need to increment the version number and then the dev count from the most recent tag22:31
clarkblifeless: both dev and post are public versions22:31
fungiwell, post is "public" in the sense that it denotes minimal changes to metadata/packaging22:32
lifelessclarkb: yes22:32
lifelessclarkb: public and defined semantics22:32
lifelessclarkb: "Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes).22:32
fungiwhereas dev is something the pep440 authors didn't envision being uploaded to pypi22:32
lifelessclarkb: that is *not* what pbr 0.10 uses post or.22:32
clarkbon a number line we have 1.1.dev1 < 1.1 < 1.1.post122:32
openstackgerritAlexander Yip proposed openstack-infra/project-config: Add congress.yaml file
clarkbthis is important because for post versioning we need to sort after 1.122:33
lifelessclarkb: Can we step back.22:33
clarkbsure, we need to ensure consistent sorting22:33
clarkblets start from there22:33
lifelessclarkb: I'm confused. We have a spec, for using dev. It works with every single case, and there was an impl bug in pbr that fixes this bug.22:33
clarkblifeless: it does not work for sorting this is specifically why we used post22:33
clarkbwe had to or pip sorts wrong22:33
lifelessclarkb: I'm sorry, but you're wrong.22:33
clarkbhow? can you show me the sort order?22:34
clarkbis the order above wrong?22:34
lifelessignore post22:35
*** jamesmcarthur has quit IRC22:35
lifelessI know pbr 0.10 uses it. But just ignore it, ok?22:35
*** erikmwilson is now known as Guest6178022:35
*** erikmwilson has joined #openstack-infra22:35
lifelessWe can come back to *why* to ignore it later.22:35
*** whoops has quit IRC22:35
clarkbou seem to agree that dev sorts first based on that post22:35
clarkblifeless: I can't ignore it22:36
lifelesswhy can't you?22:36
clarkbthat was a very important part of the puzzle when we unbroke everything 4 months ago22:36
clarkbbecause of sort order22:36
lifelessso the break that occured 4 months ago was because we didn't honour rc tags at all in master22:36
clarkband we broke post versioning22:36
lifelesswhich was a genuine problem, and in deed this patch is fixing another instance of that.22:37
clarkband about half a dozen other things22:37
*** Guest61780 has quit IRC22:37
mordreddev was how we fixed 0.10, but it's not applicable in master22:37
lifelessmordred: s/dev/post/ -22:37
mordredyeah. sorry - it's late here22:37
*** ZZelle_ has joined #openstack-infra22:37
lifelessclarkb: I am aware of that issue, and if you dig it up you'll see we only ever emitted dev versions, never rcX.devN versions22:37
clarkbcan someone explain why using sort order?22:37
lifelessclarkb: and thats *why* this fix is fine.22:37
clarkblifeless: that is for preversioning22:38
lifelessclarkb: no, everything.22:38
clarkbwhere we have a target that we are moving towards22:38
clarkbin the case where we have tagged a release and use post versioning the next commit must sort higher than the tag itself22:38
lifelessclarkb: again, all harmonised in master a year ago, except for a shallow bug that was fixed recently.22:38
clarkbusing dev does not do that22:38
lifelessclarkb: I think it might help if you read the pbr master docs on this, because thats exactly what master does.22:38
lifelessclarkb: it bumps the version number automatically.22:39
fungipart of the new world order for pbr is insisting on preversioning everywhere and inventing a new goal you're developing toward if you haven't specified one in some metadata for your packaging22:39
lifelessclarkb: and there are a raft of tests.22:39
lifelessclarkb: and docs about this22:39
fungii can't say i agree with it, but it seems to be what some people want22:39
clarkblifeless: ok thats the missing piece and all you had to say "pbr emits the next version"22:39
lifelessclarkb: I feel like I'm arguing with you about what pbr 0.9 did22:39
lifelessclarkb: which prompted *all* this wwork in the first place22:39
clarkbso on a number line we have 1.1.dev1 < 1.1 < 1.222:39
clarkb^ is all I needed22:39
lifelessfungi: actually pbr is saying 'you don't need preversioning ever'22:40
lifelessfungi: it doesn't make any assumptions about what you're working towards. It makes the smallest version bump it can.22:40
lifelessfungi: based on your commits22:40
fungipbr is guessing at what your next tag will be22:40
lifelessfungi: you can choose to have a bigger gap22:40
clarkbsorry, 1.1.dev1 < 1.1 < 1.2.dev122:40
fungiand preversioning based on that22:40
lifelessfungi: no, its using semver to know what your next tag *must be no lower than*22:41
mordredfungi: pbr is working towards what the next logical tag would be not caring if you later decide to make a larger jump22:41
lifelessfungi: this is really quite a big conceptual difference.22:41
lifelessfungi: preversioning - some arbitrarily huge jump.22:41
*** melwitt has quit IRC22:41
fungimy point was, i _personally_ like things normal humans sort without having to know that dev means you sort before (a la ~ in dpkg)22:41
lifelessfungi: pbr - you must increment by *no less than Y*22:41
mordredso, if you have 1.1, it'll make a 1.2.dev1 - and then if you decide the next release is going to be 2.0 at some point, it still sorts fine22:41
*** dangers is now known as dangers_away22:41
*** melwitt has joined #openstack-infra22:41
fungiwhereas post-versioning means i just get x commits since my last tag, appended to the version of the last tag. done. it's easy to remember conceptually, easy to sort on a number line22:42
mordredfungi: totally understand22:42
lifelessfungi: ok that I get.22:42
lifelessclarkb: yes, 1.1.dev1 < 1.1 < 1.2.dev122:43
lifelessclarkb: the issues 4 months back were with22:43
fungii'm not protesting the default behavior in pbr because it seems to be the versioning scheme that openstack has enshrined for some reason i'm unable to fathom22:43
mordredfungi: you and me both22:43
lifeless 1.1.dev1 < 1.1.rc1 < 1.1.rc1.dev1 < 1.122:43
lifelessclarkb: and pbr wasn't generating 1.1.rc1.dev1 ever, but instead going back to 1.1.dev1 at the first commit after 1.1.rc122:44
lifelessclarkb: that was a deliberate choice we made, which broke when pep-440 shifted under us.22:44
clarkblifeless: yes which is why I initially asked if maybe using tag + distance would be better here22:44
clarkblifeless: because tag + distance is more readable and more closely follows the 1.1.rc1.dev122:44
lifelessclarkb: the fix a month or so back to pbr master was to undo that deliberate choice and use the pep-440 blessed numbers22:44
clarkbbut I understand now there is a third factor which is distance'22:45
*** imcsk8 has joined #openstack-infra22:45
lifelessclarkb: given a tag of 1.1.rc1 and a distance of 1, what version are you proposing that pbr would create?22:45
clarkbthe modficaton to tag to accomodate distance22:45
clarkblifeless: 1.1.rc2.dev122:45
lifelessok, so my number above were bad :) . yes, that the number that pbr will create with my patch22:45
lifelessit creates it without my patch for postversioning22:46
lifelesspreversioning is a special case and had a bug.22:46
lifelessthe intent was it create 1.1.rc2.dev122:46
clarkbyes I think your patch is sound, I am saying I think there is a better way to represent the modifcations here so that the code is readable and we don't have bugs like this22:46
*** bknudson has quit IRC22:46
*** gordc has quit IRC22:46
clarkbtag + distance' + distance22:47
lifelesswhats distance' ?22:47
clarkblifeless: the bump to accomodate distance22:47
clarkblifeless: since we are never using post22:47
lifelessthats new_version, up about 10 lines out of the diff context22:47
clarkbin the above its 1.1.rc1 + distance' + dev$distance22:47
*** Swami has joined #openstack-infra22:47
fungii'm interpreting +distance' there as a d()22:48
lifelessclarkb: and you're missing the 'scan git for sem-ver: pseudo headers' in your rephrasing of things22:48
fungid(1.1.rc1) = 1.1.rc222:48
clarkblifeless: no distance' can deal with the sem-ver22:48
clarkbfungi: ya its basically the derivative of raw distance to determine offset from tag22:49
lifelessclarkb: I'd like to land the minimal patch. What you're describing makes no sense to my head atm - I don't understand it22:49
fungioh, as i "i've declared a an api break, bump me harder"22:49
lifelessclarkb: so if we want to refactor the code to make more sense to more people, thats fine, but its certainly not a minor tweak to fix an rc bug22:49
lifeless(and I support making it make more sense to more people)22:50
mordredmore sense == ++22:50
clarkblifeless: basically distance determines two things. It determines the N in devN and it determines what tag should be incremented to22:50
fungimaking sense to people is something it seems to not be doing currently ;)22:50
lifelessclarkb: no it doesn't22:50
clarkblifeless: yes it does22:50
lifelessclarkb: thats not how the code uses distance today, at all all. It uses it for just one thing - the N in devN.22:50
lifelessclarkb: no, it doesn't.22:51
clarkblifeless: yes I know the current code doesn't hence my suggestion that maybe this is easier without the crazy special casing22:51
*** dboik has quit IRC22:51
clarkblifeless: and instead we calculate 3 values, and combine them always22:51
lifelessclarkb: what22:51
clarkbyou have your tagged version, the amount to increment that version by and devN22:51
clarkbthis is true for post and pre versioning22:51
*** dboik has joined #openstack-infra22:51
clarkbso rather than special case aroudn it, take the tuple and emit a new version22:52
fungithe amount to increment that version by is always 1 yeah? (excepting the declared api break case?)22:52
lifelessclarkb: so thats last-semver, increment, and distance22:52
clarkbfungi: I think its 0 for preversioning and >=1 for post versioning22:52
lifelessclarkb: so new_dev = new_version.to_dev(distance) is exactly what you're asking for22:52
lifelessclarkb: its not special cased22:52
lifeless10:52 < clarkb> fungi: I think its 0 for preversioning and >=1 for post versioning22:53
lifeless^ I don't understand what that can be in reference to22:53
*** ZZelle_ has quit IRC22:53
lifelessclarkb: if the 'its' there refers to distance, then no. distance is only and always the number of commits back to the tag.22:53
clarkblifeless: 1.1.rc1 + one commit in post versioning is 1.1.rc2.dev1, in preversioning its 1.1.rc1.dev122:53
lifeless1.1.rc1 + 1 commit22:54
lifelesspost versioning -> 1.1.rc2.dev122:54
lifelesspre_versioning if the target version is 1.1 -> 1.1.rc2.dev122:54
lifelesspre_versioning if the target version is (say) 1.2 -> 1.2.dev122:54
lifeless1.1.rc1.dev1 would be going backwards22:55
lifelessand is the bug this patch fixes22:55
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Story Search Results now show loading spinner.
clarkbhuh why distinguish then? is it purely because we want a target?22:55
fungihrm... my understanding of post versioning was yet something else entirely. i thought those were both pre-versioning22:55
clarkb(I am noting that your two versions are identical)22:55
lifelesstarget_version is not None <- this is what determines pre_versioning22:55
lifelessclarkb: they are yes.22:56
lifelessclarkb: why distinguish what?22:56
clarkblifeless: pre vs post if the version comes out the same either way22:56
fungithose seem like both pre-versioning, one with a target version and one without22:56
lifelessclarkb: because preversioning when the preversion is *higher* lifts teh version.22:56
clarkblifeless: ya but you can do that easier with a tag22:57
lifelessclarkb: yes which is why I say that preversioning can be deprecated22:57
lifelessclarkb: there's no point and it can lead to bugs because its harder to reason about.22:57
lifelessclarkb: but until we get 0.11 out the door, we don't have the generation of versions needed to allow us to deprecate it22:57
clarkbfungi: basicall new world order is preversion everything against the calculate target version22:58
lifelessbecause 0.10 doesn't bump the version ever22:58
clarkbfungi: calculated target may come from a hard coded version or from a tag + 122:58
lifelesstarget_version == None <- postversioning22:58
*** rm_work is now known as rm_work|away22:58
lifelesstarget_version == something <- preversion22:58
lifelessclarkb: there is no calculated target for postversioning22:58
fungiclearly there needs to be yet another name for the sort of versioning i like, which none of this is. where is the "i released 1.1.0 and have 3 commits since then so call my local version 1.1.0.something3?"22:58
clarkblifeless: it does bump the version but only by using post22:59
fungithat's what i thought we used to refer to as postversioning22:59
clarkblifeless: it doesn't bump in bigger steps22:59
lifelessclarkb: it does the minimum bump yes.22:59
clarkbfungi: thats what 0.10 does22:59
clarkbfungi: so 0.10 does proper postversioning, master does preversion everything22:59
lifelessfungi: postversioning in the docs was 'calculates a new version for you', preversioning was 'counts down to your desired version'22:59
fungii think i'll call it "versioning" to distinguish if from these other types22:59
lifelessuntil the pep-440-gate, 0.10's postversioning was entirely broken. the change to use .post made it sort-order compatible23:00
lifelessbut not semantics compatible23:00
clarkblifeless: I am still not sure I follow on that but ok23:00
clarkblifeless: the pep clearly says both dev and post are public version23:00
lifelessfungi: we could add a mode to use local versions for what you want23:00
lifelessclarkb: they are23:00
lifelessclarkb: they defined meaning for postN is 'minor errors in a final release'23:01
fungii don't understand why "3rd commit since 1.1.0" isn't semantically compatible23:01
lifeless"The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release."23:01
fungii guess for projects with multiple branches which have the 1.1.0 tag in their history, 1.1.0.post3 is vague. was that the main concern with them?23:02
clarkbwe never use post releases to publish anything23:02
*** camunoz_gone is now known as camunoz23:02
lifelessfungi: thats a big part of it23:02
lifelessone of the use cases we were solving was generating public versions for folk doing CD23:03
lifelessincluding building into rpms etc23:03
*** patrickeast has quit IRC23:03
lifelesspostN versions that are defined as not affecting the distributed software aren't suitable for that23:03
*** tsg has joined #openstack-infra23:03
*** mase_x200 has joined #openstack-infra23:03
lifelessoh yay23:03
lifeless2015-04-29 21:59:00.241 | E: Unable to locate package libxslt-dev23:03
lifeless2015-04-29 21:59:00.242 | E: Package 'pkg-config' has no installation candidate23:03
lifeless2015-04-29 21:59:00.257 | E: Package 'libsqlite3-dev' has no installation candidate23:04
clarkblifeless: thats what devstack was doing for you23:04
clarkbso, basically pep440 has broken us in another way23:04
clarkband this is pbr finally dealing with it23:04
clarkbI can deal with that, its just annoying23:04
mordredclarkb: yah23:04
bswartzcould someone do me a favor and take another look at this patch tonight?
lifelessclarkb: ok, so we need to put the devstack call back ?23:04
bswartzwe have some work that depends on the new repo getting created23:04
fungii mean, with pep440-semver pbr and no target_version the versioning on two separate branches each with the same most recent tag is equally vague/conflicted, so it doesn't necessarily help, but gives you some additional avenues you can leverage to make it less vague by declaring target_version on your branches23:04
clarkblifeless: or otherwise install the C libs that you need to install ppypi package23:04
lifelessclarkb: huh? thats apt failing23:05
*** signed8bit has quit IRC23:05
lifelessclarkb: that I'm pointing at23:05
clarkblifeless: oh for some reason I read it as pipfailure23:05
lifelessclarkb: nah-uh23:05
clarkblifeless: I do not know then, maybe mirror exploded23:05
jogopypi mirror issues?23:05
clarkbthey do that23:05
clarkbjogo: no apt mirror23:05
lifelessso I think I can make this code a teeny bit cleaner23:05
lifelessnow we've gotten past the angst23:06
jogoclarkb: see the link I shaerd23:06
clarkbit occurs to me that pep440 doesn't seem to have addressed at all what you should do for commits beyond a release23:06
clarkbpbr is probably the only thing that will exist in the python world that addresses it, and it does so by making you use git and follow commmit rules23:06
clarkbwhich isn't good for the other half of python using hg23:06
lifelessclarkb: would you prefer ?23:07
*** tqtran_ has joined #openstack-infra23:07
*** tqtran_ is now known as tqtran23:07
lifelessclarkb: so there are other vcs-inferring setuptools plugins out there23:07
*** dalgaaf has quit IRC23:07
clarkbjogo: thats an hpcloud networking error23:08
clarkbjogo: no route to host23:08
fungilifeless: on that failure log, did we maybe previously rely on devstack to apt-get update?23:08
jogoclarkb: ahh thanks23:08
*** dannywilson has quit IRC23:08
*** ildikov has quit IRC23:08
clarkblifeless: ya I think thats a bit clearer23:08
fungii don't see any apt-get update before the apt-get install (including in the setup logs)23:08
lifelessclarkb: pep-440's basic answer is a) use dev versions, and/or b) use local versions23:08
lifelessok, so should we put the devstack call back for now ?23:09
*** patrickeast has joined #openstack-infra23:09
clarkblifeless: but dev versions always sort lower than the version they append23:09
lifelessor should I include an apt-get update call in my patch ?23:09
clarkblifeless: so you basically need machinery to do what pbr does and prediect what it will be23:09
lifelessclarkb: yes23:09
clarkblifeless: yes, I think you may need an apt-get update because things may have gotten unsycned23:09
lifelessclarkb: for your chosen version scheme. Since pep-440 supports three or so schemes.23:09
*** bhunter71 has joined #openstack-infra23:09
fungilifeless: include apt-get update in the integration test script23:10
fungiin pbr23:10
fungishould be self-testing at this point23:11
openstackgerritlifeless proposed openstack-dev/pbr: Issue #1450210: Preversioning rc tag handling
*** dannywilson has joined #openstack-infra23:11
lifelessclarkb: there is c) add a nother point to your versions and use that. x.y.z.N23:11
lifelessclarkb: we can review a few of those if we wish for future pbr enhancements23:12
lifelessright now though I just want to get a years worth of stuff sitting pending released and in use23:12
clarkblifeless: but c) isn't semver :)23:13
* mordred punches semver in the mouth23:13
lifelessso its not pep-440 purely giving us pause23:13
lifelessits the interaction of a few things23:13
lifelessmultiple folk's branches23:13
lifelesspublic versions23:13
lifeless(blame mordred)23:13
lifelessthe nasty preversion or not-preversion stuff which is now a lot cleaner23:14
*** Sukhdev has quit IRC23:14
lifelessthe's now one try:except and that if target_version is None if-block around it23:14
*** kmartin_ has quit IRC23:17
*** viglesias has quit IRC23:17
*** kmartin has joined #openstack-infra23:17
*** ivar-lazzaro has quit IRC23:18
*** ivar-lazzaro has joined #openstack-infra23:19
*** ivar-lazzaro has quit IRC23:20
*** baoli has quit IRC23:20
*** viglesias has joined #openstack-infra23:20
*** ivar-lazzaro has joined #openstack-infra23:20
*** Rockyg has joined #openstack-infra23:20
*** alexsyip has joined #openstack-infra23:20
*** ildikov has joined #openstack-infra23:22
cineramapleia2: oh hey i bet you want an update on how things are going23:23
*** sarob_ has joined #openstack-infra23:24
pleia2cinerama: whenever you have one :)23:24
cineramapleia2: so the code to generate the zanata.xml is done. it's a little dodgy :)23:24
pleia2and that's ok, that's why we have good code review tooling :)23:24
cineramapleia2: the best way to noninteractively get a zanata.xml with a full list of enabled locales was to just use the server REST API23:25
*** packet has quit IRC23:25
cineramapleia2: which is also how we're going to get our percentage completion stats to use when we go to pull files23:26
openstackgerritlifeless proposed openstack-dev/pbr: Issue #1450210: Preversioning rc tag handling
lifelessclarkb: sadly the pastebin doesn't work23:26
pleia2cinerama: so are these both things that the client polls for?23:26
lifelessclarkb: because devs are complex, so I've reverted to the thing that worked23:26
cineramapleia2: so i'm working on the pull now. then there will be modified proposal scripts that use a mix of the existing bash functions we source in together with our new stuff23:26
pleia2cinerama: gotcha23:27
fungisubclassing optparse.OptionParser is turning out to be gnarly. not only does it want to call sys.exit() but it also wants to write directly to sys.stdout. i think i just need to punt on an end-to-end test and simply test that the get_version() function isn't broken23:27
cineramapleia2: so if we use the client itself we can get the statistics in a somewhat fiddlier format. we can't easily get a base zanata.xml out of it as the "init" subcommand is interactive only and i didn't really feel like doing the expect thing23:28
pleia2cinerama: ah, right23:28
clarkblifeless: ok23:29
pleia2cinerama: I'm thinking lots of inline comments for these scripts? :)23:29
cineramapleia2: so we need the xml to actually do anything, that requirement's not going away but i do wish they supported the percentage thing as we have in the transifex client. but we can work around it as i said23:30
lifelessclarkb: (because expressing the interactions inline in that function would retard comprehension even more)23:30
* pleia2 nods23:30
*** jtriley has quit IRC23:30
cineramapleia2: that's probably a good idea. comments are usually an artifact of my development process but also there are things here which are obvious to me that are not going to be obvious to someone who hasn't been kicking this client around23:31
lifelessfungi: can you pastebin your patch ?23:31
pleia2cinerama: yeah, and they will help us to identify sections of the code we can rework if the zanata devs can make the changes we need (like for percentage pain)23:31
lifelessfungi: since pbr has a helper for this anyhow..23:31
fungilifeless: sure, just a sec23:31
lifeless is looking happier23:32
openstackgerritJeremy Stanley proposed openstack-infra/bindep: Add a --version option
fungilifeless: that's ^ where i'm at so far with trying to subclass it23:32
cineramapleia2: yep. bear in mind they have a variety of clients to pick from so hopefully that rest is going to be pretty stable23:33
*** melwitt has quit IRC23:33
pleia2cinerama: yes, one hopes :)23:34
fungilifeless: i'm thinking i also need to subclass optparse.OptionParser.print_version() to not write to stdout, but am having trouble figuring out how to feed that back into the logger23:34
fungier, not subclass but shadow23:34
lifelessfungi: no, gotta decide what to test.23:35
lifelessfungi: my main concern is that you have logic about versions in there at all23:35
fungilifeless: is the current wisdom that we should import pbr and make it a run-time dependency?23:36
lifelessfungi: its what we've been doing in many places23:36
fungilifeless: i can switch to that easily enough, but it doesn't solve my immediate dilemma about whether and how to test optparse's magic --version handler23:37
lifelessfungi: there's some panic about performance23:37
lifelessfungi: don't test it, valueless.23:37
fungiwe've definitely had instances in the past where --version didn't work in utilities, because it was untested23:38
lifelesswe could even push the "%prog " thing back down23:38
lifelessfungi: sure, but it will be evaluated23:38
lifelessfungi: any test of the CLI will invoke the version code with this approach23:38
fungilifeless: fair enough. i guess we'll generate the version on every invocation whether we display it or not in this case23:39
fungiso it's a safe assumption to make23:39
lifelessit might not be a good version23:39
lifelessbut it will be one23:39
*** marun has quit IRC23:39
fungiokay, i'll consider it an afternoon of sisyphean effort put to good use familiarizing myself with testing conventions and take the pragmatic route ;)23:40
*** wenlock_ has quit IRC23:40
*** emagana has quit IRC23:40
lifelessmordred: <- any thoughts?23:41
openstackLaunchpad bug 1447666 in PBR "PBR doesn't support builder option for build_sphinx command" [Undecided,New]23:41
mordredlifeless: I mean, "thoughts is a strong word23:41
*** mase_x200 has quit IRC23:41
lifelessmordred: I just haven't looked at that stuff at a ll23:41
mordredlifeless: oh - bother - I mean, yeah - we should probably not override command line options if one is given23:42
mordredlifeless: we should provide an option if one _isn't_ given23:42
mordredso I agree it's a bug23:42
mordredI'm not sure how much I care this instant23:42
*** wenlock has quit IRC23:42
*** tnovacik has quit IRC23:44
lifelessmordred: if you could say that - 'yes its a bug, right fix would be to honour the supplied command line' thats enough to move it to traiged  yada yada yada23:45
lifelessmordred: or I can23:45
openstackgerritAlexander Yip proposed openstack-infra/project-config: Add congress.yaml file
mordredlifeless: got it23:46
alexsyipHi, I’m a newbie trying to enable some tempest tests for congress in jenkins.23:46
alexsyipI just pushed a change to gerrit
mordredlifeless: I cannot set triaged or importance23:47
alexsyipI’m wondering if that change looks right for turning on our tempest tests23:47
*** dims has quit IRC23:50
*** chlong has joined #openstack-infra23:50
*** ddieterly has joined #openstack-infra23:51
*** dannywil_ has joined #openstack-infra23:52
*** dannywilson has quit IRC23:55
alexsyipAlso, is there anyway to test this congress.yaml without merging it?23:55
lifelessmordred: wat23:56
lifelessmordred: LOL23:56
lifelessmordred: I have anyhow23:57
*** dannywil_ has quit IRC23:57
*** sputnik13 has quit IRC23:57
*** yamada-h has joined #openstack-infra23:58
*** HeOS has quit IRC23:58
openstackgerritMonty Taylor proposed openstack-dev/pbr: Unbreak the build_sphinx builder option
mordredlifeless: there ^^23:59
mordredlifeless: turns out the fix was easy23:59

Generated by 2.14.0 by Marius Gedminas - find it at!