Thursday, 2013-10-03

*** thomasm has joined #openstack-infra00:04
clarkbjog0: logs coming up00:06
jog0clarkb: so the bug isn't in classify00:09
openstackgerritClark Boylan proposed a change to openstack-infra/config: Increase log worker retry timeout length.
*** hogepodge has quit IRC00:10
clarkbjog0: do you understand what is going on?00:10
jog0clarkb: nope00:11
*** w_ has joined #openstack-infra00:12
*** olaph has quit IRC00:15
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add even more debug logs.
jog0this ^ may help shine some light?00:16
jog0gerritlib is logging it received the right information but Stream isn't working or something00:16
clarkbjog0: I will take a look once I have convinced myself the new backoff inthe gearman works is not broken00:18
jog0clarkb: cool00:19
jog0so the strange thing about e-r failing is when I run locally in non-bot mode it all works00:19
*** nati_ueno has quit IRC00:21
*** nati_ueno has joined #openstack-infra00:21
*** matsuhashi has joined #openstack-infra00:32
openstackgerritClark Boylan proposed a change to openstack-infra/config: Increase log worker retry timeout length.
clarkbI did break it >_> latest patchset should be good. I am applying it00:34
*** dkliban has joined #openstack-infra00:44
*** thomasm has quit IRC00:49
clarkbok logstash workers all kicked. Going to review the bot change and merge then kick the bot00:49
clarkbjog0: still around? you added a log.setlevel but aren't we configuring that in a file?00:50
openstackgerritA change was merged to openstack-infra/config: Make gate-hacking-pypy non-gating
openstackgerritA change was merged to openstack-infra/zuul: doc: point directly to zuul.conf section
openstackgerritA change was merged to openstack-infra/config: Retrieve logs > DEBUG level.
clarkbjog0: I -1'd with a comment. Let me know if I am crazy01:00
clarkbgoing to restart elastic-recheck now to see if it is any better with the new log workers01:01
*** nosnos has joined #openstack-infra01:04
*** oubiwann has joined #openstack-infra01:09
*** jaypipes has quit IRC01:11
*** dcramer_ has quit IRC01:24
*** melwitt has quit IRC01:25
*** ryanpetrello has quit IRC01:29
*** schwicht has quit IRC01:36
*** dhellman_ has joined #openstack-infra01:37
*** dhellman_ has quit IRC01:40
*** ryanpetrello has joined #openstack-infra01:41
*** esker has joined #openstack-infra01:42
*** ryanpetrello has quit IRC01:48
*** nati_ueno has quit IRC01:51
*** oubiwann has quit IRC01:56
clarkbjog0: I wonder if it is related to the elastic-recheck html/js page01:57
clarkbjog0: because that page appears to be updating maybe01:57
*** ryanpetrello has joined #openstack-infra01:57
dstufftmordred: ping02:12
*** dcramer_ has joined #openstack-infra02:18
clarkbjog0: doubling the test processes for tempest seems to help02:20
clarkbjog0: that change takes the tests to just under 30 minutes on hpcloud VMs02:20
clarkbjog0: and tempest didn't fail in new and exciting ways02:20
jeblairclarkb: i don't think the /srv/static/status/elastic-recheck/graph.json file is being updated, despite the cron job being run.  i'm not sure what the problem is.02:23
*** oubiwann has joined #openstack-infra02:24
*** oubiwann has quit IRC02:24
clarkbjeblair: jog0 has a change up to add more debug output, that should help us sort this, but it needs a small change02:24
*** oubiwann has joined #openstack-infra02:25
*** oubiwann has quit IRC02:25
jeblairclarkb: different thing i think.02:25
clarkbI see you ahve reviewed it02:25
clarkbjeblair: ah02:25
jeblairclarkb: which uses
clarkbI see. I will have to look at all of this stuff much more closely tomorrow. Very excited about the elastic-recheck status02:27
clarkbjeblair: would it maybe be more beneficial to have elastic-recheck output to graphite?02:27
clarkbrather than using a cron?02:27
jeblairclarkb: since we're interested in data about bugs, which are transient, i don't think it's a great match (we'll end up with lots of whisper files for bugs we don't care about)02:28
jeblairi mean, ideally they are transient.  :)02:28
clarkbjeblair: gotcha02:28
jeblaireg, if we _do_ care about how much we've seen a bug over the past 3 years, we're doomed.  doomed.02:29
*** erfanian has joined #openstack-infra02:30
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add /usr/local/bin to recheck graph cron path
jeblairclarkb: that's it ^02:34
jeblairgrumble puppet cron path grumble02:35
clarkbone thing about that query timeframe is we will need to be careful about not adding really expensive queries to the list (so far so good though)02:38
clarkbit really helps that we restrict queries to specific files02:38
clarkbalso the gate tests seem to be a lot better today02:38
jeblairclarkb: yeah i just timed it at 6 seconds for all queries (i've seen it take 20 or 30 seconds though)02:40
jeblairclarkb: but that's also part of the reason it's on a 15 min cron (i mean, we _could_ actually do it all in realtime, but that would be silly)02:41
clarkbI am super excited about all of the stuff happening around logstash/elasticsearch. I was secretly hoping that it would pick up traction with other people reducing the amount of work I need to do :) I do need to upgrade the whole system sometime soon though. that will be fun02:43
jesusaurusclarkb: yeah, you need to get es up to a 0.90.x version02:44
clarkbjesusaurus: yup, it is on the list right after spinning up a new etherpad02:44
jesusaurusclarkb: are you going to worry about reindexing old data into the new system? if so i think i have a python wrapper around the reindex plugin somewhere02:46
clarkbneed to test a new etherpad with cloud db before the summit02:46
clarkbjesusaurus: I am going to do an in place upgrade which converts the indexes I think and if that fails I will discard the old stuff02:46
clarkbjesusaurus: I can go in a forward direction with the data but not backward02:47
clarkbjesusaurus: logstash schema changes though, so I may not worry about it02:47
clarkbthey have flattened fields into the global keyspace02:48
jesusaurusoh, right02:48
*** oubiwann has joined #openstack-infra02:48
*** oubiwann has quit IRC02:48
clarkb(which I think is a good thing otherwise I would try to keep the old schema)02:48
jesusaurusi forgot about the logstash upgrade part02:48
clarkbjesusaurus: though now that we are dealing with much less volume (thank you sdague) I may switch to the http output rather than the normal output for es02:49
clarkbnot sure yet02:49
jesusaurusthe http output has been working well for me02:50
*** ArxCruz has quit IRC02:52
*** oubiwann has joined #openstack-infra02:54
*** jerryz has quit IRC03:04
*** dguitarbite has joined #openstack-infra03:08
*** markmcclain has quit IRC03:16
*** ryanpetrello has quit IRC03:20
*** ryanpetrello has joined #openstack-infra03:32
*** UtahDave has joined #openstack-infra03:36
*** vipul has quit IRC03:37
*** vipul has joined #openstack-infra03:37
*** CaptTofu has quit IRC03:51
*** CaptTofu has joined #openstack-infra03:51
openstackgerritKhai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module
*** UtahDave has quit IRC03:55
openstackgerritKhai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module
*** dguitarbite has quit IRC04:04
openstackgerritKhai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module
openstackgerritKhai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module
*** sarob has joined #openstack-infra04:17
*** mtreinish has quit IRC04:28
*** anteaya has quit IRC04:30
zaroanybody know why pip install git-review doesn't work?
zaroi get error: invalid command 'egg_info'04:36
*** AlexF has joined #openstack-infra04:40
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Create new gates for openstack-manuals
*** mtreinish has joined #openstack-infra04:42
*** asalkeld has joined #openstack-infra04:54
asalkeldhi. Heat has users that are keen to vote/ask for features. besides writing up blueprints what can they do?04:55
*** SergeyLukjanov has joined #openstack-infra04:56
asalkeldactually a vote on blueprints would work well04:56
clarkbasalkeld: two possible methods come to mind. users can vote on changes in gerrit (maybe not code implementing features but those that propose them) or you can use launchpad team polls05:03
asalkeldI'll look at team polls05:04
*** sarob has quit IRC05:12
*** sarob has joined #openstack-infra05:12
clarkbsurvey monkey or similar may be useful too. I didn't take the openstack user survey but maybe what they used?05:14
asalkeldyeah, that's right05:15
*** sarob has quit IRC05:16
*** AlexF has quit IRC05:22
*** AlexF has joined #openstack-infra05:34
*** davidhadas has quit IRC05:35
*** shardy is now known as shardy_afk05:37
*** AlexF has quit IRC05:38
*** jaypipes has joined #openstack-infra05:39
*** pblaho has joined #openstack-infra06:00
*** sergmelikyan has quit IRC06:02
*** yolanda has joined #openstack-infra06:08
*** tvb has joined #openstack-infra06:14
*** tvb has quit IRC06:14
*** tvb has joined #openstack-infra06:14
*** SergeyLukjanov has quit IRC06:17
*** rwsu is now known as rwsu-away06:21
*** davidhadas has joined #openstack-infra06:21
*** sarob has joined #openstack-infra06:23
ttxjeblair: nice (elastic-recheck)06:26
ttxjeblair: I would still leave the bug title visible though06:26
*** mkoderer_ is now known as mkoderer06:26
*** sarob has quit IRC06:28
*** SergeyLukjanov has joined #openstack-infra06:41
*** SergeyLukjanov has quit IRC06:42
*** odyssey4me has joined #openstack-infra06:48
ttxjeblair: (if it's meant as a replacement for the original page)06:50
*** ryanpetrello has quit IRC06:54
*** osanchez has joined #openstack-infra07:01
*** jcoufal has joined #openstack-infra07:08
*** afazekas has joined #openstack-infra07:11
*** Ryan_Lane has joined #openstack-infra07:14
*** shardy_afk has quit IRC07:16
*** flaper87|afk has quit IRC07:17
*** flaper87|afk has joined #openstack-infra07:18
*** AlexF has joined #openstack-infra07:18
*** flaper87|afk is now known as flaper8707:18
*** shardy_afk has joined #openstack-infra07:22
*** Ryan_Lane has quit IRC07:26
*** flaper87 has quit IRC07:30
*** flaper87 has joined #openstack-infra07:30
*** che-arne has quit IRC07:34
*** boris-42 has joined #openstack-infra07:38
*** dkliban has quit IRC07:41
*** osanchez has quit IRC07:43
*** mkerrin has quit IRC07:48
*** DinaBelova has joined #openstack-infra07:54
*** fbo_away is now known as fbo07:55
*** dizquierdo has joined #openstack-infra07:55
*** AlexF has quit IRC07:59
*** johnthetubaguy has joined #openstack-infra08:00
*** mrmartin has joined #openstack-infra08:01
*** davidhadas_ has joined #openstack-infra08:02
*** odyssey4me has quit IRC08:03
*** davidhadas has quit IRC08:04
*** SergeyLukjanov has joined #openstack-infra08:04
*** tkammer has joined #openstack-infra08:05
*** davidhadas_ has quit IRC08:05
*** davidhadas has joined #openstack-infra08:07
*** DinaBelova has quit IRC08:09
*** locke105 has joined #openstack-infra08:10
*** locke1051 has quit IRC08:10
*** odyssey4me has joined #openstack-infra08:11
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Bump hacking version in requirements
*** mkerrin has joined #openstack-infra08:15
*** yassine has joined #openstack-infra08:23
*** thomasbiege1 has joined #openstack-infra08:27
*** thomasbiege1 has quit IRC08:36
*** jpich has joined #openstack-infra08:38
*** shardy_afk is now known as shardy08:41
*** che-arne has joined #openstack-infra08:43
*** DinaBelova has joined #openstack-infra08:54
*** AlexF has joined #openstack-infra08:54
*** tkammer has quit IRC08:56
*** saper_ is now known as saper08:59
*** matsuhashi has quit IRC09:01
*** matsuhashi has joined #openstack-infra09:09
*** thomasbiege1 has joined #openstack-infra09:16
*** thomasbiege2 has joined #openstack-infra09:23
*** johnthetubaguy1 has joined #openstack-infra09:24
*** johnthetubaguy has quit IRC09:24
*** thomasbiege3 has joined #openstack-infra09:25
*** johnthetubaguy has joined #openstack-infra09:26
*** thomasbiege1 has quit IRC09:26
*** thomasbiege2 has quit IRC09:27
*** johnthetubaguy1 has quit IRC09:28
*** DinaBelova has quit IRC09:28
*** thomasbiege3 has quit IRC09:35
*** DinaBelova has joined #openstack-infra09:36
*** AlexF has quit IRC09:36
*** tvb has quit IRC09:36
*** davidhadas has quit IRC09:38
*** davidhadas_ has joined #openstack-infra09:38
*** tvb has joined #openstack-infra09:40
*** derekh has joined #openstack-infra09:44
*** DinaBelova has quit IRC09:45
*** AlexF has joined #openstack-infra10:00
*** tvb has quit IRC10:02
openstackgerritJulien Danjou proposed a change to openstack/requirements: Update sphinxcontrib-pecanwsme
openstackgerritSalvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack is launched with proper tenant isolation setting
*** DinaBelova has joined #openstack-infra10:07
*** AlexF has quit IRC10:12
*** boris-42 has quit IRC10:18
*** AlexF has joined #openstack-infra10:21
*** schwicht has joined #openstack-infra10:29
*** tvb has joined #openstack-infra10:33
*** tvb has joined #openstack-infra10:33
*** AlexF has quit IRC10:39
*** AlexF has joined #openstack-infra10:40
*** flaper87 is now known as flaper87|afk10:40
*** tvb has quit IRC10:41
*** tvb has joined #openstack-infra10:45
*** mrmartin has quit IRC10:55
*** nosnos has quit IRC10:56
*** matsuhashi has quit IRC10:57
*** weshay has joined #openstack-infra10:59
*** esker has quit IRC11:00
*** dprince has joined #openstack-infra11:00
*** esker has joined #openstack-infra11:00
*** esker has quit IRC11:05
*** DinaBelova has quit IRC11:06
*** AlexF has quit IRC11:07
*** markmc has joined #openstack-infra11:09
*** esker has joined #openstack-infra11:10
*** DinaBelova has joined #openstack-infra11:13
*** esker has quit IRC11:15
*** AlexF has joined #openstack-infra11:15
*** boris-42 has joined #openstack-infra11:15
*** tvb has quit IRC11:19
*** pblaho has quit IRC11:22
*** dprince has quit IRC11:24
*** adalbas has joined #openstack-infra11:25
*** CaptTofu has quit IRC11:27
*** CaptTofu has joined #openstack-infra11:27
*** che-arne has quit IRC11:30
*** esker has joined #openstack-infra11:31
*** giulivo has joined #openstack-infra11:32
giulivoguys, again I'm sorry but I'm not sure why this stable/grizzly is failing on the neutron vm... apparently even before getting to tempes11:32
giulivoany idea what is going wrong there?11:33
*** alexpilotti has joined #openstack-infra11:33
giulivoI see it "leaked file descriptor" again but seems related to jenkins not the actual job11:34
*** esker has quit IRC11:35
*** AlexF has quit IRC11:37
*** pblaho has joined #openstack-infra11:40
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added base_dir parameter to get_projects_info
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added the server parameter to get_changes
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Pep8 compliant
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added some optinos to openreviews
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added some options to openapproved
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added some options to reviewers
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added some options to reviews_for_bugs
*** pcm_ has joined #openstack-infra11:41
*** jpich has quit IRC11:44
*** jpich has joined #openstack-infra11:44
*** esker has joined #openstack-infra11:49
*** esker has quit IRC11:54
*** adalbas has quit IRC11:58
*** thomasm has joined #openstack-infra11:59
*** CaptTofu has quit IRC12:03
*** CaptTofu has joined #openstack-infra12:03
*** flaper87|afk is now known as flaper8712:04
*** CaptTofu has quit IRC12:08
*** dcramer_ has quit IRC12:08
*** asalkeld has left #openstack-infra12:09
*** hashar has joined #openstack-infra12:10
*** jcoufal_ has joined #openstack-infra12:13
*** jcoufal has quit IRC12:16
openstackgerritLucas Alvares Gomes proposed a change to openstack/requirements: Added upper version boundary for netaddr
*** hashar has quit IRC12:22
*** che-arne has joined #openstack-infra12:23
*** pblaho has quit IRC12:24
*** ArxCruz has joined #openstack-infra12:25
*** hashar has joined #openstack-infra12:26
openstackgerritLucas Alvares Gomes proposed a change to openstack/requirements: Added lower version boundary for netaddr
*** pblaho has joined #openstack-infra12:33
*** hashar has quit IRC12:35
EmilienMHi here, i'm currently investigating how could i implement in JJB. The relevant fact here is that add a top level XML value.12:36
*** tvb has joined #openstack-infra12:39
*** tvb has joined #openstack-infra12:39
fungiEmilienM: by "top level xml value" do you mean in the global config.xml but outside the plugins container?12:39
EmilienMfungi: hey, yeah. Actually <com.cloudbees.plugins.flow.BuildFlow plugin="build-flow-plugin@0.10"> replaces <poject>12:40
*** hashar has joined #openstack-infra12:41
hasharJJB is very active12:41
hasharI am wondering how it suddenly got attention12:41
EmilienMhashar: i'm wondering how could i bring buildflow support in jjb12:42
hasharnever used that plugin12:43
EmilienMXML regarding, it replaces <project> by <com.cloudbees.plugins.flow.BuildFlow plugin="build-flow-plugin@0.10">12:43
hasharI am using Zuul definition for the workflow :/12:43
*** CaptTofu has joined #openstack-infra12:44
EmilienMand "dsl" in "properties"12:44
EmilienMhashar: interesting. I guess this is already supported by JJB, right ?12:44
hasharnot really12:45
hasharI got a flow example with a job fetching a debian package sources to build a tarball12:45
hasharthe tar ball is then saved as an artifact12:45
hasharwhen complete, Zuul would trigger another job that fetch the artifact from the source job and build the package12:45
hasharbut you never know whether you are fetching the proper sources since another job can have generated another tar ball :/12:46
hasharanyway, my use case was merely to fail the change as soon as possible, for example when a PHP file is invalid12:46
hasharor skip the integration tests when the fast tests are failling12:46
hasharcan show you a rough demo tonight if you want12:47
EmilienMhashar: sounds like a plan we have :)12:47
EmilienMhashar: while this, i would like to start the work. Do you know if JJB is able now to detect if a plugin is installed or not ?12:48
hasharI know the XML configuration generated by Jenkins list the plugin version12:49
hasharbut haven't found an API entry point to list the plugin versions12:49
hasharmight need something upstream12:49
hasharwould be very nice to have JJB fail the job generation when a plugin is missing12:49
EmilienMhashar: ok, i'm working on that now. Let's talk irl about that later :)12:49
openstackgerritA change was merged to openstack-infra/config: Add /usr/local/bin to recheck graph cron path
*** flaper87 is now known as flaper87|afk12:52
*** dkranz has quit IRC12:54
*** rfolco has joined #openstack-infra12:54
*** psedlak|afk is now known as psedlak12:56
*** pblaho has quit IRC12:57
hasharEmilienM: /ci/pluginManager/api/json?pretty=true&depth=112:57
hasharEmilienM: list all the plugins with short / long names, their version , whether there is an update available and dependencies12:58
hasharI guess that should be added to python-jenkins bzr+ssh://
fungigiulivo: the "process leaked file descriptors" message is benign. seems like i see that at the end of every devstack test even when successful. looks like devstack setup died somewhere in the middle of executing quantum commands, so possibly something is still not 100% supported in the quantum->neutron compat wrapper? looking at some of the more detailed logs now12:59
EmilienMhashar: nice shot, thanks !13:00
giulivofungi, thanks for that, apparently there is a bug about
uvirtbotLaunchpad bug 1234181 in openstack-ci "stable/grizzly patches are failing jenkins in check-tempest-devstack-vm-neutron" [Undecided,New]13:00
hasharEmilienM: JJB already extend the Jenkins object, so you can probably hack something inside JJB13:00
*** dprince has joined #openstack-infra13:00
fungigiulivo: ahh, yes i thought maybe you were working on that bug. i should have pointed you to it13:00
hasharEmilienM: and we can submit the code back to upstream.  James / Monty used a similar approach to manages slaves in Jenkins.  They have their own myjenkins class in node pool.13:00
fungistill not sure why the bug is in openstack-ci13:01
EmilienMhashar: everything should be upstream :)13:01
*** matty_dubs|gone is now known as matty_dubs13:01
hasharpoor Old Gerrit13:02
fungihashar: did something break?13:02
*** oubiwann_ has joined #openstack-infra13:02
*** osanchez has joined #openstack-infra13:03
hasharfungi: na it works.  Just missing a few features :-]13:03
fungiokay, good13:04
* fungi gets back to getting things done for a change13:04
* hashar heads to some conf room13:04
*** hashar has quit IRC13:04
*** pblaho has joined #openstack-infra13:05
*** flaper87|afk is now known as flaper8713:06
*** davidhadas_ has quit IRC13:06
*** weshay has quit IRC13:08
*** davidhadas has joined #openstack-infra13:08
*** mriedem has joined #openstack-infra13:08
*** pblaho has quit IRC13:09
*** pblaho has joined #openstack-infra13:09
openstackgerritA change was merged to openstack-infra/config: Use twine to upload python package to pypi.
*** hashar has joined #openstack-infra13:14
*** michchap has quit IRC13:16
*** michchap_ has joined #openstack-infra13:16
*** fbo is now known as fbo_away13:16
*** ryanpetrello has joined #openstack-infra13:17
openstackgerritA change was merged to openstack-infra/config: Remove gate-django_openstack_auth-noop job.
*** esker has joined #openstack-infra13:22
*** julim has joined #openstack-infra13:23
*** julim has quit IRC13:23
*** julim has joined #openstack-infra13:25
*** fbo_away is now known as fbo13:25
*** davidhadas_ has joined #openstack-infra13:29
*** davidhadas has quit IRC13:31
*** pblaho has quit IRC13:31
*** oubiwann_ has quit IRC13:35
*** davidhadas has joined #openstack-infra13:40
*** changbl has quit IRC13:40
*** adalbas has joined #openstack-infra13:40
*** markmcclain has joined #openstack-infra13:40
*** davidhadas_ has quit IRC13:42
*** oubiwann has quit IRC13:49
*** oubiwann has joined #openstack-infra13:49
*** dkliban has joined #openstack-infra13:49
*** amotoki_ has joined #openstack-infra13:52
*** garyk has joined #openstack-infra13:54
garykmordred: ping13:54
garyksdague: ping13:55
mordredgaryk: sup?13:55
garykmordred: hi, ups and downs. you?13:55
garykmordred: i am trying to look into the stable/grizzly failures13:55
*** weshay has joined #openstack-infra13:56
garykseems like the problem occurs when the ovs external bridge is set to up13:56
garykany idea why the VM's now running on gating have 2 interfaces. previously they had one13:56
garyknot sure that this is related13:56
mordredgaryk: same. possibly occasionally too much wine13:57
*** prad has joined #openstack-infra13:57
garyki think the problem happens on (not 100% though).13:57
mordredI'm not sure - we've been running them on hpcloud the whole time - but perhaps hpcloud has grown a second interface13:57
sdaguegaryk: can you point to a fail?13:58
mriedemgaryk: ah, so thinking that causes the network to drop and that's why the devstack neutron gate job silently fails13:58
uvirtbotLaunchpad bug 1234181 in openstack-ci "stable/grizzly patches are failing jenkins in check-tempest-devstack-vm-neutron" [Undecided,Invalid]13:59
*** alcabrera has joined #openstack-infra13:59
mriedemthere are 4 stable/grizzly patches in there, they all fail on the same job13:59
mriedembut from what i see in the logs, they fail in different parts of the quantum network setup in devstack13:59
mriedemsometimes setting up a router, sometimes setting up a subnet, etc13:59
*** davidhadas_ has joined #openstack-infra14:00
garykmriedem: they are all around the same set of code - the l3 agent runs every interval so it is just updating the routing table etc following the config chnage14:00
sdaguegaryk: can you set -o xtrace in lib/neutron, should at least let the fail get explained better14:01
garyksdague: sure. i am also adding in debug info14:01
garykwill push a patch in 514:01
*** dkranz has joined #openstack-infra14:02
*** davidhadas has quit IRC14:03
*** amotoki has quit IRC14:03
*** amotoki_ is now known as amotoki14:04
*** hashar has quit IRC14:05
*** DennyZhang has joined #openstack-infra14:07
*** rwsu-away is now known as rwsu14:07
garykmriedem: sdague: i have pushed a patch with debug info. hopefully it will shed some light14:09
garykis there anyway we can get access to a terminal that is running with gating? that will help us solve the issue in less than no time?14:09
*** hashar has joined #openstack-infra14:10
sdaguegaryk: you need fungi to hold the guest14:11
sdaguethat being said, these programs should really be written for first failure discovery, and if neutron can't handle machines with 2 interfaces... that seems weird14:12
fungii need to see if the nodepool hold option is working as intended these days. i think it's functional now14:12
fungimight be easier if i just spun up a vm and ran it myself though, so i don't have to worry about it getting inadvertently scrapped14:13
*** sileht has quit IRC14:14
*** thedodd has joined #openstack-infra14:14
fungigaryk: worth noting, it's not hard at all to fire up a vm and run tests in exactly the same way. i went to extreme lengths to document the process in detail in the devstack-gate readme, because we do that regularly to vet embargoed security fixes14:14
*** hashar has quit IRC14:15
*** sileht has joined #openstack-infra14:16
garyksdague: i think that neutron does handles this. problem is the l3 agent starts to mess with the routing table14:16
garykit should be in a network namespace. i was unable to reproduce on a new and clean instance that i ran with the stable/grizzly code14:17
garykfungi: first and foremost thanks for the documentaion. it is brilliant14:17
amotokigaryk: ping14:17
garykamotoki: hi, whats up?14:18
amotokiam investigating stable grizzly issue.14:18
amotokido we need to use quantumclient branch for python-neutronclient?14:18
garykcool. have you been able to see anything14:18
*** DinaBelova has quit IRC14:18
fungigaryk: i'll give the tempest-neutron job a shot on grizzly branches and see what happens14:18
garykfungi: thanks14:18
amotokiexercise script does not work in stable/grizzly branch of devstack14:19
fungigaryk: and once it (inevitably) fails, i'll get you ssh access into it14:19
*** thedodd has quit IRC14:19
garykamotoki: sadly we have not even got there with the devstack run. the instance hangs after the subnet is create or a router is created14:19
garyki am able to use quantum <X> commands14:19
amotokigaryk: yes. i see that. it is very strange. i haven't identify it yet.14:21
*** thedodd has joined #openstack-infra14:21
sdaguegaryk: the instance hangs, or you just cut off all the network connectivity to it?14:21
garykamotoki: i think that when mark did the transition he added support for quantum/neutron14:22
*** tvb has quit IRC14:22
fungisdague: network access seems not to be cut off, because it still transfers logs to jenkins afterward14:23
amotokigaryk: thank. i need more investigation. let you know if i have a progress.14:25
*** oubiwann_ has joined #openstack-infra14:26
garykamotoki: thanks14:26
*** anteaya has joined #openstack-infra14:28
*** mrodden has joined #openstack-infra14:29
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Inspect the Gearman queue for immediate demand
mordredanteaya: morning14:38
anteayaI think my cold is finally gone, yay14:38
mordredanteaya: could you give me the owncloud url again?14:39
mordredanteaya: and would it possible for me to get admin access so that I could add a couple of accounts? I want to show work-in-progress at the board meeting14:39
* fungi will hopefully remember to dial into it for a change14:40
*** datsun180b has joined #openstack-infra14:40
anteayamordred: you are admin14:40
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user
anteayaexercise your powers, so that I know the setting is correct14:41
anteayafungi: in the list of priorities it is kinda low right now, but your feedback is always welcome14:42
fungianteaya: yeah, i'm hoping to dig myself free enough to poke at it, but may wait until after i hear mordred talk about it with the board14:42
anteayamakes sense14:43
anteayalet's hear what the board are wanting from it14:43
*** changbl has joined #openstack-infra14:43
dkranzmordred: Did your patch to run stable jobs against master client libs ever get merged?14:46
dkranzmordred: I seem to have lost the patch url for that.14:46
*** DinaBelova has joined #openstack-infra14:46
sdaguedkranz: yes, that's our default14:46
sdagueI'm pretty sure14:46
dkranzsdague: If so we should be able to close the blueprint about that14:47
dkranzjeblair: Thanks. So it has not merged.14:47
dkranzjeblair: Is there an issue with it, or just got lost in reviews?14:48
jeblairdkranz: see the comment; it requires this change
sdagueoh, this is the other piece of that, sorry got confused14:49
dkranzjeblair: That has multiple +2 but no +A14:49
jeblairdkranz: and actually, i think the first patch needs to be updated because of changes to the second.14:49
* fungi wanted to let mordred decide when/how it merged because of the interdependency between those changes14:50
mordredwhat did I do?14:51
mordredHis question was about standards and he was interested in how Swift and Cinder were compatible with ISO standards around storage?  He also wanted to know if there was a CDMI conflict.14:51
mordredtwo times in two days14:51
mordredspeaking of - does anyone here know the answer to that? I was about to ask people. :)14:51
fungimordred: time for a better mouse14:51
mordredfungi: it's my fault - I've started taking advantage of middle-click on url opens the url14:51
mordredit's a bad choice14:52
fungiahh, yup14:52
fungimiddle-click in your xterm pastes your secret conversations14:52
jeblairmordred: more like: middle-click performs some action; indeterminate as to which.14:52
mordredjeblair: and yes, the first patch needs to be updated to match the second14:52
mordredjeblair: it's almost like sabdfl designed middle-mouse-click14:53
jeblairmordred, fungi, dkranz: but i think the second patch is safe to merge now, without the first being updated yet.14:53
mordredjeblair, fungi: I belive the d-g patch is safe to land at any time - it should be a meaningless patch14:53
mordredjeblair: yes14:53
jeblairbut i'll still let mordred press the button on that.  :)14:53
mordred(by meaningless, I mean it should be a no-op until we land it14:53
dkranzmordred: Sure. I was just kicking this because it is a valuable test that is already implemented but not yet running.14:55
fungiokay, agreed14:55
dkranzfungi: And there is a blueprint about it assigned to me :)14:55
mordreddstufft: agree14:56
mordreddkranz: agree14:56
fungidkranz: clearly your fault when it fails then ;)14:56
mordreddkranz: and thank you14:56
dstufftmordred: lol I was like "what am I being agreed with?"14:56
mordreddkranz: we can't make it go live until stable/grizzly works again though :)14:56
dkranzmordred: Thank you for doing it! I'm sure it would have taken me longer.14:56
mordreddstufft: everything14:56
dkranzmordred: Ugh14:56
*** rcleere has joined #openstack-infra14:56
jeblairmordred: man, you're really mordreding your irc conversations this morning.14:56
dstufftmordred: btw can you do a mirror run against
*** mrodden has quit IRC14:57
dstufftto see if it works for you14:57
*** mrodden has joined #openstack-infra14:58
mordreddstufft: yes. yes I can.14:58
dstufftmordred: awesome thanks15:00
dstufftmordred: it's not behind a CDN or anything so it may be slower fwiw15:00
openstackgerritSalvatore Orlando proposed a change to openstack-infra/config: Rename to DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION
*** salv-orlando has joined #openstack-infra15:02
openstackgerritSalvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting
sdaguefungi / mordred / jeblair: and
sdagueI miffed actually getting tenant isolation turned on correctly for those neutron jobs15:06
sdaguesalv-orlando has the right fixes there if people could take a look15:06
jeblairsdague: second url isn't correct15:07
jeblairthat looks better15:07
sdaguesorry, it's the last 2 revs in channel above my comment as well :)15:07
sdagueI apparently am not as good with middle click, though not pasting foundation email in the process15:07
fungisdague: any particular order you want those merged in?15:08
fungialso mtreinish beat me to the typo in that first one15:08
*** boris-42 has quit IRC15:09
*** dstufft has quit IRC15:10
sdaguefungi: I think any order is fine15:10
mtreinishsdague: I think the 2nd one should also be reworked to handled like the other variables15:10
sdaguenothing happens until both are in15:10
jeblairsalv-orlando, sdague: there's a fix needed for each of those; but once they're updated, we should be able to merge config change first, and then when it lands, merge the d-g change15:10
jeblairsalv-orlando, sdague: that order makes it self-gating.15:11
jeblairfungi: ^15:11
*** flaper87 is now known as flaper87|afk15:11
sdaguejeblair: right15:11
*** rnirmal has joined #openstack-infra15:11
*** flaper87|afk is now known as flaper8715:11
* fungi nods15:12
*** dstufft has joined #openstack-infra15:12
openstackgerritSalvatore Orlando proposed a change to openstack-infra/config: Rename to DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION
dstufftmordred: oh yea, waiting for some feedback before making them "official"15:21
mordreddstufft: nice!15:22
*** johnthetubaguy has quit IRC15:22
dstufftthey include setuptools in them15:22
dstufftshould make things a bit nicer for windows users at least15:22
*** davidhadas_ has quit IRC15:24
openstackgerritA change was merged to openstack-infra/config: Rename to DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION
*** jcoufal_ has quit IRC15:25
amotokigaryk: still around?15:26
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Gate and check oslo.version against PyPy
*** johnthetubaguy has joined #openstack-infra15:26
fungigaryk: it definitely killed my inbound ssh when setting up neutron...
garykamotoki: yes15:28
amotokigaryk: it seems that quantum_create_initial_network seems succeed. q-svc log says router-gateway-set was called.15:29
garykfungi: yes, that is exactly the spot15:29
garykamotoki: please look at
amotokigaryk: it seems that quantum_setup_debug is called.15:29
garyklast command was quantum router-gateway-set15:30
fungigaryk: i can still ssh back into it. seems to have two ethX interfaces as previously asserted, though no idea if that has anything to do with what's going on...
openstackgerritJames E. Blair proposed a change to openstack-infra/elastic-recheck: Fix zero-fill logic in graphs
fungigaryk: this particular vm is an 8gb flavor on rackspace nova15:31
amotokigaryk: according to, after router-gateway-set, GET /networks and GET /subnets are called twice for each.15:32
garykfungi: thanks15:32
*** dizquierdo has left #openstack-infra15:32
garykamotoki: correct.15:32
*** tvb has joined #openstack-infra15:32
fungigaryk: if you want ssh access into it in the current (post-failure) state, i'm happy to oblige15:33
fungii'm eager to get things flowing in the grizzly gate again, since this puts us in a bad way for security backports until it's fixed15:34
garykin my local setup i did not set  Q_USE_DEBUG_COMMAND as True. I'll try that now15:34
garykfungi: amotoki has pointed out something that i missed on my local setup.15:34
fungiooh, excellent15:34
garykhopefully i'll be able to reproduce. restacking now15:34
amotokigaryk: we need to investigate quantum_setup_debug. the next of it is start_quantum_agent and there is no log of q-agt.15:35
*** tvb has quit IRC15:35
*** tvb has joined #openstack-infra15:35
*** tvb has quit IRC15:35
*** tvb has joined #openstack-infra15:35
amotokidevstack log is from screen log, so there is some time lag and somthing may not be logged. it is very trick....15:36
fungigaryk: for the record, i followed the "simulating devstack gate tests" section of and then substituted these shell commands at the end
garykamotoki: (i had print outs before this and I did not see them)15:36
fungioh, and used ZUUL_BRANCH=stable/grizzly of course15:37
garykamotoki: i owe you a beer in hong kong. when debug is true then it terminates15:37
fungioh, yikes15:37
amotokigaryk: :-)15:38
*** DinaBelova has quit IRC15:40
openstackgerritSalvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting
garykamotoki: i added in debug info and it passed. hmmm.15:43
garykamotoki: please see (setup_debug_quantum)15:44
garykwhen I add the echo "…" locally this passes (it failed without it).15:45
*** johnthetubaguy has quit IRC15:45
*** johnthetubaguy1 has joined #openstack-infra15:45
sdaguejeblair: so when trying to carve tempest off into another user, we get issues like this15:45
amotokigaryk: hmm.. am runnign too. but it is a great hint.15:46
sdagueyou cool with chowning the whole tempest dir?15:46
*** david-lyle has quit IRC15:46
sdagueor should we issolate further15:46
jeblairsdague: yes; does devstack need to write to it?  we may either need to give it permissions, or chown it after devstack runs and before tempest.15:47
sdagueyeh, I was going to chown just before we run tempest15:47
jeblairsdague: ++15:47
zaroanybody know why pip install git-review doesn't work?
zaroi get error: invalid command 'egg_info'15:49
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user
*** dkehn_ has joined #openstack-infra15:50
garykamotoki: + quantum-debug --os-tenant-name admin --os-username admin --os-password openstack probe-create --device-owner compute 72011347-4a72-4598-b3a2-c702b440226f15:50
garyk'ClientManager' object has no attribute 'quantum'15:50
*** dkehn has quit IRC15:51
amotokigaryk: i got the same. it is exactlly same when i ran exercise scripts after succeeds.15:51
*** yassine has quit IRC15:53
*** dkehn has joined #openstack-infra15:54
fungizaro: i can't reproduce that issue. what operating system/distribution are you trying it on?15:55
garykamotoki: looks like the quantum-debug command is redirected to the client instead of calling the executable15:55
garykmaybe we can just use the full path - but that is fugly15:55
*** dkehn_ has quit IRC15:56
fungizaro: ahh, nevermind. just noticed your paste url15:58
*** davidhadas has joined #openstack-infra15:59
zarofungi: i'm on ubuntu hp vm15:59
*** matty_dubs is now known as matty_dubs|lunch15:59
jeblairfungi: did you look into ?16:00
jeblairfungi: are there changes in gerrit for those repos?16:00
fungijeblair: yes, renaming fuel-ostf-tests to fuel-ostf and deleting fuel-ostf-plugin16:01
jeblairfungi: does fuel-ostf-plugin have changes?16:01
*** marun has quit IRC16:01
fungioh, i thought you meant did we need to make changes to their layout in gerrit16:01
*** marun has joined #openstack-infra16:02
fungii have not looked at the current state of those repositories, but they've separately asked me to blow away and reimport their contents16:02
fungibecause they asked to be added to stackforge but then kept developing on their old github repos instead16:02
jeblairthere is a change in the fuel-ostf-plugin repo16:03
jeblairso i don't think we should delete it16:03
*** Ryan_Lane has joined #openstack-infra16:04
*** DinaBelova has joined #openstack-infra16:05
jeblairthe only time we've deleted a project, to my knowledge, is due to a typo, and before it actually got used16:05
jeblairwe don't have a process for deleting a real, used project from gerrit16:05
fungiahh, okay. what's the best way to deprecate a project in that case?16:06
jeblairand we've never even decided whether that would even be desirable16:06
jeblairfurther, stackforge projects are intended to be self-service and not take away from time we spend working on openstack16:06
jeblairso i don't believe we should spend the time needed to come up with a policy and process for this case16:07
*** markmcclain has quit IRC16:07
fungigranted. i can tell them they'll have to live with a dead project there16:07
amotokiall, i have a question about stable/grizzly gate. which quantumclient/neutornclient should we use? quantumclient branch or master branch?16:07
garykamotoki: i spoke with mark and he has added -
amotokiwhen investigating stable/grizzly gate failure, quantum-debug command calls quantumclient. We need to fix either quantumclient or quantum-debug.16:08
fungijeblair: we probably need *much* stronger wording about projects needing to stabilize their organization and pause development to migrate to stackforge. we've had far too many jump the gun and ask to be added before they knew how they wanted to work, and then keep coming back to us asking to fix things which aren't/can't be made self-serve16:09
amotokithanks garyk.16:09
garyki am testing it now. i need to head on home i'll continue a bit later16:10
jeblairfungi: i have no problem with that.16:10
jeblairfungi:  look good?16:11
jeblairgah timezone url is wrong16:12
*** afazekas has quit IRC16:12
amotokigaryk: one more question. stable/grizzly gate uses master branch of neutronclient. Should we use quatnumclient branch?16:12
jeblairfungi correct url is
fungiahh, yeah, date was wrong16:12
*** AlexF has joined #openstack-infra16:13
fungiyup, lgtm other than that url16:13
*** garyk has quit IRC16:13
*** odyssey4me has quit IRC16:15
*** flaper87 is now known as flaper87|afk16:16
clarkbfungi ++++++++16:16
*** boris-42 has joined #openstack-infra16:16
*** flaper87|afk is now known as flaper8716:17
fungitop issues are, 1. imported cruft branches/tags which they want deleted, 2. kept merging changes outside gerrit and want to re-level the repositories because submitting changes for X00 commits would be too time consuming, 3. want to split/combine/delete projects16:18
*** AlexF has quit IRC16:20
fungiany others spring to mind? i'll work up a paragraph or two for the stackforge howto intro with strong cautionary wording16:20
jeblairfungi: i have found that you can join the board conf call without registering16:20
fungijeblair: i pressed #16:20
jeblairfungi: yep16:20
*** osanchez has quit IRC16:21
clarkbjeblair: fungi: does it still require webex?16:22
*** DinaBelova has quit IRC16:22
*** sarob has joined #openstack-infra16:22
fungiclarkb: depends on whether you want to see the death by powerpoint16:22
fungii used a telephone16:22
*** SergeyLukjanov has quit IRC16:22
jeblairme too16:22
fungiarchaic i know16:22
*** sarob has quit IRC16:23
*** sarob has joined #openstack-infra16:24
clarkb doesn't document a phone number? does the webex site provide that? in any case I didn't bother dialing in because the wiki said webex16:24
*** shardy is now known as shardy_afk16:24
jeblairclarkb: webex site has the #16:24
* clarkb notes this for next time though that will be in hong kong, maybe we can all listen in together16:25
jeblairi guess the budget document jbryce referred to was sent to the sekrit board list :(16:25
*** thingee_zzz is now known as thingee16:26
*** afazekas has joined #openstack-infra16:27
openstackgerritA change was merged to openstack-infra/config: Increase log worker retry timeout length.
jeblairthere's an #openstack-foundation channel if anyone feels like chatting about the board meeting16:30
fungiclarkb: if you get the invitation e-mails (i forget which ml they go to, maybe the public foundation one?) it says phone numbers and meeting code16:33
*** AlexF has joined #openstack-infra16:34
*** afazekas_ has joined #openstack-infra16:36
*** DennyZhang has quit IRC16:38
*** DennyZhang has joined #openstack-infra16:38
jog0the dates aren't the same for each graph16:38
jeblairjog0: (that will at least align the end-times to "now", though the start times will still be "first-hit".16:40
jog0jeblair: thats not ideal IMHO but much better16:40
clarkbfungi: thanks16:40
jeblairjog0: yeah, shouldn't be hard to align the start times too in a later change16:41
fungiclarkb: it's a fairly low-volume list and a good project-governance-oriented one to subscribe to if you don't already16:42
clarkbfungi: I don't think I am on it. The problem with all of these lists is between -dev, general, and -infra ETOOMUCHMAIL16:42
*** AlexF has quit IRC16:42
*** weshay has quit IRC16:43
fungii have actually started filtering -infra into a separate maildir so i can give it more priority, but i just dump all the rest of the openstack lists together into one and sift through them as time permits16:43
pleia2clarkb: if we could carve out some time today or tomorrow to chat about and confirm it's what we discussed in seattle I'd appreciate it (also if there are any glaring problems I need to fix)16:43
*** jpich has quit IRC16:44
anteayatime isn't permitting much these days16:45
clarkbpleia2: sure, catching up on mail now should have time shortly16:45
*** weshay has joined #openstack-infra16:48
openstackgerritAnita Kuno proposed a change to openstack-infra/puppet-dashboard: Changing the module to use the sodabrew fork of puppet-dashboard
*** Ryan_Lane has quit IRC16:50
*** dizquierdo has joined #openstack-infra16:51
*** alcabrera is now known as alcabrera|afk16:51
*** gyee has joined #openstack-infra16:53
*** hashar has joined #openstack-infra16:54
*** jerryz has joined #openstack-infra16:55
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Fix zuul status graph with failed change at head
clarkbwhat ever happened to the jenkins slowness a couple days ago?16:56
*** alcabrera|afk is now known as alcabrera16:57
anteayaclarkb: the periodic jobs running over night North American time seems to have played a role16:58
anteayanot sure if that has been addressed16:58
fungiclarkb: it magically cleared up with no indication16:58
*** mrodden1 has joined #openstack-infra16:58
fungianteaya: the time did not actually correlate to our periodic jobs16:58
fungicurrent baseless guess is that it's periodic jvm gc16:59
anteayayes you had mentioned that as a possibility17:00
anteayaa monthly run perhaps17:00
fungiuser cpu shot up to 80% solid and stayed there for about 14 hours straight, then quietly returned to normal17:00
*** odyssey4me has joined #openstack-infra17:00
* anteaya looks at cacti17:00
fungii only speculated maybe something monthly because it was the 1st of the month17:00
*** mrodden has quit IRC17:00
*** matty_dubs|lunch is now known as matty_dubs17:01
anteayait is a worthy speculation17:01
anteayaand it didn't occur in the last 24 hours17:01
*** jerryz has quit IRC17:02
clarkbjeblair: I have approved I am assuming that it will update properly on its own17:02
*** moted has quit IRC17:03
openstackgerritA change was merged to openstack-infra/elastic-recheck: Fix zero-fill logic in graphs
*** moted has joined #openstack-infra17:04
*** hashar has quit IRC17:04
*** odyssey4me2 has joined #openstack-infra17:04
*** odyssey4me has quit IRC17:05
*** oubiwann has quit IRC17:05
*** nati_ueno has joined #openstack-infra17:05
*** oubiwann has joined #openstack-infra17:06
*** garyk has joined #openstack-infra17:08
*** derekh has quit IRC17:11
jeblairclarkb: yeah, it shoud (vcsrepo)17:13
*** garyk has quit IRC17:13
mtreinishjeblair: I think it's better now17:15
*** markmcclain has joined #openstack-infra17:15
*** amotoki has quit IRC17:17
*** hashar has joined #openstack-infra17:18
*** garyk has joined #openstack-infra17:19
*** jerryz has joined #openstack-infra17:20
*** jerryz has quit IRC17:20
*** jerryz has joined #openstack-infra17:20
*** flaper87 is now known as flaper87|afk17:21
*** SergeyLukjanov has joined #openstack-infra17:23
*** hashar has quit IRC17:23
jog0clarkb: ping17:26
jog0how hard would it be to inject extra jobs into zuul now that dev is quite17:27
jog0and thus elasticsearch doesn't have a big dataset17:27
clarkbjog0: not terribly difficult depending on how you want to do it. We could just set up hourly periodic jobs initially17:30
clarkbjog0: its more complicated if you want to do something that uses available resources17:30
*** nati_ueno has quit IRC17:31
*** hogepodge has joined #openstack-infra17:31
*** markmcclain1 has joined #openstack-infra17:31
*** nati_ueno has joined #openstack-infra17:31
jog0clarkb: available resources would be preferred. as  don't need this if zuul is busy17:31
*** nati_ueno has quit IRC17:31
jog0clarkb: can you periodically check zuuls queue and base it on that17:32
*** markmcclain has quit IRC17:32
*** nati_ueno has joined #openstack-infra17:32
clarkbjog0: you can. jeblair is working on nodepool stuff that looks at demand to determine how to schedule pool builds. We should probably try to make it similar to that, but I haven't had a chance to look at it yet17:34
clarkbjeblair: ^ any thoughts on using available d-g slave resources to do opportunistic testing to run more tests during quiet times?17:34
jog0cool thanks17:35
*** ArxCruz has quit IRC17:35
clarkbpleia2: reviewed17:37
jeblairclarkb, jog0: i think there's a bug about it, but anyway, yeah, i think it should be a new trigger type in zuul to do that.17:39
pleia2clarkb: thanks! the current plan is to not run toci yet, we're just getting the framework up to run this script:
*** ArxCruz has joined #openstack-infra17:39
pleia2small steps :)17:39
pleia2once we have POC we'll go from there17:39
clarkbpleia2: yup17:41
jomarajeblair: hey, just a summary from yesterday17:42
*** johnthetubaguy1 has quit IRC17:42
*** AlexF has joined #openstack-infra17:42
jog0jeblair: awesome17:42
jog0with elastic-recheck etc we should get some great data from doing this17:42
jomarajeblair: i'm setting up openshift ask bot really quick as a stop gap, in the background we will work on puppet for askbot + postgres for openstack-infra - that's work that I'm supposed to do, right? and then we hand that off to you guys, probably described in a doc i havent read yet?17:43
*** melwitt has joined #openstack-infra17:43
pabelangerclarkb, fungi: For your puppetmaster, how to do manage client (node) certs?  Do you just back them up or actually store them in a repo?17:45
pabelangerlooking at launch-node, it looks like they are just accessed from the local puppetmaster17:45
fungipabelanger: right now we just live with the need to regenerate and distribute them in a disaster17:46
pabelangerfungi, okay, that's what I thought17:46
fungibut that said, we are looking at secure ways to back them up17:46
pabelangerTrying to come up with a plan for easy disaster recovery when the puppetmaster is lost17:47
pabelangerfigured I see what you guys were doing17:47
fungii don't recall whether we settled on a specific solution yet. my favorite was to encrypt the sensitive data and back that up with bup while excluding the plaintext locations for that data17:48
*** jergerber has joined #openstack-infra17:49
*** afazekas_ has quit IRC17:49
*** afazekas has quit IRC17:49
clarkbhub_cap: jeblair: do cloud dbs provide SSL connectivity to clients? I can't find anything indicating a cert is generated or how to provide a pregenerated cert17:49
hub_capnawr not yet17:50
hub_capclarkb: ^ ^ its somewhere on the list17:50
hub_capbut low i think17:50
clarkbhub_cap: not sure yet. But rackspace's network security certainly isn't trusted. I am not sure we want to rely on the internal network for data and credential security17:51
hub_capclarkb: rot13 dude17:51
clarkbmordred: fungi jeblair ^ thoughts on using cloud dbs in the clear via rackspace's internal network?17:52
pabelangerfungi, okay, will keep an ear out when you guys work it up again.17:52
fungiit does worry me, though for the moment that's only traversing rackspace's network17:53
*** che-arne has quit IRC17:53
clarkbfungi: re puppetmaster backups the idea to encrypt stuff we care about then bup that sounded good to me17:53
fungibut certainly dns hijacking would be a quick way to mitm it outside rackspace17:53
*** Ajaeger has joined #openstack-infra17:53
fungikeeping the db hostname hidden in hiera helps there17:54
mordredclarkb: reading17:54
*** qba73_ has joined #openstack-infra17:54
Ajaegerinfra team, could you review and approve, please?17:54
Ajaegerthe build will fail now since somebody was a bit to eager to remove the documentation instead of waiting for the patch above to go in.17:55
mordredhub_cap: are we saying that other customers at rackspace can snoop a network between our compute and db nodes and potentially steal traffic?17:56
fungiAjaeger: lgtm. approved17:56
clarkbmordred: I think it depends on how the internal network is implemented. Without direct control we can't assume it will always do what we expect (iirc this is why we run local firewalls on all machines)17:56
Ajaegerfungi and clarkb : you're ultraquick! Thanks a lot!17:56
fungimordred: in theory our db traffic should not be reflected to another vm17:57
clarkbbut what happens when someone cohabitant with us floods arp tables?17:57
clarkband so on17:57
fungii hope tricks like arp cache poisoning wouldn't work through the rackspace cloud at any rate17:57
openstackgerritA change was merged to openstack-infra/config: Removes openstack-api-programming doc build
fungibut yes17:57
fungiarp flooding too17:58
*** qba73_ has quit IRC17:58
openstackgerritKhai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module
hub_capmordred: i can find out, gimme ~1017:58
*** thomasbiege1 has joined #openstack-infra17:59
fungiin theory the vswitch should filter arp replies for ip addresses not assigned to the vm? not familiar enough with nova network or whatever's in use there17:59
clarkb(I am trying to be more paranoid about things :) )18:00
*** nati_ueno has quit IRC18:01
*** nati_ueno has joined #openstack-infra18:02
fungiyes, you make very good points18:02
fungias i said, it does worry me18:03
annegentle_pleia2: around?18:03
pleia2annegentle_: yep18:04
*** AlexF has quit IRC18:05
annegentle_pleia2: hey I see your codechix workshop handout is cc-by-sa, I'm copying some of the links, do you know if I have to use cc-by-sa for my copy or can I use cc-by?18:05
*** qba73_ has joined #openstack-infra18:05
*** AlexF has joined #openstack-infra18:05
annegentle_pleia2: I am not a lawyer but I do want to ask permission!18:06
pleia2annegentle_: I hereby grant you the permission to relicense :)18:06
annegentle_pleia2: I can show you what I have and you can see if you want attribution (and where to put it)?18:06
* jeblair goes afk for a few hours18:06
annegentle_pleia2: or I could just use cc-by-sa just like yours, but all the OpenStack docs are cc-by 3.018:06
annegentle_pleia2: woo thanks!18:07
pleia2annegentle_: I think it's all generic enough that it's not really neccesary to attribute18:07
annegentle_pleia2: yeah I'm with ya18:07
annegentle_pleia2: would you mind giving mine a once-over if you have a minute?18:07
pleia2annegentle_: sure thing18:07
*** AlexF has quit IRC18:07
zarofungi: 'related-bug' & 'partial-bug' git commit comments are not reported back to LP ?18:08
clarkbzaro: they should be18:08
clarkbzaro: do you have an example?18:08
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user
fungizaro: your first patchset lacked the header. comments are only added for the initial patchset so as to avoid spamming the bug subscribers18:10
fungisince the update_bug script does not scrape the bug comments to see if it already added a comment, nor does it keep a database to which it can refer18:11
* fungi needs to go grab an extremely late lunch now. bbiab18:11
zarofungi: ahh, but i always forget to add the bug info on my first patch.18:11
*** thomasbiege1 has quit IRC18:12
clarkbwe could probably have it query the bug to see if the existing change has been linked already (use lp as the state DB)18:12
*** AlexF has joined #openstack-infra18:13
* zaro enters bug for this matter18:13
*** w_ is now known as olaph18:15
clarkbzaro: thanks18:15
*** mrodden1 has quit IRC18:16
*** mrodden has joined #openstack-infra18:17
*** Ajaeger has quit IRC18:17
*** changbl has quit IRC18:18
*** markmcclain1 has quit IRC18:26
clarkbmordred: I figured protections are in place. But since we don't have visibility into them we tend to be extra cautious. This is why we run local firewalls for example. In this case it may be an acceptable risk. Just pointing it out there as I was looking for SSL connectivity as part of the etherpad move18:27
mordredI believe SSL connectivity with mysql is broken anyway, btw18:28
*** CaptTofu has quit IRC18:28
clarkbmordred: :(18:28
*** CaptTofu has joined #openstack-infra18:28
pleia2clarkb: re: first comment on so does the job template have to do with the target (ie rackspace vs hpcloud)? or something else?18:29
*** markmc has quit IRC18:29
pleia2pipelines, nodes, what :)18:29
clarkbpleia2: the templates that your template is based on exist so that different pipelines (eg gate, check) can run on different nodes18:30
clarkbpleia2: right now we want all gate tests to run on hpcloud for throughput reasons, but have started running check tests on both hpcloud and rackspace18:30
clarkbpleia2: in the case of tripleo there is only one provider so probably don't need distinctions like we have with the other tempest jobs18:31
pleia2ah, I see18:31
clarkb(all pipelines will run this test on the same cloud and node)18:31
pleia2so we do hope to have a 2nd provider at some point, but I don't think it's close18:32
clarkbpleia2: yeah, I think we can worry about that when it happens as we may want to distribute load differently than we do with hpcloud and rackspace for the other tests18:32
pleia2so how do I fix this?18:32
*** CaptTofu has quit IRC18:33
clarkbpleia2: for now I lean towards making it not a template and keeping it simple. change the job-template: string to job: and replace instances of {pipeline} with gate and instances of {node} with the label you have configured in nodepool.yaml.erb18:33
*** AlexF has quit IRC18:33
pleia2clarkb: ok, thanks18:33
hub_capclarkb: mordred: got some more info about the networks, and its public info...
hub_cap^ ^ cant spoof between vm's so u cant spoof between hosts or anything etc...18:34
uvirtbothub_cap: Error: "^" is not a valid command.18:34
hub_capi hate u u virt bot18:34
clarkbmordred: is it too late for that for havana? (I can approve but don't want to break RCs)18:35
*** oubiwann is now known as Guest795018:36
mordredclarkb: it should not matter18:38
mordredclarkb: it changes the version on the non-released dev versions18:38
*** Ryan_Lane has joined #openstack-infra18:38
*** CaptTofu has joined #openstack-infra18:39
clarkbmordred: cool. I gave it a +2 do you want me to approve it as well or should I let you do that?18:40
mordredclarkb: I'll do it - thanks18:41
*** sarob has quit IRC18:44
clarkbjog0: is the elastic-recheck bot stuck again? it has been very quiet18:51
*** julim has quit IRC18:51
clarkbbut we are also not running so many tests and we seem to not fail as frequently18:51
sdagueok, tempest user for tests is ready now -
clarkbsdague: it has been starred, will be reviewed after lunch18:53
sdagueclarkb: the gate queue is so light today18:53
clarkbsdague: iterested to know what you think about that change18:53
sdagueI think we need to start thinking about our analysis in percentages of runs18:54
clarkbsdague: yup we do. I also think jog0's idea to schedule jobs when we are quiet is a good one (to keep the numbers up)18:54
*** sarob has joined #openstack-infra18:59
*** garyk has quit IRC19:00
*** DennyZhang has quit IRC19:01
sdaguethat was my idea :)19:01
*** DinaBelova has joined #openstack-infra19:01
*** yolanda has quit IRC19:03
clarkbsdague: well it is a good one :)19:03
sdaguehonestly we may have had it independently, but mordred / jeblair and I were talking about it over the weekend19:03
sdaguethe "idle swarm"19:03
*** AlexF has joined #openstack-infra19:05
*** AlexF has quit IRC19:05
*** nati_ueno has quit IRC19:06
*** gyee has quit IRC19:06
*** sirushti has quit IRC19:07
*** nati_ueno has joined #openstack-infra19:07
*** dizquierdo has left #openstack-infra19:08
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add initial files for tripleo in the gate.
*** davidhadas has quit IRC19:11
*** mrodden1 has joined #openstack-infra19:12
*** davidhadas has joined #openstack-infra19:13
*** rfolco has quit IRC19:13
*** sirushti has joined #openstack-infra19:13
*** mrodden has quit IRC19:14
*** qba73_ has quit IRC19:21
*** marun has quit IRC19:21
openstackgerritA change was merged to openstack-infra/reviewstats: Added base_dir parameter to get_projects_info
*** odyssey4me has joined #openstack-infra19:27
openstackgerritA change was merged to openstack/requirements: Update sphinxcontrib-pecanwsme
*** odyssey4me2 has quit IRC19:29
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add initial files for tripleo in the gate.
lifelesscould someone add tripleo-core into the tuskar-core group and remove all the other members? likewise with the tuskar-ptl group ?19:32
openstackgerritA change was merged to openstack-infra/reviewstats: Added the server parameter to get_changes
*** marun has joined #openstack-infra19:32
openstackgerritA change was merged to openstack-infra/reviewstats: Pep8 compliant
*** vipul is now known as vipul-away19:35
*** _david_ has joined #openstack-infra19:35
*** odyssey4me has quit IRC19:36
openstackgerritA change was merged to openstack-infra/reviewstats: Added some optinos to openreviews
*** julim has joined #openstack-infra19:39
*** sarob has quit IRC19:40
*** marun has quit IRC19:41
*** DinaBelova has quit IRC19:41
*** yolanda has joined #openstack-infra19:42
*** vipul-away is now known as vipul19:44
*** vipul is now known as vipul-away19:44
*** _david_ has quit IRC19:45
openstackgerritlifeless proposed a change to openstack-infra/reviewstats: TripleO merged with Tuskar.
*** AlexF has joined #openstack-infra19:47
NobodyCamgood afternoon infra just a quick question if I may19:48
NobodyCamshould I be getting a forbidden on urls like :
lifelessI'd say so, the cfp is over  papers were selceted19:50
*** marun has joined #openstack-infra19:50
*** yolanda has quit IRC19:51
fungilifeless: i've added tripleo-core to tuskar-core but did not remove anyone. i'll leave that to you...,members19:52
fungialso there were no other tuskar groups (-ptl or otherwise)19:52
mgagnelifeless: is there a way to see all topics including ones that were rejected?19:53
lifelessfungi: cool, thanks19:54
lifelessmgagne: I don't know19:54
fungiis that what you're looking for?19:54
mgagnefungi: thanks!19:55
fungimgagne: not sure if you were looking for design sessions or conference presentations19:55
zulttx: ping still no cinder rc1?19:55
mgagnefungi: conferences ^^'19:56
openstackgerritPierre Rognant proposed a change to openstack-infra/jenkins-job-builder: Flow project support
sdaguejeblair: when you get a chance, that will let us have the functioning sudo based debug when things go funny19:57
fungimgagne: as a track chair i can see them, but they're behind a login so i'm not sure what i'm allowed to disclose (if anything)19:57
*** dprince has quit IRC19:58
*** AlexF has quit IRC19:59
fungimgagne: if you're really interested in a copy of all the abstracts i'm happy to ask the right people and find out what can be released20:00
mgagnefungi: I'm logged in and don't see the one I'm looking for. =)20:00
fungimgagne: oh, you were a track chair too? go to "browse presentations" in the chairs interface and then click on the track link and change the drop-down to "all categories" i think20:01
fungii don't see you on the track chairs directory though, so it might be restricted20:02
*** julim has quit IRC20:02
*** rnirmal has quit IRC20:02
funginote i'm talking about the interface20:02
mgagnefungi: ok, I'll look20:03
openstackgerritPierre Rognant proposed a change to openstack-infra/jenkins-job-builder: Flow project support
*** adalbas has quit IRC20:03
mgagnefungi: I don't have access once logged.20:03
fungiif you weren't chairing one of the summit tracks then i don't think it will let you see the list20:03
fungilike i said, i can ask around and find out if any of what's in there is secret or can be disclosed20:04
fungiif you need it20:04
mgagnefungi: more curiosity about one presentation. every links I found was linking to the vote system which is now over.20:05
*** rnirmal has joined #openstack-infra20:05
sdaguemgagne: these are design summit sessions, so no voting20:05
sdaguethe PTLs just accept what they want in their track20:05
fungisdague: i think he's looking for a list of the conference presentation abstracts instead20:06
mgagnefungi: yes20:06
sdagueoh, yeh that's not :)20:06
*** CaptTofu has quit IRC20:07
*** markmcclain has joined #openstack-infra20:07
*** resker has joined #openstack-infra20:07
*** CaptTofu has joined #openstack-infra20:07
*** SergeyLukjanov has quit IRC20:09
*** pcm_ has quit IRC20:09
*** esker has quit IRC20:11
*** resker has quit IRC20:11
jeblairfungi, mordred: are either of you interested in reviewing ?20:13
jeblairclarkb, mordred: ^20:13
jeblairfungi reviewed it20:13
*** gyee has joined #openstack-infra20:15
zaroclarkb: just wondering, would this be a good thing?
*** dkliban has quit IRC20:27
*** marun has quit IRC20:29
*** sarob has joined #openstack-infra20:30
*** jcoufal has joined #openstack-infra20:34
*** CaptTofu has quit IRC20:37
*** CaptTofu has joined #openstack-infra20:37
pleia2browsing through reviews, this one got stuck because the dependency is abandoned
pleia2still want, or should it be abandoned as well?20:39
pleia2(the other was abandoned due to timeout, rocketships!)20:40
pleia2so it looks like we still want these20:40
clarkbback from lunch /me reads scrollback20:40
openstackgerritA change was merged to openstack-infra/reviewstats: TripleO merged with Tuskar.
*** tvb has quit IRC20:41
openstackgerritA change was merged to openstack-infra/elastic-recheck: Cleanup tests
openstackgerritA change was merged to openstack-infra/elastic-recheck: Make real unit tests
clarkbjeblair: yes very interested. looking now20:42
clarkbzaro: yes, I think that is a good thing. We probably need to think through the security ramifications of clouddbs but in general being configurable is good20:43
clarkbjeblair: did you see the clouddb discussion? any input on that?20:43
clarkbpleia2: I think we need to think through that stuff a bit more20:43
zaroclarkb: cool.  just wanted to know if i should rebase my future changes on that.  thnx.20:43
clarkbpleia2: eg how to do it without potentially breaking stuff (moving to clouddbs and splitting DB config from service config is a good start)20:44
pleia2clarkb: ah, ok20:44
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Reorganize tests into unit and functional tests
openstackgerritA change was merged to openstack-infra/devstack-gate: run tempest tests under tempest user
mtreinishmordred, clarkb, jeblair: if you've got a sec can you take a look at: so we can start failing neutron runs again. :)20:49
jeblairclarkb: i'm okay but not thrilled with the arrangement.  perhaps we should ask hub_cap nicely for ssl?  :)20:49
clarkbit looks like my jeepyb delay for gerrit groups to be created change went into manage_projects but we haven't released a new version of gerritlib with the feature to support that20:49
clarkbare there plans to release a new gerritlib that I am not aware of?20:50
jeblairclarkb: i can cut one real quick20:51
clarkbjeblair: thanks20:51
jeblairclarkb: pushed20:52
hub_capjeblair: patches welcome ;) As much as id love to do it, itd kill my existing priorities for openstack compatibility... but if u want to convince hp to make it a priority, then it will be so :)20:52
jeblairokay, asking nicely clearly didn't work.  lesson learned.  :)20:53
*** vipul-away is now known as vipul20:54
hub_capjeblair: sry :(20:54
jeblairclarkb: can you aprv ?20:55
clarkbjeblair: ya reviewing20:55
clarkbjeblair: done20:56
jeblairmtreinish: looking; i want to make sure the most recent check ran with the variable set20:56
clarkbjeblair: gerritlib release may not have succeeded. I don't see it on pypi but jobs seem to have finished20:56
*** jpeeler has quit IRC20:56
clarkbtwine barfed20:56
jeblairclarkb: you mind looking into that?20:57
clarkb ya looking now20:57
*** rnirmal has quit IRC20:57
openstackgerritA change was merged to openstack-infra/config: Handle null in this function
clarkbdstufft: ^ I think twine is not python2.X compatible20:58
openstackgerritA change was merged to openstack-infra/config: Fix zuul status graph with failed change at head
clarkbdstufft: am I reading that correctly?20:58
dstufftclarkb: I accidently made it not compat on 2.6, I'm gonna be fixing that later tonight20:58
dstufftit should work fine on 2.7 though20:58
mtreinishjeblair: yeah the issue is the variable is only set on the neutron queue20:58
clarkbdstufft: it is importing urllib.parse which is python320:59
dstuffthmm, I thought that was guarded by an importerror20:59
dstufftclarkb: sorry i'm sick so i'm a little out of it20:59
dstufftclarkb: one minute20:59
jeblairmtreinish: eek, asymmetric..20:59
clarkbdstufft: no problem. I can poke at it myself, but figured I would ask to avoid duplicating work20:59
dstufftclarkb: oh I see what happened21:00
dstufftshould be urlparse not urllib21:00
dstufftclarkb: gimme a minute i'll fix this21:00
clarkbdstufft: ty21:00
dstufftclarkb: this is what I get for being "well this is short and I want it now so I won't write tests for it right now"21:01
dstufftguess I'll be writing tests tonight or tomorrow21:01
mtreinishjeblair: yeah it was just to unbreak the gate on the neutron runs. Enabling tenant isolation increased the number of requests a lot for a tempest run which triggered a race in neutron.21:01
mtreinishlook at the elastic recheck graph for 122400121:02
*** alcabrera has quit IRC21:02
mtreinishbut it's a real neutron bug so we're gating on neutron with it turned on21:02
mtreinishjust not for the other projects21:02
mtreinish(tenant isolation with neutron that is, tenant isolation is the default for tempest on every other run)21:03
jeblairmtreinish: right, but we're currently in a place where changes to d-g, devstack, and tempest could break gating for neutron21:03
*** chris613 has joined #openstack-infra21:03
mtreinishjeblair: that's sort of the intent21:03
*** johnthetubaguy has joined #openstack-infra21:04
jeblairmtreinish: ok, if i approve this and there's a structural problem with it, will you be around to fix it? :)21:04
jeblairmtreinish: (i don't like making untested changes to devstack-gate, which is what this is if the job that uses it isn't in the d-g job set)21:04
mtreinishjeblair: I should be21:04
mtreinishwhere else would I be :)21:05
jeblairmtreinish: that's the spirit!21:05
jeblairmtreinish: done21:05
mtreinishjeblair: ok thanks21:05
*** rfolco has joined #openstack-infra21:05
clarkbjeblair: you have implemented a priority based load balancer for nodepool node boots21:07
clarkbjeblair: similar to a heap based priority queue? I am reading docstrings21:07
*** dims has quit IRC21:08
*** dims has joined #openstack-infra21:09
*** matty_dubs is now known as matty_dubs|gone21:09
*** rfolco has quit IRC21:11
dstufftclarkb: try twine >= 1.2.221:11
dstufftclarkb: I just uploaded twine using python 2.7 using twine 1.2.2 so it should work21:11
clarkbdstufft: awesome will do (I have to wait for puppet to upgrade twine then I will retrigger the job)21:12
*** zojoncj has joined #openstack-infra21:12
dstufftclarkb: np, I'll get some tests written for it to keep it from happening again :)21:12
*** mrodden1 is now known as mrodden21:13
*** marun has joined #openstack-infra21:14
*** boris-42 has quit IRC21:15
*** vipul is now known as vipul-away21:17
clarkbpuppet doesn't ensure twine is latest. I am going to upgrade by hand on pypi.slave. Not sure if want auto updates for that package (I assume yes?)21:20
clarkbrerunning gerritlib pypi push now21:23
fungiseems safe enough21:23
fungigiven assurances above ;)21:23
clarkboh hmm doesn't have an option for log basepath. Will this copy the log into a funny place?21:25
clarkbI am going to use a local copy that supplies the new arg and will propose a zuul chnage with that fix21:26
dstufftclarkb: I don't plan on breaking compatability of twine (where compatability is the cli)21:26
clarkbdstufft: cool. We should just ensure latest then21:27
*** dizquierdo has joined #openstack-infra21:29
*** thomasm has quit IRC21:35
clarkbdstufft: that familiar to you at all?21:35
dstufftclarkb: hrm, can you past your ~/.pypirc with the passwords stripped out?21:36
dstufftclarkb: moment21:37
*** chris613 has quit IRC21:37
openstackgerritA change was merged to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting
*** vipul-away is now known as vipul21:38
dstufftclarkb: you're missing the [distutils] section21:38
jog0clarkb: it is stuck21:39
jog0(e-r bot)21:39
clarkbdstufft: thanks21:39
dstufftclarkb: I'll probably give twine it's own format at some point, but it'll fall back to ~/.pypirc21:40
dstufftor extend pypirc21:40
dstufftsomething like that21:40
dstufftto support more options21:40
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add missing distutils section to pypirc.
clarkbdstufft: ^ that should fix it21:41
clarkbfungi: jeblair mordred can get quick reviews on that?21:41
clarkbjog0: thanks. I will wait for the change to add logging before restarting it21:41
* jog0 dusts off his patch21:42
dstufftclarkb: I +1'd it not that it'll help you at all :)21:42
*** jcoufal has quit IRC21:43
*** mriedem has quit IRC21:45
pabelangerwell, that was somewhat easy.  Managed to get nodepool running and launching instances21:45
*** oubiwann_ is now known as oubiwann21:46
*** fbo is now known as fbo_away21:49
*** boris-42 has joined #openstack-infra21:49
*** dizquierdo has quit IRC21:50
*** flaper87|afk is now known as flaper8721:51
*** davidhadas has quit IRC21:51
clarkbfungi: thanks for the review. I went ahead and approved it21:52
*** rnirmal has joined #openstack-infra21:52
fungii'm taking your word for it on that (and the author's obviously). couldn't find it mentioned in the readme and didn't see a manpage21:52
fungioh, distutils doc mentions it, not twine. that's why21:53
dstuffttwine just uses the distutils ~/.pypirc21:53
clarkbya twine is just being nice and compatible21:53
openstackgerritA change was merged to openstack-infra/config: Add missing distutils section to pypirc.
dstufftinfact it reaches into distutils to use the same routine21:53
dstufftto ensure compatability :D21:54
fungiaha! neat solution. no code duplication21:54
*** dkranz has quit IRC21:56
openstackgerritPaul Belanger proposed a change to openstack-infra/nodepool: Don't use max_overflow with sqlite
pabelangerTrying to decided if nodepool could be used to also setup the master side of the pool.  Eg: launch masters, then bring up slaves21:57
clarkboh neat21:58
*** johnthetubaguy has quit IRC21:58
fungipabelanger: that could be awesome21:58
*** boris-42 has quit IRC21:59
fungibut setting up a jenkins, even puppeted, still takes a bit of extra monkeying, so it could turn out to be more effort than it's worth21:59
fungiat least insofar as the global config is concerned22:00
pabelangerSo, my though process for nodepool is more generic, it's based more on developers.  Imagine they each have a tenant in openstack (the cloud).  Of course adding logic to nodepool, you specific your master( puppet), then setup each slave(node) with their specific profile.  Developer launch nodepool, blam, their tenant reproduces the production environment.22:01
pabelangerEven extending it, add hooks into messagebus, spinning up each specific node that needs more resources22:01
clarkbdstufft: still doing it (I checked on the slave the new distuitls section is in place)22:03
* clarkb looks closer for tyops and the like22:03
*** sarob has quit IRC22:03
dstufftclarkb: can you ssh into that server and do python -c "from twine.utils import get_distutils_config; print(get_distutils_config('pypi'))"22:04
dstufftthat'll print out the password so don't paste the results directly22:04
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add even more debug logs.
jog0clarkb: ^22:05
dstufftclarkb: sorry this isn't working as well as it could22:05
mtreinishjeblair: it looks like the d-g change works:
clarkbdstufft: no problem. The end result should be much better :)22:05
clarkbjog0: thanks22:05
*** changbl has joined #openstack-infra22:05
clarkbdstufft: that spits out a dict22:07
dstufftdoes it have a repository key?22:07
clarkbdstufft: it does22:08
clarkbdstufft: I wonder if the repository arg at isn't correct22:08
clarkbor maybe it is looking in $PWD and not ~/.pypirc22:08
* clarkb chdirs and tries again22:08
clarkbno works fine from the job workspace22:09
dstufftclarkb: what happens if you pass None instead of "pypi"22:09
clarkbdstufft: empty dict22:10
dstufftah ha22:11
dstufftnow why is it doing that..22:11
dstufftclarkb: what version of python?22:11
dstuffton 2.7.5 None gives me a full dict22:12
dstufftclarkb: you can fix this for right now by passing a -r pypi22:12
dstufftto the twine command22:12
clarkbok I will do that. thanks22:12
dstufftI'll dig into it more, I might end up parsing the config file on my own22:12
ttxzul: tomorrow morning I think22:13
openstackgerritClark Boylan proposed a change to openstack-infra/config: Specific twine repository when publishing packages
clarkbdstufft: fungi ^22:14
*** datsun180b has quit IRC22:15
dstufftclarkb: LGTM22:15
dstufftI think i'm going to remove the distutils dependency, stupid piece of crap22:15
*** mrda has joined #openstack-infra22:16
clarkbI am happy to use -r if that is the sanest work around (duplicating code is silly but so are silly bugs)22:17
dstufftwell it's not a straight forward use of distutils22:17
dstufftI have to trick it into letting me do it22:17
dstufftclarkb: I'm not happy with an api that requires -r, I'm annoyed it requires that for y'all22:19
clarkbwow that is full of trickery22:20
*** thedodd has quit IRC22:21
dstufftOriginally I figured i'd hit less edge cases by reusing that code22:21
dstufftbut I guess not!22:21
*** oubiwann has quit IRC22:26
*** sarob has joined #openstack-infra22:27
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add even more debug logs.
*** ryanpetrello has quit IRC22:31
openstackgerritClark Boylan proposed a change to openstack-infra/zuul: Update to pass LOG_PATH
clarkband that is what I am using to trigger the jobs22:32
*** ryanpetrello has joined #openstack-infra22:33
*** prad has quit IRC22:33
*** weshay has quit IRC22:36
*** flaper87 is now known as flaper87|afk22:39
*** jhesketh has joined #openstack-infra22:40
*** changbl has quit IRC22:47
*** changbl has joined #openstack-infra22:48
openstackgerritA change was merged to openstack-infra/config: Specific twine repository when publishing packages
*** rnirmal has quit IRC22:49
*** sarob has quit IRC22:50
*** ryanpetrello has quit IRC22:53
openstackgerritDavid Caro proposed a change to openstack-infra/reviewstats: Added some options to openapproved
*** sarob has joined #openstack-infra22:58
*** rcleere has quit IRC22:58
*** rfolco has joined #openstack-infra23:02
clarkb new gerritlib released23:06
*** dims has quit IRC23:06
clarkbtwine made it easy once the problems were sorted23:08
jeblairclarkb: yay!23:09
clarkbjeblair: now back to reviewing the nodepool change23:09
*** dkehn has quit IRC23:14
*** sarob_ has joined #openstack-infra23:17
*** jhesketh has quit IRC23:17
*** sarob has quit IRC23:20
*** dims has joined #openstack-infra23:23
lifelessclarkb: if you're poking at gerritlib, I have patches...23:24
lifelessclarkb: etc23:24
mordredlifeless: you're a patch23:25
lifelessmordred: patch thsi23:26
clarkb-patch thsi\n+patchthis23:27
clarkbwow fail.23:27
clarkblifeless: I have starred those changes and will attempt to get to them after this nodepool change23:28
lifelessclarkb: may interest (or horrify) you23:28
lifelessclarkb: my WIP count is ridiculously high at the moment23:29
mordredlifeless: before I nitpick - is there any specific reason for class GerritWatcher: rather than class GerritWatcher(object): ?23:30
*** jhesketh has joined #openstack-infra23:31
lifeless(object) is consideref bad form in Python 3.x23:31
jeblairisn't omitting it considered bad form in python2?23:32
lifelessthe canonical thing to do is set __metaclass__=type23:32
lifelessjeblair: it matters in Python 2 if you're doing anything requiring new style classes, which this isn't.23:33
mordredgreat. I love it that python3 changed that23:33
mordredbecause I thought what jeblair said23:34
mordred"it matters in Python 2 if you're doing anything requiring new style classes" is a terrible benchmark23:34
mordredsince new style classes have been just classes for like 10 years23:34
mordredand I can't remember what's new style and what's not anymore23:34
openstackgerritA change was merged to openstack-infra/zuul: Update to pass LOG_PATH
clarkbjeblair: the nodepool change seems to be more complicated than it could be. I keep wanting to think of it as a proper datastructure with well known methods and instead need to translate back and forth betwene the various *Request*23:41
*** jd__ has quit IRC23:42
jeblairclarkb: i agree it's complicated.  i tried two ways of making it less complicated that didn't work before this one.23:42
clarkbcould we have a single priority queue that was updated on each node being created?23:42
*** yaguang has joined #openstack-infra23:42
clarkbthen simply grab the item at the head of the queue when booting a node23:42
dstufftclarkb: did that work then?23:43
clarkbdstufft: yup, thanks23:43
dstufftsorry for the troubles, glad it works now :D23:43
*** jd__ has joined #openstack-infra23:44
jeblairclarkb: can you explain your idea a bit further?23:44
clarkbjeblair: I think that is essentially what we have ended up with, but the state of the queue is managed in many places23:44
*** sarob has joined #openstack-infra23:45
clarkbjeblair: a single that is recreated each time external state changes eg building a node or a test starting. To spin up a node you recalculate priority then grab the head of that queue and grant the request, continue until all available resources are used23:48
*** sarob_ has quit IRC23:49
jeblairclarkb: so calculating the priority is likely to be just as complex as the current logic23:49
clarkbjeblair: from my understanding of the current code this is basically what you have but the operations aren't explicit from a data structure perspective. Instead there are several cooperating objects that together form that queue23:49
clarkbjeblair: right, but we can do it in one place I think23:49
jeblairclarkb: it's managed in several places because it's solving several problems -- which providers to use for which images, how much of each provider to use for each image, and where to attach each node23:50
jeblairclarkb: the current system ends up with a count of nodes for each provider-image-target, and loops over that to spin them up.  it sounds like the major change would be that it would remove items from a queue instead.  but all the logic to figure out what items need to be supplied/attached would still need to exist, and likely similarly complicated.23:53
clarkbjeblair: ok. I will keep thinking about how to reduce complexity as I finish up the review23:53
jeblairclarkb: note that it accomplishes those tasks in 3 phases, so it's taking a somewhat holistic view -- i think that's part of the complexity.23:53
*** dkehn has joined #openstack-infra23:54
clarkbjeblair: I think what I want is the ability to rely on preexisting semantic operations, but they may not help too much here (difficulty for me is keeping track of what each of the discrete thing is supposed to do)23:54
*** che-arne has joined #openstack-infra23:54
*** dkehn has quit IRC23:54
zaroclarkb: which puppet module creates the cgit machines (git01.o.o, git02.o.o, etc.)?23:55
clarkbzaro: cgit23:55
*** dkehn has joined #openstack-infra23:56
*** jhesketh has quit IRC23:56
clarkbjeblair: the docstring at the top of the allocation file is great, but mapping the pieces onto objects makes my brain tired :)23:57
*** jhesketh has joined #openstack-infra23:57
jeblairclarkb: specific cases that drive this complexity are things like: image A can be supplied only by provider 1, image B can be supplied by providers 1, 2, and 3; requests for image A and B each exceed the available capacity of all providers23:57
*** jcoufal has joined #openstack-infra23:58
clarkbjeblair: you are ensuring that B doesn't starve A23:58
jeblairclarkb: yep23:58
jeblairclarkb: and then on top of that, there's taking nodes of image A from provider 1 and mapping them to target X in the right proportions (because the target is specified in terms of image _and_ provider)23:59
jeblairclarkb: i admit, i kinda drew a blank on succinct names.  so the class names got a bit long.23:59

Generated by 2.14.0 by Marius Gedminas - find it at!