Tuesday, 2015-02-10

mordredwhich is where these get built00:00
marunclarkb: I'm afraid the refactored testr artifact processing has uncovered an assumption that testr and subunit are installed system-wide.00:00
clarkbmordred: no it was exploding locally too00:00
mordredclarkb: yeah?00:00
clarkbmordred: so this is some progress if not all the progress00:00
mordredclarkb, SpamapS woot!00:00
clarkbmordred: the mysql on nodepool will be an issue too00:00
SpamapSthe answer there is to -hlocalhost isn't it?00:00
marunclarkb: since the neutron functional job only installs in a venv, this assumption is violated and the test artifacts aren't saved.00:00
SpamapSjust make sure the socket is used.00:00
clarkbmordred: SpamapS the thing to check is boot off that image and see that the openstack_citest or whatever the user name is user is on the server00:00
mordredSpamapS: probably00:00
clarkbSpamapS: mordred if you can convince puppet-mysql to do that00:01
clarkbmarun: doesn't it know to run out of a vnev?00:01
*** bswartz has joined #openstack-infra00:01
marunclarkb: devstack-gate doesn't, but I could fix that00:01
*** bknudson has joined #openstack-infra00:01
SpamapS/dev/loop0p1             9.0G  4.8G  3.8G  56% /tmp/image.mHQiUpcS/mnt00:02
SpamapSrather large image00:02
marunclarkb: there is currently no concept of 'venv' in d-g, though.00:02
mordredSpamapS: yes00:02
mordredit's a very large image00:03
mordredSpamapS: it contains a cache of everyhting you might ever need in any gate job00:03
marunclarkb: Can you think of another way to fix that than adding YACV (yet another configuration variable) to point to it?00:03
*** melwitt has quit IRC00:03
SpamapSsolid HDD light for the last 4 minutes00:03
SpamapSmordred: cacheallthethings.qcow200:03
clarkbmarun: I am trying to figure out how it worked for tempest first00:03
clarkbmarun: my guess is do something like what tempest did00:03
*** dimssum__ has joined #openstack-infra00:04
*** dimssum__ is now known as dimsum__00:04
marunclarkb: tempest relies on testr and subunit being installed system-wide00:04
clarkboh I see00:04
*** dimsum__ has quit IRC00:04
openstackgerritMerged openstack-infra/zuul: Check liveness of changes before removing  https://review.openstack.org/15381300:05
marunclarkb: so, would you prefer that d-g can be informed of which venv to use?00:05
clarkbmarun: and devstack must install them for us when tempest is used but not during your tests?00:05
marunclarkb: or just install testr/subunit automatically?00:05
marunclarkb: correct00:05
clarkbmarun: no I think likely the best thing to do is just install testr globally since that is what tempest expects00:05
marunclarkb: ok, I'll submit a patch shortly.00:05
marunclarkb: danke00:05
*** lnxnut_ has quit IRC00:06
*** gyee has quit IRC00:06
*** markvoelker has quit IRC00:07
mordredclarkb: what was the thing you wanted me to review?00:07
*** Sukhdev has joined #openstack-infra00:07
clarkbmordred: 154188 its the "simple" fix for spec_helper00:07
*** chlong has joined #openstack-infra00:07
*** nelsnelson has joined #openstack-infra00:08
*** lnxnut_ has joined #openstack-infra00:08
mordredclarkb: +#00:08
mordredclarkb: +300:08
clarkbmordred: thanks00:08
* mordred is at his favorite part of the process now- uploading the image00:09
*** crc32 has quit IRC00:09
dougwighi infra.  stackforge/octavia was rolling along until last week, when things won't merge anymore. is there something wrong with the project config for that?  example review: https://review.openstack.org/#/c/149789/00:10
openstackgerritRichard Jones proposed openstack-infra/infra-specs: Add initial draft of bower spec  https://review.openstack.org/15429700:11
r1chardj0n3smordred: ^^00:12
mordredoh my00:12
*** lnxnut_ has quit IRC00:12
*** r-daneel has quit IRC00:12
jeblairdougwig: 149789 depends on 151496 which is WIP00:13
dougwigd'oh, i totally missed that.  thanks.00:13
* dougwig gets more coffee.00:13
jeblairnp.  just call me 'zuul'00:14
*** dims__ has joined #openstack-infra00:14
openstackgerritMerged openstack-infra/project-config: Fix rspec installs during image builds  https://review.openstack.org/15418800:14
*** gyee has joined #openstack-infra00:14
*** carl_baldwin has quit IRC00:15
mordredO M G could someone at rackspace please update the SSL certs??? the python warning is driving me batty00:15
mordredjroll: you fixed the config-drive thing - next task - fix all the SSL certs00:16
* mordred hands jroll a fluffy sea otter00:16
openstackgerritMaru Newby proposed openstack-infra/devstack-gate: Install testr system-wide for artifact processing  https://review.openstack.org/15429900:18
marunclarkb: ^^00:18
jheskethmordred: re r1chardj0n3s's bower.js stuff, I've been talking to him about it and in my opinion AFS for a shared cache is overkill. I think we should start with the simpler implementation (as outlined in the spec) and go from there00:18
jheskethmordred: if the failure rate at that point turns out to be worse than we'd like, we can look at building more complicated solutions00:18
*** salv-orlando has quit IRC00:19
SpamapSmordred: working now?00:20
mordredjhesketh: I do like the general idea of starting with things that are smaller00:20
*** salv-orlando has joined #openstack-infra00:20
mordredSpamapS: dunno - still polling on glance task-show00:20
openstackgerritClark Boylan proposed openstack-infra/puppet-project_config: Provide way to specify repo ref  https://review.openstack.org/15430000:21
mordredoh - clarkb is doing it00:22
jeblairr1chardj0n3s: thanks!  i've left you some feedback00:22
clarkbmordred: ya I figured I would00:22
mordredclarkb: woot00:22
clarkbmordred: working on the second half now00:22
jeblairclarkb: aha! i see what you are doing!  what problems is that solving?00:23
mordredjeblair: race condition00:23
jeblairmordred: that much i got00:24
mordredjeblair: because the git pull is happening on each host00:24
jeblairmordred: what is the race breaking?00:24
clarkbjeblair: earlier today I merged a change to add a project00:24
jeblairclarkb: is this causing the empty git.o.o repos?00:24
clarkbjeblair: puppet ran on gitXX with project_config prior to that merge and review.o.o ran after00:24
clarkbjeblair: yes00:24
*** salv-orlando has quit IRC00:24
clarkbso next step is to have ansible check what HEAD is at the beginning of puppet_run_all then feed that specific ref to all the puppet runs00:24
mordredso we really want the orchestration end to be the one that knows "hey, all y'all - how bout you pull ref X"00:24
mordredSpamapS: I'm booting a server ... let's see what breaks this time!00:25
jrollmordred: I've tried, I promise :x00:26
jogoclarkb: \o/ thanks00:27
r1chardj0n3scheers jeblair!00:27
* r1chardj0n3s AFKs for a bit now00:27
* mordred hands jroll a box of fluffy sea otters00:27
*** salv-orlando has joined #openstack-infra00:27
jheskethjeblair: I've responded to your cache query on r1chardj0n3s's spec. Richard will need to check what I've said is true, but I think it's in line with his thinking00:27
*** r1chardj0n3s is now known as r1chardj0n3s_afk00:27
clarkbmordred: except I have to figure out how to pass an env var from env calling ansible into the env ansible runs commands in00:27
mordredclarkb: it's possible we want to extend our puppet module00:28
mordredway too much overload00:28
openstackgerritMerged openstack-infra/zuul: Avoid leaking dynamic change queues  https://review.openstack.org/15381400:28
*** sabeen has quit IRC00:29
mordredclarkb: it's possible we might want to extend our ansible module that calls puppet with a facility to pass in arbitrary key-value things to pass to the puppet invocation00:29
*** tmcpeak has joined #openstack-infra00:30
*** tmcpeak has quit IRC00:32
jeblairjhesketh, r1chardj0n3s_afk: thanks.  i've responded to your responses.  :)00:32
clarkbmordred: `git ls-remote --heads https://git.openstack.org/openstack-infra/project-config master | cut -f 1` is the magic to run to get the ref00:33
clarkbdon't need the --heads even00:33
*** sarob has quit IRC00:35
*** ashleighfarnham has quit IRC00:35
*** ZZelle has quit IRC00:36
*** signed8bit is now known as signed8bit_ZZZzz00:37
*** dangers is now known as dangers_away00:38
clarkbmordred: I think I may have it00:39
clarkbmordred: http://docs.ansible.com/faq.html#how-do-i-access-shell-environment-variables00:39
jheskethjeblair: and more responses to your responses :-)00:39
jhesketh*to my responses00:39
mordredSpamapS, clarkb, jeblair: BOOM! I have networking00:41
*** yamamoto has joined #openstack-infra00:42
mordredok. so - I have booted ubuntu on rax00:42
mordredI need to verify that centos/systemd work ...00:42
mordredand I need to verify that this all works as expected on HP00:42
mordredbut I think we're close to this task being done-ish00:42
*** Ng has joined #openstack-infra00:42
openstackgerritMaru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs  https://review.openstack.org/14676200:43
clarkbmordred: how are roles and tasks different?00:44
mordredclarkb: a role is a named collection of tasks00:44
clarkbmordred: and if I attach environemnt to a role should I expect that to work properly see http://docs.ansible.com/playbooks_environment.html00:44
openstackgerritMerged openstack-infra/zuul: Add ignore-dependencies option  https://review.openstack.org/15427400:44
marunjhesketh: I've added back test_model at your request ^^00:45
jheskethmarun: shiny :-)00:45
mordredclarkb: no - I think you'd still  need the task to consume it too00:45
funginikhil_k: i've added you to https://launchpad.net/~glance-coresec now and set you as an administrator for it. please adjust the membership on that group to reflect glance core reviewers who should be engaged in confirming/debunking embargoed glance security bugs and vetting the non-public patches attached to them00:45
clarkbmordred: so fwiw this is the clunkiest bit of ansible00:46
clarkbmordred: you have to do far too much plumbing for something as simple as make this env var appear over there happen00:46
jeblairjhesketh: i've responded, but that's about the max length conversation i can have in a review00:47
mordredclarkb: yes, I agree00:47
mordredI think we need to re-convene on how we unclunky it00:47
marunjeblair: Your approval would be much appreciated on https://review.openstack.org/#/c/146762/  No changes other than adding test_model.py back at jhesketh's request.00:47
*** Guest46399 has quit IRC00:47
jeblairjhesketh, r1chardj0n3s_afk: i have two points that i'm very firm on: (1) we can not require that packages be published to github, and (2) we can not require that the gate access github for tests00:47
mordredclarkb: I was mostly waiting to make it betterer until we'd gotten angry at it so that the conversation would be fuller00:47
mordredjhesketh, r1chardj0n3s_afk: I concur with jeblair on those two points00:48
marunjhesketh: thank you!00:48
* mordred dinners00:48
jheskethjeblair: I thought we rebuilt the nodepool images nightly?00:48
jeblairjhesketh: we _attempt_ to do so nightly00:49
jeblairjhesketh: they often fail for weeks on end00:49
nikhil_kfungi: awesome, thanks!00:49
jeblairjhesketh: because things like github failing, or some puppet thing breaking, or any number of things going wrong00:49
r1chardj0n3s_afkjeblair: bower supports many schemes that no-one uses...00:49
jheskethjeblair: okay, noted00:49
jheskethfair enough, you've convinced me :-)00:49
jeblairjhesketh: as i mentioned (and was discussed in scrollback), our current images are more than 170 hours old00:49
*** sarob has joined #openstack-infra00:49
* jhesketh fails to keep up with scrollback00:50
r1chardj0n3s_afkjeblair: we cab look into supporting other git repos, but I'm not sure even the bower tool itself does (it doesn't implement all of its own spec ;)00:50
r1chardj0n3s_afkjeblair: I will investigate though00:50
marunsdague: regarding https://review.openstack.org/#/c/153365/, am I just confused as to the utility of grep/egrep?00:51
jeblairr1chardj0n3s_afk: thanks.  if that's the case, that's kind of terrible (the spec seems halfway sensible)00:51
fungisounds like the spec was wishful thinking. rfc it and they will come00:52
openstackgerritClark Boylan proposed openstack-infra/puppet-ansible: Pass PUPPET_CONFIG_REF through to the puppet run  https://review.openstack.org/15430400:54
openstackgerritClark Boylan proposed openstack-infra/system-config: Set PROJECT_CONFIG_REF for ansible  https://review.openstack.org/15430500:56
clarkbmordred: ^ that is probably all wrong, feel free to update as necessary00:56
*** Longgeek has joined #openstack-infra00:56
*** MarkAtwood has quit IRC00:57
*** ildikov has quit IRC01:01
openstackgerritClark Boylan proposed openstack-infra/puppet-ansible: Pass PROJECT_CONFIG_REF through to the puppet run  https://review.openstack.org/15430401:04
*** shashankhegde has quit IRC01:06
*** zhiwei has joined #openstack-infra01:07
jogoclarkb: very excited about aiopcpu patches landing, thanks01:07
clarkbjogo: hopefully they get in ;)01:08
*** hdd has quit IRC01:12
*** gokrokve has quit IRC01:14
*** Longgeek has quit IRC01:15
*** ildikov has joined #openstack-infra01:15
*** achanda has quit IRC01:16
*** zz_dimtruck is now known as dimtruck01:18
*** ddieterly has joined #openstack-infra01:21
*** gokrokve has joined #openstack-infra01:22
*** tmcpeak has joined #openstack-infra01:25
*** otter768 has joined #openstack-infra01:27
openstackgerritMerged openstack-infra/elastic-recheck: Add query for ironic bug 1418833  https://review.openstack.org/15427201:27
openstackbug 1418833 in Ironic "check-tempest-dsvm-ironic-agent_ssh fails due to ConfigDriveWriteError" [Critical,Fix committed] https://launchpad.net/bugs/1418833 - Assigned to Jay Faulkner (jason-oldos)01:27
*** marun has quit IRC01:28
jheskethjeblair: btw, did you see my change for zuul to disable a pipeline after too many consecutive failures?01:29
*** woodster_ has quit IRC01:30
*** tiswanso has quit IRC01:31
*** ChuckC_ has quit IRC01:32
*** tiswanso has joined #openstack-infra01:32
*** ChuckC_ has joined #openstack-infra01:32
*** ChuckC_ is now known as ChuckC01:32
*** signed8bit_ZZZzz is now known as signed8bit01:33
*** otter768 has quit IRC01:33
*** ChuckC has quit IRC01:35
*** ChuckC has joined #openstack-infra01:36
*** hdd has joined #openstack-infra01:38
*** shashankhegde has joined #openstack-infra01:38
*** Sukhdev has quit IRC01:39
*** ddieterl_ has joined #openstack-infra01:43
*** ddieterly has quit IRC01:43
openstackgerritDarragh Bailey proposed stackforge/python-jenkins: Provider helper classes for plugin data management  https://review.openstack.org/15431601:47
dstufftDoes Openstack rely on apt-get installed python at all in debstack or the CI or anything?01:50
sdakehow many physical machines compose the jenkins gate?01:50
jeblairjhesketh: i think i saw it briefly but haven't reviewed it yet.  seems useful even if we don't end up wanting to use it upstream01:52
jeblairsdake: around 1000 virtual machines on public cloud providers.  no idea how much that translates to physically.01:53
jheskethjeblair: cool. It's mostly targeted at 3rd parties. Even if upstream infra (ie 1st parties) breaks terribly I suspect we want to keep voting01:53
sdakejeblair cool thanks01:53
clarkbdstufft yes all python installations are from system packages01:53
jeblairsdake: (plus some physical hardware in use by tripleo, but that probably wasn't your question)01:53
clarkbdstufft we dont custom python01:53
sdakesay I want to run docker in the gate01:54
dstufftclarkb: er, I meant python-pip sorry01:54
sdakewould that be possible?01:54
*** gokrokve has quit IRC01:54
jeblairsdake: are you asking in reference to a particular project?01:54
clarkbyes its already done by the ipa image builds01:54
sdakewhat I'd like to do is run docker with some specific arguments (could be in a shell script) and then run tempest after01:54
clarkbdstufft no thats all get pip now iirc01:54
sdakesweet got a pointer to how they did it?01:54
sdake(github link?)01:54
sdakejeblair for kolla01:55
dstufftclarkb: ok, then you won't be affected by the patch that Ubuntu landed into vivid ~5 hours ago01:55
sdakeI want to add functional pre-commit testing01:55
*** ChuckC_ has joined #openstack-infra01:55
jheskethjeblair: however my initial implementation was to simply set a cut-off on the window. That is, once the window falls below X we stop voting. This solution turns out not to work because of the way independant pipelines have a change queue for each project change. What do you think about just counting failures at a pipeline level? Or possibly a more complicated solution of having a pipeline wide window01:55
jeblairsdake: basically you have root on those machines so you can do whatever in the tests :)01:56
sdakewhat starts the test?  some specific command?01:56
*** bhuvan has quit IRC01:56
sdakeliek I know there is a job pep8 which runs testr pep8 iirc01:57
clarkbsdake https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ironic-python-agent-jobs.yaml01:57
sdakebut how could I get a specific shell script ran?01:57
sdakejeblair line 10 is from the python-agent repo?01:58
*** ChuckC has quit IRC01:58
jeblairsdake: yep01:58
sdakeand that returns 0 if the test cases pass?01:58
jeblairsdake: yep01:59
*** dmorita has quit IRC01:59
sdakeam I permitted to create jobs for non-integrated projects (i.e. stackforge project)01:59
jeblairsdake: sure thing.  stick 'em in their own file (kolla.yaml)01:59
jeblairjhesketh: oops, our changes are colliding (since i just added that for crd).  so we could do those, or instead of dynamically creating them, we could statically create one for each project and then work out how to get the correct one for the 'live' change that is being enqueued02:00
sdakejeblair I hate to sound like a noob, but I dont totally understand the structure of that file - do you have a link to appropraite documentation?02:00
jeblairjhesketh: but that still may not be what you want because then each project would have to independently hit the min02:01
jheskethyeah I saw something fly by for crd, but it was already merged so I haven't read the code yet02:01
jeblairjhesketh: so probably something on the pipeline is what you want.02:01
jheskethcertainly I think on the pipeline is easier to do02:02
jeblairjhesketh: i did write it in december :)02:02
jheskethoh, I must have missed one of the changes to review then02:02
sdakejeblair is there a fedora 21 node tyep?02:02
*** ChuckC has joined #openstack-infra02:02
* jhesketh tries to stay on top of zuul changes02:02
*** tmcpeak has quit IRC02:02
*** rwsu is now known as rwsu-afk02:02
jeblairsdake: http://docs.openstack.org/infra/manual/drivers.html#running-jobs-with-zuul02:02
jheskethjeblair: to rephrase the question, can you see an easy way, or a useful case, for having a pipeline window?02:03
*** nelsnelson has quit IRC02:04
*** ChuckC_ has quit IRC02:04
*** nelsnelson has joined #openstack-infra02:04
jeblairjhesketh: i feel like the main reason to do so is for this, and given that, it's worth considering an alternate implementation.02:04
jheskethI guess I was asking if there was another reason to do it. But otherwise I agree02:05
*** shashankhegde has quit IRC02:05
jeblairsdake: i believe there is.  note that fedora nodes come and go faster than other node types because of their support period.  you may need to keep your scripts up to date as that changes.02:05
jheskethjeblair: thanks, I should be able to roll something this afternoon :-)02:06
*** stevemar has quit IRC02:06
sdakeI see- reason I ask is I need a newer version of docker then is in 14.0402:06
jeblairsdake: (i'm not sure what the overlap or notification is going to look like, but in general, i don't expect it to be long)02:06
*** stevemar has joined #openstack-infra02:06
sdakeand I don't know how to package for ubuntu02:06
jeblairsdake: it's fine to use, just making sure you understand that they are a bit more 'best-effort' as far as the support period goes.02:07
sdakeI'll try ubuntu then if I can figure out how to package a new version of docker02:07
sdakeI assume we can install stuff from the network with the gate?02:07
jeblairsdake: for a stackforge project, yeah, you should be able to add a ppa and install that way.02:08
jeblair(we wouldn't do that for an openstack project for policy reasons)02:08
sdakeya, this is more proof of concept prototype work, Im sure the new version of ubuntu will have the proper version of docker02:09
sdakeby the time we are done, ubuntu should have the proper version02:09
clarkbsdake we stick to ubuntu lts for the most part fwiw02:10
clarkbubuntu intermediate releases are only supported for 9 months02:10
sdakeI can work with that, I just need to find an ubuntu packager in the short term :)02:10
*** Longgeek has joined #openstack-infra02:12
sdakejeblair is it possible for a test case to consume multiple machines?02:13
sdakevms rather02:13
*** salv-orlando has quit IRC02:15
jeblairsdake: yes, but it's pretty difficult and we're still just getting started with openstack.  i'd recommend against it for now.02:18
*** spzala has quit IRC02:18
sdakebummer so single node only02:18
jeblairyou can do a lot with a node :)02:18
*** masayukig has quit IRC02:19
sdakeI want to test HA which requires 3 nodes02:19
*** masayukig has joined #openstack-infra02:19
sdakespecifically i have a rabbitmq container02:19
sdakeand I want to run it on 3 nodes and run test cases against it02:19
jeblairsdake: can you run 3x on one node?  then shoot one of them to test ha?02:20
sdakeI'm asking around atm about that exact question02:20
*** nelsnelson has quit IRC02:22
*** woodster_ has joined #openstack-infra02:23
*** dims__ has quit IRC02:23
sdakestill asking :)02:23
mordredclarkb: I'll take a look at those when I wake up02:26
*** lnxnut has joined #openstack-infra02:26
*** gokrokve has joined #openstack-infra02:27
*** sdake has left #openstack-infra02:32
*** sdake has joined #openstack-infra02:32
*** bhunter71 has quit IRC02:35
*** Longgeek has quit IRC02:37
sdakejeblair last Q then I'll leave you alone :)  is it possible to do nested virt - in other words could I start 3 vms from within 1 vm to run the test case?02:38
*** gyee has quit IRC02:38
*** rfolco has joined #openstack-infra02:39
*** lnxnut has quit IRC02:39
jeblairsdake: possible, but it's so slow as to be unusable (we tried with devstack)02:39
*** lnxnut has joined #openstack-infra02:40
sdakecontainer launch for me is much tidier then devstack02:40
jeblairsdake: so the nested virt is suitable for running something like cirros (which is what the devstack jobs do), but not real workloads nested02:40
sdakedevstack takes 20 minutes on my high end hardwware02:40
sdakeso I couldn't do a docker pull docker run in a nested virt then02:40
mordredour cloud providers do not support accelerated passthrough02:41
mordredso nested virt is legit nested02:41
mordredand not usuable speed-wise02:41
*** Liu_xinguo has joined #openstack-infra02:44
*** rackertom has joined #openstack-infra02:45
*** markvoelker has joined #openstack-infra02:47
*** gokrokve_ has joined #openstack-infra02:49
*** yamahata has quit IRC02:49
*** unicell has quit IRC02:50
*** gokrokve_ has quit IRC02:50
*** wuhg has joined #openstack-infra02:50
*** gokrokve_ has joined #openstack-infra02:51
*** cinerama has quit IRC02:51
*** gokrokve has quit IRC02:52
*** tqtran has quit IRC02:54
openstackgerritMerged openstack-infra/devstack-gate: Separate public and flat interfaces for multinode  https://review.openstack.org/15309302:54
*** cinerama has joined #openstack-infra02:54
openstackgerritMerged openstack-infra/devstack-gate: Diagram the nova net multinode network setup  https://review.openstack.org/15367202:54
openstackgerritMerged openstack-infra/devstack-gate: Enable nova multi host on aiopcpu tests  https://review.openstack.org/13160402:55
openstackgerritMerged openstack-infra/devstack-gate: Enable nova-api-metadata for multi host nova-network  https://review.openstack.org/13617902:55
sdakethanks jeblair I'm off02:55
sdakeyou rock as usual :)02:55
sdakemordred cool thanks, I guess I'll nix that idea of nested virt then02:56
*** otter768 has joined #openstack-infra03:01
*** sdake has quit IRC03:03
*** r1chardj0n3s_afk is now known as r1chardj0n3s03:03
*** dims__ has joined #openstack-infra03:03
*** dims__ has quit IRC03:07
*** dims__ has joined #openstack-infra03:07
*** dimtruck is now known as zz_dimtruck03:07
*** ddieterl_ has quit IRC03:08
*** ddieterly has joined #openstack-infra03:09
*** ddieterly has quit IRC03:09
mordredsdague: it would be cool if it worked ...03:09
*** ddieterly has joined #openstack-infra03:10
*** jerryz has joined #openstack-infra03:10
*** tqtran has joined #openstack-infra03:11
clarkbso if using containeds anyway why not test it that way03:12
clarkbshould even be testable running 3 daemons without any virt03:12
clarkbalso protip rabbitmq ha recovery is bonghits03:12
clarkbany messages on the losing partition go poof into the ether03:13
*** jyuso has quit IRC03:14
*** ddieterly has quit IRC03:14
mordredclarkb: from what I can tell, rabbitmq ha is not ha by any definitions I've ever had of that term03:15
clarkbyes pretty much03:15
*** jyuso1 has joined #openstack-infra03:15
clarkbit just detects partitions and one side loses badly03:15
mordredwell, that's better than not detecting partitions03:15
openstackgerritSteve Baker proposed openstack-infra/project-config: check-heat-dsvm-functional-mysql raise timeout  https://review.openstack.org/15432503:15
jheskethclarkb: so... next steps for swift logs. Should we switch over some more jobs?03:15
* mordred loves swift logs03:16
clarkbjhesketh maybe? also possibly concurrent uploads03:16
*** salv-orlando has joined #openstack-infra03:16
jheskethclarkb: I was thinking about that.. we are already uploading 10 files at a time as it turns out03:16
clarkbbut I am pretty confident in it actually working at this point that more jobs sounds good03:16
jheskethdue to the way the formpost works03:16
clarkboh thats probably good enough03:16
jheskethmaybe we need another way of looking into the high processing times03:17
jhesketheg making sure it's actually a bandwidth issue and not something else?03:17
*** lnxnut has quit IRC03:17
*** salv-orlando has quit IRC03:21
*** boris-42 has quit IRC03:22
jheskethclarkb: maybe the upload script should print out what it is doing so we can get the timestamps in the jenkins logs (we'll never see them in the actual swift logs as the uploading all happens after the console is fetched)03:22
*** spzala has joined #openstack-infra03:22
clarkbjhesketh that seems reasonable03:22
*** baoli has quit IRC03:23
jheskethokay, gimme a sec :-)03:23
*** gokrokve_ has quit IRC03:25
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Add extra information to zuul swift uploading  https://review.openstack.org/15432603:28
jheskethclarkb: ^ pretty crude but should give us an idea of what's slow03:29
*** david-lyle is now known as david-lyle_afk03:29
*** lnxnut has joined #openstack-infra03:29
*** baoli has joined #openstack-infra03:29
*** BharatK has joined #openstack-infra03:29
*** nelsnelson has joined #openstack-infra03:30
*** dmsimard_away is now known as dmsimard03:30
*** dmsimard is now known as dmsimard_away03:32
*** ayoung is now known as ayoung_sleep03:34
*** amrith is now known as _amrith_03:34
*** nelsnelson has quit IRC03:35
*** salv-orlando has joined #openstack-infra03:35
*** ashleighfarnham has joined #openstack-infra03:36
*** Longgeek has joined #openstack-infra03:37
*** salv-orlando has quit IRC03:37
*** ashleighfarnham has quit IRC03:40
*** gokrokve has joined #openstack-infra03:42
*** salv-orlando has joined #openstack-infra03:44
*** unicell has joined #openstack-infra03:44
*** _amrith_ is now known as amrith03:45
*** achanda has joined #openstack-infra03:45
*** Ryan_Lane has quit IRC03:46
*** koolhead17 has joined #openstack-infra03:46
*** dims_ has joined #openstack-infra03:47
*** dims__ has quit IRC03:49
*** hdd has quit IRC03:50
greghaynesclarkb: thers settings for what rmq does when partitions happen03:51
greghaynesclarkb: really where you lose data is when things rejoin the cluster03:51
greghayneswhoever is the joiner just drops all its data on the floor, which may or may not be an issue03:52
clarkbya there are manual options but iirc even they drop data on the floor03:52
greghaynesyep. You can set it to require quorum to accept a message, which is kind of the best you can get in terms of HA03:52
greghaynesits not totally reliable even so though03:53
greghaynesclarkb: We had a lot of input on the tripleo rmq ha configs btw, if youre looking for reference03:54
clarkbgreghaynes: nah, I was just commenting on sdake's interest in testing rabbitmq ha03:54
clarkbthe big thing being one node should be sufficient you just need 3 daemons03:55
greghaynesah yep03:55
*** tqtran has quit IRC03:55
*** carl_baldwin has joined #openstack-infra03:58
*** tmcpeak has joined #openstack-infra03:58
greghaynesrunning a build using the openstack-repos element really makes me want to parellelize source-repositories...03:58
mordredgreghaynes: I think we'd like you if you did03:59
*** salv-orlando has quit IRC04:00
*** yamahata has joined #openstack-infra04:00
*** tmcpeak has quit IRC04:00
clarkbgreghaynes: its not that bad, though you have liekly never built a snapshot image so don't have that to compare against04:00
*** vigneshvar has joined #openstack-infra04:00
greghaynesclarkb: yea, im spoiled04:00
greghaynesi was building images in a couple mins04:00
*** sdake has joined #openstack-infra04:02
clarkbgreghaynes: for me the longest step with a relatively up to date cache is the file copy04:04
clarkbgreghaynes: it is why I hate the file copy and wish dib would mount the image and chroot on top of it04:04
openstackgerritMerged openstack-infra/nodepool: Fix nodepool --version  https://review.openstack.org/14593004:04
clarkb(I understand the tmpfs case but you can switch based on tmpfs I just haven't had the time to actually change dib to do that yet)04:05
greghaynesI had big plans at one point to use lvm and be able to lvm snapshot a base image and chroot into it04:05
clarkbgreghaynes: we spend all this time extracting a qcow2 image into a tarball, exploding the tarball, chrooting into that, mounting a filesystem, then copying everything from the chroot into that04:05
*** salv-orlando has joined #openstack-infra04:05
clarkbgreghaynes: when all we need to do is take the source qcow2, mount a copy, and write into it, done04:06
clarkbgreghaynes: also if we did that precise could build centos7 images04:06
*** achanda has quit IRC04:07
greghaynesI thought you either had to nbd or convert to !qcow2 to edit it?04:07
*** mriedem has quit IRC04:07
clarkbmaybe? oh especially since you want raw to convert to other outputs04:07
clarkbbut the copying of files at the end should be avoidable at lesat04:08
greghaynesyea, thats the other problem, but you could in theory optimize the same in and out format case04:08
greghaynessure, and really we should at a minimum be able to make a source cache04:08
*** achanda has joined #openstack-infra04:08
clarkbbut ya looks like nbd is required04:08
greghayneswhere we dont rexplode the tarball unless the source file isnt in the cache04:08
greghayneser, dont reconvert04:09
clarkbmordred: did SpamapS say if he had to change anything to make mysql work or if he checked the resulting image?04:09
*** ddieterly has joined #openstack-infra04:10
*** baoli has quit IRC04:10
greghaynesIIRC he ran into the missing versionedobjects repo issue when he was trying out the in_chroot setting04:11
clarkbthen I fixed it and I thought he said he got mysql working04:11
*** salv-orlando has quit IRC04:12
clarkbwell "fixed" the proper race killing fix is in progress04:12
*** salv-orlando has joined #openstack-infra04:14
*** Longgeek has quit IRC04:14
*** ddieterly has quit IRC04:15
*** salv-orl_ has joined #openstack-infra04:18
*** salv-orlando has quit IRC04:19
*** tiswanso has quit IRC04:22
*** salv-orl_ has quit IRC04:23
*** mtanino has quit IRC04:23
mordredclarkb: I do not know if he's booted the image04:26
*** salv-orlando has joined #openstack-infra04:27
*** spzala has quit IRC04:27
*** dims_ has quit IRC04:32
*** salv-orlando has quit IRC04:33
*** jdg is now known as jgriffith04:34
*** harlowja is now known as harlowja_away04:34
*** jgriffith is now known as Guest328204:34
*** vigneshvar has quit IRC04:35
*** Guest3282 is now known as jgriffith04:36
*** dumbhead has left #openstack-infra04:43
*** Sukhdev has joined #openstack-infra04:48
openstackgerritgreghaynes proposed openstack-infra/project-config: set -x in cache-testrepository install  https://review.openstack.org/15433704:49
*** lnxnut has quit IRC04:50
*** lnxnut has joined #openstack-infra04:51
*** coolsvap_ is now known as coolsvap04:52
*** otter768 has quit IRC04:53
*** lnxnut has quit IRC04:56
*** gokrokve_ has joined #openstack-infra04:56
*** gokrokve has quit IRC04:59
*** gokrokve_ has quit IRC05:00
*** unicell1 has joined #openstack-infra05:04
*** unicell has quit IRC05:04
*** wuhg has quit IRC05:09
*** garyh has quit IRC05:09
*** ddieterly has joined #openstack-infra05:11
*** nikil89_ has joined #openstack-infra05:15
*** ddieterly has quit IRC05:16
*** nikil89_ is now known as naggappan05:18
*** ivar-laz_ has joined #openstack-infra05:19
*** achanda has quit IRC05:19
*** salv-orlando has joined #openstack-infra05:21
*** lnxnut has joined #openstack-infra05:21
*** ivar-lazzaro has quit IRC05:22
*** harlowja_at_home has joined #openstack-infra05:25
*** salv-orlando has quit IRC05:27
openstackgerritRichard Jones proposed openstack-infra/infra-specs: Enable bower for managing Horizon components  https://review.openstack.org/15429705:29
*** abhirc has quit IRC05:37
*** lnxnut has quit IRC05:38
*** daneyon has quit IRC05:40
*** BharatK has quit IRC05:41
*** garyh has joined #openstack-infra05:41
*** unicell has joined #openstack-infra05:42
*** salv-orlando has joined #openstack-infra05:42
*** unicell1 has quit IRC05:43
*** salv-orlando has quit IRC05:45
*** camunoz has quit IRC05:45
*** hdd has joined #openstack-infra05:45
*** carl_baldwin has quit IRC05:46
*** zz_dimtruck is now known as dimtruck05:47
*** ildikov has quit IRC05:49
*** oomichi has quit IRC05:50
*** harlowja_at_home has quit IRC05:51
*** mrmartin has joined #openstack-infra05:53
*** enikanorov_ has joined #openstack-infra05:53
*** salv-orlando has joined #openstack-infra05:55
*** BharatK has joined #openstack-infra05:58
*** salv-orlando has quit IRC06:00
*** harlowja_away has quit IRC06:01
*** jyuso1 has quit IRC06:09
*** jyuso1 has joined #openstack-infra06:09
*** mrmartin has quit IRC06:09
*** Ryan_Lane has joined #openstack-infra06:10
*** ddieterly has joined #openstack-infra06:12
*** ddieterly has quit IRC06:17
*** salv-orlando has joined #openstack-infra06:18
*** achanda has joined #openstack-infra06:20
*** salv-orlando has quit IRC06:21
*** r1chardj0n3s is now known as r1chardj0n3s_afk06:22
*** ivar-laz_ has quit IRC06:22
*** imcsk8 has quit IRC06:25
*** imcsk8 has joined #openstack-infra06:25
*** ashleighfarnham has joined #openstack-infra06:26
*** signed8bit has quit IRC06:29
openstackgerritYAMAMOTO Takashi proposed openstack/requirements: Add ryu>=3.18  https://review.openstack.org/15435406:29
*** camunoz has joined #openstack-infra06:34
*** Ryan_Lane has quit IRC06:35
*** unicell has quit IRC06:37
*** unicell has joined #openstack-infra06:37
openstackgerritSteve Martinelli proposed openstack/requirements: Add oslo.policy to global requirements  https://review.openstack.org/14862906:37
*** hdd has quit IRC06:44
*** ildikov has joined #openstack-infra06:44
*** Sukhdev has quit IRC06:46
openstackgerritJoshua Hesketh proposed openstack-infra/infra-specs: Enable bower for managing Horizon components  https://review.openstack.org/15429706:46
*** vigneshvar has joined #openstack-infra06:46
*** yamahata has quit IRC06:49
openstackgerritSteve Martinelli proposed openstack/requirements: Add oslo.policy to global requirements  https://review.openstack.org/14862906:49
*** mrunge has joined #openstack-infra06:50
*** abhi1892 has joined #openstack-infra06:50
*** lnxnut has joined #openstack-infra06:51
*** shashankhegde has joined #openstack-infra06:52
*** otter768 has joined #openstack-infra06:54
*** salv-orlando has joined #openstack-infra06:56
*** lnxnut has quit IRC06:56
*** salv-orlando has quit IRC06:58
*** otter768 has quit IRC06:59
*** abhi1892 has quit IRC06:59
*** camunoz is now known as camunoz_gone06:59
*** achanda has quit IRC07:02
*** Ryan_Lane has joined #openstack-infra07:03
*** Ryan_Lane has quit IRC07:04
*** achanda has joined #openstack-infra07:07
*** scheuran has joined #openstack-infra07:08
*** salv-orlando has joined #openstack-infra07:09
*** ashleighfarnham has quit IRC07:10
*** ddieterly has joined #openstack-infra07:13
*** Longgeek has joined #openstack-infra07:15
*** salv-orlando has quit IRC07:16
*** Longgeek has quit IRC07:16
*** ddieterly has quit IRC07:17
*** sdake has quit IRC07:29
*** doude has joined #openstack-infra07:32
*** avozza is now known as zz_avozza07:32
*** HeOS has quit IRC07:32
*** dims__ has joined #openstack-infra07:33
*** achanda has quit IRC07:33
*** markvoelker has quit IRC07:33
*** markvoelker has joined #openstack-infra07:34
*** armax has quit IRC07:36
*** dims__ has quit IRC07:37
*** markvoelker has quit IRC07:38
*** achanda has joined #openstack-infra07:39
*** jyuso1 has quit IRC07:42
*** yfried_ is now known as yfried|afk07:43
*** jyuso1 has joined #openstack-infra07:44
*** belmoreira has joined #openstack-infra07:45
*** Ryan_Lane has joined #openstack-infra07:47
*** yfried|afk is now known as yfried_07:48
*** achanda has quit IRC07:48
*** sputnik13 has joined #openstack-infra07:50
sputnik13are there instances where tempest failure is an anomaly and just rerunning it makes it succeed?07:51
*** unicell has quit IRC07:51
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for disabling bad pipelines  https://review.openstack.org/15436707:51
*** lnxnut has joined #openstack-infra07:52
*** unicell has joined #openstack-infra07:52
jheskethsputnik13: yes, a lot of these can be categorised by bugs that elastic-recheck is tracking07:52
jheskethif you get a comment from elastic-recheck it probably means you hit a known race condition07:53
sputnik13mmm, where would I see such a thing?07:53
sputnik13sorry, I'm still relatively new07:53
sputnik13I can't see anything in the failure log that makes me think it's the code that I checked in as the unit tests for it work, and it's not a terribly complicated piece of code07:54
*** chlong has quit IRC07:54
sputnik13is there an easy way to force a re-run of the tests, short of submitting a new patch?07:55
*** mrunge_ has joined #openstack-infra07:56
*** lnxnut has quit IRC07:57
*** mrunge has quit IRC07:57
*** jcoufal has joined #openstack-infra07:58
jheskethsputnik13: elastic-recheck is a bot that will leave a comment on your patch with the first line being 'recheck'08:00
jheskethanybody leaving a comment with the first line being 'recheck' will actually re-run the tests (to answer your second question)08:00
jheskethso you'd see the bot in the comments section, but I'm guessing since you haven't it's not a tracked bug08:00
jhesketh a recheck (by leaving a comment yourself) is probably a good next step if you don't think the failure is due to your code08:00
jheskethsputnik13: somehow that test segfaulted with no seemingly useful info!08:02
jheskethcurious t o see if it happens again08:03
sputnik13jhesketh: me too...  do I have to essentially submit a new patch?08:03
*** frobware_ has joined #openstack-infra08:03
*** fifieldt has joined #openstack-infra08:03
jheskethsputnik13: nope, just leave a comment with nothing more than the text 'recheck'08:04
jheskeththen you'll see it appear on http://status.openstack.org/zuul/ (you can use the filter to filter by your change number 152357)08:04
sputnik13oh, neat08:04
sputnik13thank you08:04
*** markvoelker has joined #openstack-infra08:04
jheskethno worries :-)08:07
*** yamamoto has quit IRC08:07
*** mrmartin has joined #openstack-infra08:08
jheskethsputnik13: jobs a re-running. The job that failed previously is coming through here https://jenkins06.openstack.org/job/gate-tempest-dsvm-neutron-src-oslo.utils/29/console08:08
sputnik13oh that's nice08:09
*** markvoelker has quit IRC08:09
*** shashankhegde has quit IRC08:11
*** ddieterly has joined #openstack-infra08:13
*** stevemar has quit IRC08:13
*** Longgeek has joined #openstack-infra08:14
*** luqas has joined #openstack-infra08:16
*** dtantsur|afk is now known as dtantsur08:17
*** ddieterly has quit IRC08:18
ttxjeblair: +108:19
*** woodster_ has quit IRC08:20
*** mpaolino has joined #openstack-infra08:22
*** oomichi_ has joined #openstack-infra08:23
*** sc68cal has quit IRC08:23
*** frobware_ has quit IRC08:24
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Reorder settled check in tests  https://review.openstack.org/15379908:24
*** zz_avozza is now known as avozza08:25
*** arxcruz has joined #openstack-infra08:26
openstackgerritMerged openstack-infra/project-config: Updates Kolla Project Description  https://review.openstack.org/15426108:28
jheskethjeblair: just been catching up on all your zuul changes that went in over the weekend, nice stuff!08:29
jheskethjeblair: I think we need to restart zuul on the server to get https://review.openstack.org/#/c/153801/3 to reload the webrpc stuff. At the moment http://zuul.openstack.org/ is showing all grey dots because the json feed doesn't have the new 'live' attribute08:30
*** sc68cal has joined #openstack-infra08:30
*** jgallard__ has joined #openstack-infra08:31
*** achuprin_ has quit IRC08:32
openstackgerritMerged openstack-infra/zuul: Reorder settled check in tests  https://review.openstack.org/15379908:33
*** yamamoto has joined #openstack-infra08:37
*** yamamoto has quit IRC08:38
openstackgerritDavid Pursehouse proposed openstack-infra/jenkins-job-builder: Add support to customize trigger-builds block thresholds  https://review.openstack.org/15437908:41
*** e0ne has joined #openstack-infra08:43
*** andreykurilin_ has joined #openstack-infra08:43
*** gokrokve has joined #openstack-infra08:43
*** jlibosva has joined #openstack-infra08:44
*** achuprin_ has joined #openstack-infra08:45
*** Ryan_Lane has quit IRC08:45
*** jistr has joined #openstack-infra08:47
*** MaxV has joined #openstack-infra08:48
*** afazekas has joined #openstack-infra08:52
*** gokrokve has quit IRC08:54
*** gokrokve has joined #openstack-infra08:55
*** otter768 has joined #openstack-infra08:55
*** jcoufal_ has joined #openstack-infra08:57
*** HeOS has joined #openstack-infra08:59
*** jcoufal has quit IRC09:00
*** gokrokve has quit IRC09:00
*** otter768 has quit IRC09:00
*** doude has quit IRC09:01
*** jpich has joined #openstack-infra09:02
*** markus_z has joined #openstack-infra09:03
*** markvoelker has joined #openstack-infra09:05
*** mrunge_ has quit IRC09:07
*** sarob has quit IRC09:09
*** markvoelker has quit IRC09:10
*** Guest82511 has joined #openstack-infra09:11
*** mrunge has joined #openstack-infra09:11
openstackgerritAndrey Pavlov proposed openstack/requirements: Add botocore to requirements  https://review.openstack.org/15438609:11
*** vigneshvar has quit IRC09:13
*** ddieterly has joined #openstack-infra09:14
*** derekh has joined #openstack-infra09:17
*** hashar has joined #openstack-infra09:17
*** sputnik13 has quit IRC09:17
*** ddieterly has quit IRC09:18
*** vigneshvar has joined #openstack-infra09:21
*** e0ne is now known as e0ne_09:24
*** jcoufal_ has quit IRC09:27
*** salv-orlando has joined #openstack-infra09:28
*** koolhead17 has quit IRC09:32
*** dizquierdo has joined #openstack-infra09:33
*** e0ne_ has quit IRC09:35
*** salv-orlando has quit IRC09:36
*** salv-orlando has joined #openstack-infra09:40
*** andreykurilin_ has quit IRC09:41
*** hichihara has quit IRC09:41
*** salv-orlando has quit IRC09:47
*** ihrachyshka has joined #openstack-infra09:48
*** e0ne has joined #openstack-infra09:52
ihrachyshkajogo, hey09:56
openstackgerritYAMAMOTO Takashi proposed openstack/requirements: README.rst: Fix a typo  https://review.openstack.org/15439409:56
*** ssam2 has joined #openstack-infra09:57
*** salv-orlando has joined #openstack-infra10:02
*** salv-orlando has quit IRC10:04
openstackgerritMehdi Abaakouk proposed openstack-infra/project-config: gnocchi: install the git repo  https://review.openstack.org/15439810:06
*** markvoelker has joined #openstack-infra10:06
*** bradjones has quit IRC10:07
*** rlandy has joined #openstack-infra10:09
silehtHi, any idea why http://pypi.iad.openstack.org/simple/python-qpid-proton/ is not up2date, 0.8 exists in pypi.python.org: https://pypi.python.org/pypi/python-qpid-proton/0.8-010:09
*** sarob has joined #openstack-infra10:10
*** bradjones has joined #openstack-infra10:10
*** ghostpl has joined #openstack-infra10:10
*** yamahata has joined #openstack-infra10:11
*** markvoelker has quit IRC10:11
*** salv-orlando has joined #openstack-infra10:12
*** BobBall_AWOL is now known as BobBall10:12
*** jcoufal has joined #openstack-infra10:15
*** ddieterly has joined #openstack-infra10:15
*** sarob has quit IRC10:16
*** ddieterly has quit IRC10:20
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/storyboard: Fix notifications tests docstrings referencing map_resource method  https://review.openstack.org/15440210:21
*** jp_at_hp has joined #openstack-infra10:22
*** salv-orlando has quit IRC10:28
*** boris-42 has joined #openstack-infra10:29
*** pelix has joined #openstack-infra10:29
*** yamahata has quit IRC10:31
*** zhiwei has quit IRC10:31
*** Longgeek has quit IRC10:37
*** Guest82511 has quit IRC10:38
*** ldnunes has joined #openstack-infra10:39
*** cdent has joined #openstack-infra10:44
*** Longgeek has joined #openstack-infra10:45
*** andreaf_ has quit IRC10:47
*** yfried_ is now known as yfried|afk10:55
*** yfried|afk is now known as yfried_10:56
*** otter768 has joined #openstack-infra10:56
ihrachyshkachmouel, dtroyer, ianw, sdague, we need to push the following devstack patches to merge to start fixing stable gates: https://review.openstack.org/#/c/154252/ , https://review.openstack.org/#/c/154217/10:58
openstackgerritIgor Belikov proposed openstack-infra/jenkins-job-builder: Add 'last-completed' to copyartifact 'which-build' values  https://review.openstack.org/15441810:58
*** dprince has joined #openstack-infra10:58
*** koolhead17 has joined #openstack-infra10:59
*** otter768 has quit IRC11:01
*** doude has joined #openstack-infra11:05
*** bradjones has quit IRC11:07
*** markvoelker has joined #openstack-infra11:07
*** salv-orlando has joined #openstack-infra11:07
*** bradjones has joined #openstack-infra11:09
*** salv-orlando has quit IRC11:10
*** sarob has joined #openstack-infra11:11
*** markvoelker has quit IRC11:12
*** ddieterly has joined #openstack-infra11:16
*** sarob has quit IRC11:16
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Add Fuel Elastic Search/Kibana plugin  https://review.openstack.org/15442211:16
*** ddieterly has quit IRC11:20
*** salv-orlando has joined #openstack-infra11:20
*** EmilienM|afk is now known as EmilienM11:21
*** doude has quit IRC11:21
*** yfried_ is now known as yfried|afk11:22
*** yfried|afk is now known as yfried_11:22
*** ihrachyshka has quit IRC11:23
*** ihrachyshka has joined #openstack-infra11:23
*** dims__ has joined #openstack-infra11:23
*** salv-orlando has quit IRC11:24
*** aysyd has joined #openstack-infra11:27
*** jkt has quit IRC11:30
*** jkt has joined #openstack-infra11:30
*** mrmartin has quit IRC11:31
openstackgerritAntoine Musso proposed openstack-infra/zuul: Merger: ensure_cloned() now looks for '.git'  https://review.openstack.org/12343711:37
*** jgallard__ has quit IRC11:43
*** pc_m has joined #openstack-infra11:44
openstackgerritAntoine Musso proposed openstack-infra/zuul: Merger: ensure_cloned() now looks for '.git'  https://review.openstack.org/12343711:45
*** _nadya_ has joined #openstack-infra11:45
*** salv-orlando has joined #openstack-infra11:48
*** hashar has quit IRC11:54
*** claudiub_ has joined #openstack-infra11:57
*** chlong has joined #openstack-infra11:59
*** dimtruck is now known as zz_dimtruck12:03
*** e0ne is now known as e0ne_12:08
*** _nadya_ has quit IRC12:08
*** markvoelker has joined #openstack-infra12:08
*** avozza is now known as zz_avozza12:10
*** _nadya_ has joined #openstack-infra12:11
*** sarob has joined #openstack-infra12:12
*** markvoelker has quit IRC12:13
*** Longgeek has quit IRC12:13
*** ddieterly has joined #openstack-infra12:16
*** sarob has quit IRC12:17
*** e0ne_ has quit IRC12:18
*** ddieterly has quit IRC12:21
*** miqui has quit IRC12:22
*** dprince has quit IRC12:22
*** mpaolino has quit IRC12:24
*** belmoreira has quit IRC12:24
openstackgerritIlia Meerovich proposed openstack-infra/jenkins-job-builder: BugFix: pollscm trigger docs & unittests  https://review.openstack.org/14755512:25
*** changbl has quit IRC12:29
*** dizquierdo has quit IRC12:29
*** tmcpeak has joined #openstack-infra12:33
*** zz_dimtruck is now known as dimtruck12:33
*** yfried_ is now known as yfried|afk12:35
*** yfried|afk is now known as yfried_12:38
*** oomichi_ has quit IRC12:42
*** dimtruck is now known as zz_dimtruck12:43
*** dprince has joined #openstack-infra12:45
*** koolhead17 has quit IRC12:48
*** ilyashakhat has quit IRC12:51
openstackgerritMehdi Abaakouk proposed openstack-infra/project-config: gnocchi: install the git repo  https://review.openstack.org/15439812:51
*** weshay has joined #openstack-infra12:52
*** lnxnut has joined #openstack-infra12:53
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Add Fuel Elastic Search/Kibana plugin  https://review.openstack.org/15442212:53
*** radez_g0n3 is now known as radez12:55
*** denis_makogon has quit IRC12:57
*** otter768 has joined #openstack-infra12:57
*** lnxnut has quit IRC12:58
*** ildikov has quit IRC12:59
*** hashar has joined #openstack-infra13:00
*** otter768 has quit IRC13:02
*** zz_avozza is now known as avozza13:05
*** markvoelker has joined #openstack-infra13:05
*** ilyashakhat has joined #openstack-infra13:10
*** sarob has joined #openstack-infra13:14
*** ildikov has joined #openstack-infra13:15
*** ddieterly has joined #openstack-infra13:17
*** doude has joined #openstack-infra13:18
*** sarob has quit IRC13:18
*** bhunter71 has joined #openstack-infra13:19
*** BharatK has quit IRC13:20
*** skolekonov has joined #openstack-infra13:21
*** ddieterly has quit IRC13:22
*** bswartz has quit IRC13:22
*** _nadya_ has quit IRC13:23
*** Guest82511 has joined #openstack-infra13:24
*** spzala has joined #openstack-infra13:24
*** e0ne has joined #openstack-infra13:25
*** ZZelle has joined #openstack-infra13:26
*** mpaolino has joined #openstack-infra13:26
*** woodster_ has joined #openstack-infra13:36
fungisileht: what you _should_ be asking is why https://pypi.python.org/pypi/python-qpid-proton/0.8-0 doesn't have a files section and why the corresponding version is also not present at https://pypi.python.org/simple/python-qpid-proton/13:38
openstackgerritMerged openstack-infra/storyboard: Fix exception in stories db api  https://review.openstack.org/15396813:38
silehtfungi, oh ok, I will see with the maintainer, thanks13:39
dhellmannclarkb (and others): thanks for fixing up the oslo.versionedobjects repo yesterday13:39
*** bradjones has quit IRC13:40
*** ihrachyshka has quit IRC13:40
*** bknudson has quit IRC13:41
*** vigneshvar has quit IRC13:41
*** mfink_ has joined #openstack-infra13:42
*** ihrachyshka has joined #openstack-infra13:42
*** salv-orlando has quit IRC13:42
*** _nadya_ has joined #openstack-infra13:44
*** armax has joined #openstack-infra13:47
mordredfungi: my god - what the heck version number is 0.8-0?13:47
fungimordred: not one of mine ;)13:49
openstackgerritIhar Hrachyshka proposed openstack-infra/system-config: Collect #openstack-stable logs  https://review.openstack.org/15447413:49
fungimordred: their previous version is 0.7-0 so this seems consistent (for them anyway) but maybe it predates pep 440 enforcement for uploads and so their latest upload failed?13:50
pc_mclarkb: fungi: ping13:50
fungimordred: yeah, 0.7-0 was uploaded back in july13:50
fungisileht: mordred: yeah and https://pypi.python.org/pypi/python-qpid-proton/0.8-0/json just has an empty list for the new release key in the releases dict, so clearly something is very wrong there13:52
fungidstufft: ^ if you're awake already13:53
fungipc_m: what else broke?13:53
pc_mfungi: I had VPN repo functional tests working in experimental queue, and then made them non-voting in check queue with https://review.openstack.org/#/c/152602/13:53
*** carl_baldwin has joined #openstack-infra13:53
silehtfungi, I have ping kgiusti about that too13:54
pc_mfungi: However, recent reviews don't seem to be showing the check-neutron-vpnaas-dsvm-functional running13:54
pc_mfungi: Any idea what may be wrong?13:54
mordredfungi: every day brings me new joy13:54
*** dmsimard_away is now known as dmsimard13:55
fungipc_m: have an example openstack/neutron-vpnaas change where you would have expected to see it appear?13:55
pc_mfungi: Here's one that was updated 2/7 (two days after the merge) https://review.openstack.org/14500513:56
pc_mfungi: patch set 1213:57
*** marun has joined #openstack-infra13:57
*** emagana has joined #openstack-infra13:58
*** radez is now known as radez_g0n313:58
*** nicodemos has joined #openstack-infra13:59
*** yamahata has joined #openstack-infra13:59
*** mrmartin has joined #openstack-infra14:00
*** bknudson has joined #openstack-infra14:00
dstufftfungi: I'm aware now!14:01
*** salv-orlando has joined #openstack-infra14:01
*** mwagner_lap has quit IRC14:02
*** fifieldt has quit IRC14:02
fungipc_m: blame dougwig. looks like he undid your change, probably in a rebase accident, via https://review.openstack.org/15337314:02
fungidstufft: well, that make one of us14:03
* fungi still hasn't grabbed coffee yet14:03
dstufftfungi: well physically awake anyways ;) not mentally14:03
funginew rule: coffee first then irc14:03
mordredfungi: my usual route is "start water boiling for coffee, pick up laptop and start reading scrollback, pour coffee in mug, start typing"14:04
pc_mfungi: Thanks! I was going nuts... Do I push a new review to restore?14:04
fungipc_m: yeah you'll need a new change to fix that14:04
pc_mfungi: Also, I think I'm seeing pylint failures in VPN now... need to check... wondering if the change broke that too.14:05
dstuffthttps://pypi.python.org/pypi/python-qpid-proton/0.8-0 doesn't seem to be have had any files uploaded to PyPI14:05
dstufftthat's why the file list is empty14:05
dstufftnot a pypi bug, just the maintainer didn't upload14:05
*** bcourt has joined #openstack-infra14:05
*** avozza is now known as zz_avozza14:05
*** annegentle has joined #openstack-infra14:06
*** bswartz has joined #openstack-infra14:06
*** elliottk_ has joined #openstack-infra14:07
*** david-lyle_afk is now known as david-lyle14:07
pc_mfungi: Actually, looks like pylint failed for FW, I did a recheck and it passed.14:07
*** elliottk has quit IRC14:07
fungidstufft: thanks. sileht is asking the maintainer for details14:08
anteayapc_m: FW?14:08
anteayaforced water14:08
mordredfungi: testing centos dib image on rax now14:08
*** zz_avozza is now known as avozza14:08
anteayafunctional wawa14:08
mordredfungi: please cross your fingers - if this one fails I'll have to debug systemd ...14:09
*** elliottk_ has left #openstack-infra14:09
*** mjturek1 has joined #openstack-infra14:09
fungipc_m: not sure, though it looked like dougwig's change added those jobs for the first time, so possibly it was already broken14:09
*** elliottk_ has joined #openstack-infra14:10
*** reed has joined #openstack-infra14:10
fungianteaya: presumably firewall (neutron-fwaas specifically since we were previously speaking in context of the neutron-vpnaas project)14:10
*** elliottk has joined #openstack-infra14:10
anteayaokay, surprisingly my first guess has backing14:11
anteayaI was hoping for functional wawa14:11
anteayait just rolls off the tongue14:11
*** mestery has quit IRC14:11
fungimordred: yeah, i've recently been forced to learn all sorts of things i didn't want to know about systemd, since i chose to accept the new default in debian/jessie14:12
*** elliottk_ has left #openstack-infra14:14
*** sarob has joined #openstack-infra14:15
*** GheRivero has quit IRC14:15
anteayawznoinsk: hello14:17
*** lnxnut has joined #openstack-infra14:17
*** GheRivero has joined #openstack-infra14:17
anteayawznoinsk: turns out you are the intel nfv ci rep I have been looking for14:17
openstackgerritPaul Michali proposed openstack-infra/project-config: Restore lost changes to neutron-vpnaas  https://review.openstack.org/15448714:18
*** sarob has quit IRC14:19
pc_mfungi: can you and others review ^^^14:19
*** dustins has joined #openstack-infra14:21
*** dmsimard is now known as dmsimard_away14:22
*** lnxnut has quit IRC14:22
*** radez_g0n3 is now known as radez14:23
*** amotoki has joined #openstack-infra14:23
*** mriedem has joined #openstack-infra14:25
hasharhey guys :-]14:27
sdaguefungi: https://review.openstack.org/#/c/154398 - I'd like to encourage the external plugin migrations, so if you could take a look at that14:27
hasharis anyone familiar with nodepool here?  I am wondering why it needs to maintain a database connection for each node :/14:27
mordredhashar: that seems like a lot of database connections...14:28
anteayajaypipes: so I'm told wznoinsk is the intel nfv ci rep, just found that out. He has been attending meetings so should be around if you have questions. I know you were looking for the intel nfv rep as well14:28
hasharmordred: taken from http://ci.openstack.org/nodepool/installation.html#database14:28
hashar" When adding or deleting nodes, Nodepool will hold open a database connection for each node. "14:29
hasharseems that is just when creating and deleting them :-]14:29
mordredhrm. yes - I can verify this to be the case in our nodepool14:29
openstackgerritMerged openstack-infra/project-config: Remove stable-compat jobs from pycadf  https://review.openstack.org/15426814:29
jaypipesanteaya: ty much!14:30
mordredhashar: I'd have to wait for jeblair for a why14:30
anteayajaypipes: np, keep me informed if you could14:30
kgiustifungi: sileht: pong - seems I've caused you guys quite a bit of grief - sorry.  What can I do to help?14:31
jaypipesanteaya: will do.14:31
anteayajaypipes: thanks14:31
hasharmordred: I guess I can survive; will probably start with 50 VM or 100 connections which is not unreasonable :-]14:31
silehtkgiusti, the 0.8-0 file is missing on pypi14:31
wznoinskhi anteaya, nfv ci is in my timeline for this/next week not yet started14:31
anteayawznoinsk: ah okay14:31
silehtkgiusti, the entry have been created but the file is missing14:32
anteayawznoinsk: jaypipes here he is14:32
anteayawznoinsk: so here is some background14:32
anteayawznoinsk: at paris summit I agreed to work with whoever adrian sent me for nfv testing14:32
openstackgerritMerged openstack-infra/project-config: Add docs jobs for diskimage-builder  https://review.openstack.org/15381014:32
kgiustisileht: ah, so much for "learn by doing" - I obviously screwed something up there.14:32
anteayawznoinsk: so then I was given the names of some folks in china, who are great people14:32
*** ddieterly has joined #openstack-infra14:33
anteayawznoinsk: but as it turns out they aren't working on nfv testing, they are working on the inital foundation pci and sr-iov14:33
kgiustisileht: I'm on the package management page now - it shows both 0.8-0 and 0.7-0 in the release list.14:33
anteayawznoinsk: my understanding is pci is being tested and they are working on sr-iov14:33
wznoinskyes, things has shifted here a bit but the nfv ci is coming very soon14:33
kgiustisileht: is the package bad? or just not visible to you?14:33
wznoinskyes, Yongli is SRIOV CI14:34
anteayawznoinsk: however that doesn't help us as we had initiall asked for nfv, which is what I had agreed to support14:34
anteayawznoinsk: right and they are doing great work14:34
silehtkgiusti, 0.7 have a file attached to the package but not 0.814:34
anteayawznoinsk: and jaypipes who is with nova, who wants to know the status of testing nfv is also interested14:34
anteayawznoinsk: so intel has been doing some great work and is very diligent and that is good14:35
anteayawznoinsk: but my initial agreement was to support nfv testing14:35
anteayaso I need to see some work happening on nfv testing14:35
wznoinskI'm just finishing on our other Networking CI which started to comment last night, NFV to come next14:35
anteayahave a presented my position accurately14:35
anteayawznoinsk: here is the thing14:35
anteayaI have put a lot of work in to helping intel ci reps14:36
anteayaincluding being up in the middle of the night for meetings that suit them14:36
anteayaI need to see some of what I said I would be getting14:36
anteayaI have been working hard for months14:36
*** abhirc has joined #openstack-infra14:36
*** baoli has joined #openstack-infra14:36
anteayasince paris summit14:36
anteayaI need to feel that that dedication is understood apprcicated respected valued and will turn into the results we asked for in the first place14:37
anteayawhich is nfv testing14:37
silehtkgiusti, perhaps you just have to reuploaded the version14:38
anteayawznoinsk: so if things shifted around a bit and nfv got a back burner, perhaps they need to shift back14:38
anteayawznoinsk: since I had hoped by this time we would at least have some sort of prototype14:39
wznoinskAdrian mentioned your dedication and support many times, I believe getting the CI working will not be a hard to accomplish task, is there any priority you'd like to see on top of it? any particular functions tested etc?14:39
anteayaI would like to hear it is a priority for you14:39
anteayayou think it will not be a hard to accomplish task14:39
anteayaI disagree14:39
anteayanoone else is doing it14:39
anteayaand it is entirely new territory14:39
anteayaso I would like to feel someone besides me and nova addresses it with a sense of urgency14:40
wznoinskI mean from an infrastructure point of view (to get the CI working), what tests we want to run inside and how to solve them that's a neverending story14:40
anteayawell it sure is now14:40
anteayasince it isn't even started14:40
anteayaso let's start shall we?14:40
*** baoli has quit IRC14:41
anteayawznoinsk: great14:41
*** baoli has joined #openstack-infra14:41
anteayajaypipes: do let me know when you have time to discuss initial testing steps with wznoinsk14:41
anteayaor point me to the channel log after the discussion14:41
anteayawznoinsk: please co-ordinate with jaypipes14:42
anteayaI'd like to see you had an initial discussion by next week14:42
anteayaat the very least14:42
kgiustisileht: probably, but I'd like to know what I did wrong in the first place - this is my first pypi package, and it's obvious I don't know what I'm doing :(14:42
*** pradk has joined #openstack-infra14:43
*** sabeen1 has joined #openstack-infra14:43
*** dims__ has quit IRC14:43
jaypipesanteaya: will try my best.14:43
*** amrith is now known as _amrith_14:43
anteayajaypipes: thank you14:43
*** dims__ has joined #openstack-infra14:43
*** dims__ has quit IRC14:43
mordredkgiusti: the release has been registered, but there are no files there14:44
silehtkgiusti, I'm not familiar with pypi too, I just known that openstack-infra use https://pypi.python.org/pypi/twine14:44
wznoinskjaypipes: I need to synch up with Adrian on latest discussion about nfv ci here but we can start talking already, what you're upto exactly?14:44
kgiustisileht: I can't even be sure the setup.py is correct - the version #'s aren't right14:44
*** pradk has quit IRC14:44
*** joesavak has joined #openstack-infra14:44
*** dims__ has joined #openstack-infra14:44
mordredkgiusti: when you did the upload, did you run "python setup.py upload" or did you run "python setup.py sdist upload" (the first form will happily upload 0 files for you)14:44
kgiustimordred: I did the second - 'sdist upload'14:45
anteayawznoinsk: do share the channel log in your chat with adrian: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-02-10.log14:45
mordredkgiusti: hrm. weird14:45
*** esmute has quit IRC14:45
kgiustimordred: or at least that's the way I remember it.  But I can't remember what I had for breakfast this morning, so...14:45
*** radez is now known as radez_g0n314:45
*** juice has quit IRC14:46
mordredme either14:46
mordredand I'm still having it :)14:46
kgiustimordred: let me try that again.  But before I do, do you know how I can increment the '-0' portion of the version #?14:46
mordredyou can upload by hand here: https://pypi.python.org/pypi?:action=files&name=python-qpid-proton&version=0.8-014:47
mordredkgiusti: where is your source code and I'll take a look?14:47
kgiusti"Hi, my name's Kenneth, and I'm a C programmer."14:47
*** esmute has joined #openstack-infra14:47
mordredkgiusti: before we started openstack, I was hacking on the internals of a database in C++ with jaypipes - I hear you :)14:48
*** juice has joined #openstack-infra14:48
*** amitgandhinz has joined #openstack-infra14:49
*** tiswanso has joined #openstack-infra14:49
kgiustimordred: be careful of jaypipes - I heard he owns a pug.  Pug owners aren't quite normal.14:49
jaypipesI have 3 pugs actually :)14:49
kgiustimordred: https://github.com/apache/qpid-proton/blob/0.8/proton-c/bindings/python/setup.py.in14:49
kgiustijaypipes: you have me beat - I only have 2.  Primary and backup.14:49
anteayajaypipes also has a pug stroller14:50
anteayathat still kills me14:50
fungisdague: no concerns with that job cloning something from github which it could just as easily pull from our git mirror?14:50
jaypipeshaters! :)14:50
anteayait makes me laugh every time14:50
kgiustijaypipes: pug stroller: +1!14:50
fungisdague: or does the plugin layer have sufficient smarts to clone from the local copy on the worker (in which case it's just fetching updates from github, so not quite as bad)?14:51
mordredkgiusti: so - you want to set: https://github.com/apache/qpid-proton/blob/0.8/proton-c/bindings/python/setup.py.in#L28 if you want to bump the final version ... however, first let me make another suggestion (and I'll follow up with a PR if you want)14:51
mordredkgiusti: which is to change the - here https://github.com/apache/qpid-proton/blob/0.8/proton-c/bindings/python/setup.py.in#L93 to a .14:52
mordredkgiusti: at which point your versions will be compatible with normal python version semantics14:52
mordredkgiusti: happy to send a patch - but it's a one-character change, so I don't know if it's worth me doing it14:53
sdaguefungi: by adding to PROJECTS it will get updated by zuul14:53
kgiustimordred: man I owe you one big $(BEVERAGE) of Vancouver's finest.14:54
*** koolhead17 has joined #openstack-infra14:54
fungisdague: cool, i'll read the source for details on exactly what 'enable_plugin gnocchi https://github.com/stackforge/gnocchi14:54
mordredkgiusti: mmm. I like $(BEVERAGE) !14:54
fungimasterenable_plugin gnocchi https://github.com/stackforge/gnocchi14:54
fungimaster' does14:54
fungigah, my paste has gone off the rails14:54
kgiustimordred: thought so (just a lucky guess) :)14:54
sdaguefungi: ERROR_ON_CLONE is still true the gate runs, so I think it's actually fine14:55
fungisdague: interesting... so the second parameter to enable_plugin is basically a no-op in this case (and could be replaced with garbage)?14:56
*** yfried_ is now known as yfried|afk14:56
*** otter768 has joined #openstack-infra14:58
fungisdague: aha, yep. it's just adding to the GITREPO array which ends up being unused in our ci14:59
sdaguefungi: yeh14:59
fungistill would be nice to avoid future cargo cultists thinking this parameter is doing something real in the job14:59
*** jgallard__ has joined #openstack-infra14:59
sdagueso in the ec2 job definition I expressed that fact with a comment14:59
*** r-daneel has joined #openstack-infra15:00
sdagueonce we've got a few more of these out there, I'm going to write a more complete guide15:00
*** e0ne is now known as e0ne_15:02
*** otter768 has quit IRC15:02
*** sabeen2 has joined #openstack-infra15:03
fungiwould something like UNUSED_ARGUMENT in place of the url work fine with devstack?15:03
*** sabeen1 has quit IRC15:04
*** esmute has quit IRC15:04
openstackgerritMerged openstack-infra/project-config: Add devstack-plugin-sheepdog project into stackforge  https://review.openstack.org/15419915:04
*** radez_g0n3 is now known as radez15:05
openstackgerritMerged openstack-infra/project-config: Add puppet tripleo check tests  https://review.openstack.org/15415215:05
*** BadCub has quit IRC15:05
*** vipul has quit IRC15:05
*** radez is now known as radez_g0n315:05
*** mwagner_lap has joined #openstack-infra15:06
*** esmute has joined #openstack-infra15:06
*** BadCub has joined #openstack-infra15:07
*** vipul has joined #openstack-infra15:07
*** e0ne_ is now known as e0ne15:07
*** yfried|afk is now known as yfried_15:07
anteayaso neutron is discussing creating a neutron lib to address tagging issues with the *aas repos15:07
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells volume exercise bug 1420322  https://review.openstack.org/15450815:07
openstackbug 1420322 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails in volumes exercise with "Server ex-vol-inst not deleted"" [Medium,Triaged] https://launchpad.net/bugs/142032215:07
anteayaif anyone wants to join -neutron and offer an opinion15:08
*** dangers_away is now known as dangers15:08
openstackgerritMerged openstack-infra/project-config: move from macro install-chefdk to chef-bootstrap  https://review.openstack.org/15420715:09
*** emagana has quit IRC15:09
openstackgerritMerged openstack-infra/project-config: Exporting PROJECTS variable for glusterfs job  https://review.openstack.org/15422715:09
openstackgerritMerged openstack-infra/project-config: set -x in cache-testrepository install  https://review.openstack.org/15433715:09
*** emagana has joined #openstack-infra15:09
*** nelsnelson has joined #openstack-infra15:10
*** zz_jgrimm is now known as jgrimm15:10
*** thedodd has joined #openstack-infra15:11
*** nelsnelson has quit IRC15:11
silehtfungi, for your comment here: https://review.openstack.org/#/c/154398/2/jenkins/jobs/gnocchi.yaml, if I use 'enable_plugin gnocchi /opt/git/stackforge/gnocchi' like the glusterfs plugin does, it's OK ?15:13
*** Longgeek has joined #openstack-infra15:14
fungisileht: yeah, that looks like it should be just fine15:14
silehtfungi, thanks15:15
openstackgerritMehdi Abaakouk proposed openstack-infra/project-config: gnocchi: install the git repo  https://review.openstack.org/15439815:15
*** dmsimard_away is now known as dmsimard15:15
*** emagana has quit IRC15:15
fungisileht: alternatively i think we can put something there to make it more obvious it's not referenced at all by the job, but probably not as critical for now15:15
*** Longgeek has quit IRC15:16
*** sarob has joined #openstack-infra15:16
fungisileht: the enable_plugin function just expects something in that parameter, but the array it's added to shouldn't actually be getting values passed to git calls so will never be validated15:16
openstackgerritMerged openstack-infra/project-config: Zuul: Ignore dependencies in merge-check  https://review.openstack.org/15427815:16
jeblairi will restart zuul now for that ^15:17
*** coolsvap is now known as coolsvap_15:17
*** nelsnelson has joined #openstack-infra15:18
*** cnesa has joined #openstack-infra15:18
fungicool, thanks15:18
jeblairfungi: can you look at 154232?15:19
fungithat'll also address jhesketh's comments about all the grey dots on the zuul.openstack.org sitye15:19
fungithe restart will i mean, not 15423215:20
*** sigmavirus24_awa is now known as sigmavirus2415:20
*** sarob has quit IRC15:20
*** gokrokve has joined #openstack-infra15:21
jeblairalso, anyone have any depends-on: changes handy?15:22
fungii have some i think. looking15:22
*** juice has quit IRC15:22
vponomaryovHello. Does anybody know is there requirement to support SQLite in OpenStack projects?15:23
anteayavponomaryov: not that I am aware15:23
fungihrm, maybe not any of mine now... looks like the ones they depend on have merged15:23
*** esmute has quit IRC15:23
anteayavponomaryov: the db of choice is mysql to the best of my knowledge15:23
*** NobodyCam has quit IRC15:24
*** BadCub has quit IRC15:24
*** vipul has quit IRC15:24
*** jesusaurus has quit IRC15:24
fungivponomaryov: sqlite support has mostly been to provide an easy test target so that devs don't need to install a database daemon (though i agree that's a poor excuse for a proper mock)15:25
*** jedimike has joined #openstack-infra15:25
*** esmute has joined #openstack-infra15:25
*** NobodyCam has joined #openstack-infra15:26
anteayathe middle of the night meetings followed by the every other week early morning neutron meeting is not my favourite combination15:26
anteayaI'm going to see if I can get some more sleep15:26
*** juice has joined #openstack-infra15:26
*** BadCub has joined #openstack-infra15:26
*** vipul has joined #openstack-infra15:27
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Minimizing the plugin specific configuration in job definition  https://review.openstack.org/15451815:27
fungigerrit message searches don't do well with depends-on15:27
vponomaryovfungi, anteya: thanks15:27
*** jesusaurus has joined #openstack-infra15:27
jeblairfungi: yeah, i was afraid of that (that's going to make finding reverse dependencies difficult)15:28
jeblairi'll cook up something synthetic15:28
fungijeblair: aha! https://review.openstack.org/140043 looks like a great test15:29
fungijeblair: it has two depends-on headers, one of which references an abandoned change. not sure what that might break ;)15:29
jeblairfungi: the second references a change in the same repo, which is unecessary15:29
*** lnxnut has joined #openstack-infra15:29
*** lnxnut has quit IRC15:29
fungiahh, yep15:30
jeblair(but should still work as long as it does not create a cycle15:30
fungialso there's a semi-circular dep15:30
* mordred throws angry bulls at everyone around15:30
jeblairfungi: how?15:30
* mordred throws angry bulls at self15:30
*** timcline has joined #openstack-infra15:30
*** ghostpl has quit IRC15:30
*** carl_baldwin has quit IRC15:30
*** lnxnut has joined #openstack-infra15:30
mordredsigh. I just spent 2 hours trying to troubleshoot why the systemd stuff wasn't working - and the real problem was that I'd run the wrong image buidl script and had not actually included it in the image at all15:30
*** wenlock has joined #openstack-infra15:31
fungijeblair: 140043 depends on (abandoned)151910 and 151912, then 151912 depends on (abandoned)151910 too15:31
*** koolhead17 has quit IRC15:31
fungiso maybe not circular, just redundant15:31
fungimordred: you should totally blame lennart for that15:32
mordredfungi: it's clearly his fault15:32
fungiit's the fashionable choice, after all15:32
sdaguemordred: wait... you actually concluded that systemd was not the source of some problem you had? ;)15:32
*** tmcpeak has quit IRC15:32
mordredsdague: right?15:32
mordredsdague: I'm as shocked as the next person15:33
*** koolhead17 has joined #openstack-infra15:33
jeblairfungi: i'll wait a few mins for a bare-precise to register, then recheck it15:33
mordredalso - if this does work - I will quickly need to say that the systemd answer to this problem is much clearer than the upstart answer was15:33
mordredalthough both are bonghits amounts of worse than sysvinit woudl be to solve the exact same problem15:33
*** dustins_ has joined #openstack-infra15:34
*** andreykurilin_ has joined #openstack-infra15:34
jeblairwhen i quit firefox, it spends a couple of minutes in iowait.  :(15:34
*** zz_dimtruck is now known as dimtruck15:34
fungijeblair: 144167 and 152508 would make good (and somewhat more typical) examples15:34
*** achanda has joined #openstack-infra15:34
fungijeblair: one has a single dep, one has two deps one of which is merged and the other is not15:35
jeblairfungi: ah 152508 may actually change test results based on it!15:35
*** tsg has joined #openstack-infra15:36
*** dustins has quit IRC15:37
clarkbyou all are awake far too early in the morning15:39
clarkbmordred how was my ansible stuff?15:39
openstackgerritMaru Newby proposed openstack-infra/project-config: Add conditional skips for neutron jobs  https://review.openstack.org/15229815:39
jeblairclarkb: i woke up the moment the zuul config change was approved15:39
jeblairbare-precise | building | 0.7315:40
jeblairjeez ^15:40
fungigate-system-config-puppet-syntax NOT_REGISTERED15:41
fungisaw that15:41
jeblairfungi: ^^15:41
fungijeblair: is that in rax?15:41
jeblairall 3 of the ones building are15:41
fungiit's "waiting" for a floating ip to become available and get assigned i think15:41
fungier, not a floating ip15:41
jrollfloating ip in rax, huh15:41
fungibut an ip address15:41
marunsdague: I'm happy to ensure that testr is installed as part of nodepool prep.  Any pointers as to how to do that?15:42
jeblairi kind of wish that if nova did not have the resources to complete an operation it would just say so15:42
fungiat least we've been told in the past when nova boot calls don't complete for up to an hour in rackspace it's because nova didn't have a global ipv4 address to turn over and assign to the instance15:42
mordredclarkb: sorry - haven't looked yet - I dove straight back in to nodepool-dib15:42
mordredclarkb: let me look now15:42
jeblairmarun: why do you want to install testr in nodepool?15:43
*** sdake has joined #openstack-infra15:43
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Add rally-ci group with label-Verified permission  https://review.openstack.org/15453215:43
marunjeblair: this - https://review.openstack.org/#/c/154299/15:43
*** achanda has quit IRC15:43
sdaguejeblair: I suggested it become part of the base image, instead of installing on the fly15:44
clarkbit should happen as part of the job. we dont install any other python libs globally15:44
marunjeblair: sdague suggested that installing testr on-demand for jobs that don't have devstack install it system-wide was a bad idea.15:44
jeblairsdague: yeah, make sense... but i wonder, why not run it out of the venv?  what needs to use testr?15:44
*** pradk has joined #openstack-infra15:44
jeblairmarun: ^15:44
kgiustisileht: mordred: fyi - looks like python-qpid-proton-0.8.2.tar.gz should be available on pypi.  I'll try running the amqp1 functional tests against it now - fingers (toes, eyes) crossed...15:44
mtreinishjeblair: it's the subunit artifact processing15:44
mordredkgiusti: woohoo!15:44
marunjeblair: clarkb suggested that it would be preferable to do it system-wide15:44
marunjeblair: if we wanted to do it out of a venv if it wasn't installed globally, we'd have to add a config option to d-g so that the d-g wrapper would know which venv to target15:45
clarkbjeblair tempest expects it globally so rather than a bunch of logic to find testr I figured use the expectation it is global15:45
jeblairmtreinish: what artifact processing?15:45
marune.g. /top/stack/new/neutron/.tox/dsvm-functional15:46
marunjeblair: https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L51415:46
silehtkgiusti, cool thanks15:46
marunjeblair: post-processing of testr results for display as html15:46
marunjeblair: tempest already did it, I'm working on allowing non-tempest jobs to use the same code15:46
marunjeblair: we were previously copy-n-pasting into hook scripts15:47
mtreinishjeblair: pulling the most recent subunit stream out of the testrepository, to both store it on the log server and make the html page everyone is fond of15:47
marunjeblair: the catch is that tempest installs testr system-wide, but non-tempest jobs running out of venv's do not15:47
kgiustisileht: mordred: no - thank you folks, I'm the one who messed it up to begin with.  :P   I'll let you know how the functional tests go.15:47
*** sdake has quit IRC15:47
*** sdake has joined #openstack-infra15:48
*** stevemar has joined #openstack-infra15:48
*** hdd has joined #openstack-infra15:49
jeblairmarun, sdague, mtreinish, clarkb: so we probably want it in a system-wide venv, like zuul-cloner and subunit2sql, right?15:49
sdaguejeblair: seems reasonable to me15:49
*** skolekonov has quit IRC15:50
mordredI feel like AFS should be able to solve this, but for the life of me I can't think how15:50
marunjeblair: any pointers as to which repo to propose a change to, to make that happen?15:50
mtreinishjeblair: sure, although we kill the subunit2sql venv after it runs in nodepool. We could just remove the rm and reuse it for this15:51
clarkbjeblair I think that over complicates it but ok15:51
*** naggappan has quit IRC15:51
jeblairclarkb: but it's a python program, it might have dependencies, etc; shouldn't we isolate it from the global environment?15:52
*** rwsu-afk is now known as rwsu15:52
clarkbits been installed by jobs globally for a long time, and that apparently works15:52
jeblairclarkb: which jobs?15:52
clarkbtempest jobs15:52
jeblairclarkb: right, that's not globally though15:52
clarkbyes it is or the current code doesnt work15:53
clarkbmtreinish ^15:53
jeblairclarkb: i mean it's not all jobs15:53
*** dhritishikhar_ has joined #openstack-infra15:53
marunjeblair: all dsvm jobs?15:53
clarkbno just those expecting to run the testr processing15:53
jeblairclarkb: i do agree it would be nice if the job itself did this15:54
*** tsg has quit IRC15:54
*** yfried_ is now known as yfried|afk15:54
mtreinishclarkb: yeah it's being installed by devstack. Although jogo has a patch up to stop that15:54
jeblair(which it already does)15:54
*** ssam2 has quit IRC15:54
jeblairbut how to handle the different places it might be installed?15:54
*** _amrith_ is now known as amrith15:54
*** cnesa has quit IRC15:55
marunjeblair: as previously suggested, a configuration variable indicating a venv to target is one option15:55
jeblairi was initially confused because to date, this code has only run if we're running tempest in d-g15:55
jeblairbut we're talking about other jobs re-using the framework15:56
maruncorrect. re-use instead of copy-n-paste15:56
jeblairmarun: presumably you have a hook function that's something like "cd /path/to/neutron/; tox" right?15:56
marunjeblair: correct15:56
jeblairmarun, clarkb: so that could be followed by "process_results /path/to/neutron/.tox/something" ?15:57
clarkbyou could pass in the env bin path yes15:57
clarkbyou need testr and python to be run out of that15:58
marunjeblair: we are already doing something similar in our copy/paste solution15:58
*** ociuhandu has joined #openstack-infra15:58
*** ghostpl has joined #openstack-infra15:58
marunso if you'd prefer, I could remove the DEVSTACK_GATE_TESTR_ARTIFACT_TARGET option and require that hook scripts call the function to process as suggested15:59
jeblairthat seems fairly reasonable; any reason not to do that?15:59
marunthere is 116:00
jeblair(wow, nodepool gave up on the bare-precise nodes)16:00
marunif we want to change the function, we'd have to coordinate landing the change across an ever-increasing number of jobs16:00
marunif it's localized to d-g, then maintenance would be much simpler16:01
marunI'm not sure how much it will change, though.16:01
jeblairmarun: oh, i was still thinking the function would be in d-g, but would just take some args16:01
*** jaypipes has quit IRC16:01
jeblairmarun: or is that what you meant too, but you meant changing the function signature/api?16:01
marunjeblair: Sure, I'm just saying that we'd have to enforce a hard contract on that function signature16:01
marunjeblair: yeah16:02
marunjeblair: I'm not staying that's a deal-breaker, just that it's something to consider.16:02
fungijeblair: we just grew a ready bare-precise in rax-iad at least16:02
fungijust one, but that's enough to get jobs registered16:02
*** signed8bit has joined #openstack-infra16:03
openstackgerritMerged openstack-infra/elastic-recheck: Add query for cells volume exercise bug 1420322  https://review.openstack.org/15450816:03
openstackbug 1420322 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails in volumes exercise with "Server ex-vol-inst not deleted"" [Medium,Triaged] https://launchpad.net/bugs/142032216:03
fungiwind warning here today. hopefully the power will hold up16:03
marunfungi: whereabouts?16:03
fungimarun: islands off the coast of north carolina16:04
marunfungi: ah, nice.  you live on an island?16:04
jeblairclarkb, sdague: what do you think about that?16:04
*** mtanino has joined #openstack-infra16:04
fungijust finished strapping down or bringing in everything which might go airborne at categoty 1 hurricane windspeeds (though this is a noreaster, so it's just gusts not sustained)16:04
*** esmute has quit IRC16:04
fungimarun: yep16:04
marunfungi: Bridge or ferry/boat to the mainland?16:05
fungimarun: we have a couple bridges... it's only a few miles to the mainland16:05
jeblairfungi: enqueued 14004316:05
clarkbjeblair: being explicit about the python bin path is fine too16:05
fungistill, enough wind to send your trashcan a few blocks away16:06
jeblairfungi: it pulled in both changes it depended on (including the abandoned one)16:06
*** BadCub has quit IRC16:06
fungijeblair: i see that--looking good!16:06
*** BadCub has joined #openstack-infra16:06
*** NobodyCam has quit IRC16:07
marunfungi: I grew up on a peninsula whose access to civilization was a 40m ferry ride. I like living near water.  But it was sheltered ocean - no hurricanes!16:07
*** esmute has joined #openstack-infra16:07
*** NobodyCam has joined #openstack-infra16:07
jeblairfungi: enqueued 152508 and it pulled in the drupal change too16:07
*** achanda has joined #openstack-infra16:07
*** yfried|afk is now known as yfried_16:07
*** che-arne has joined #openstack-infra16:08
*** achanda has quit IRC16:08
*** enikanorov__ has joined #openstack-infra16:09
fungii suppose we could turn off the click to expand on the non-live changes, though maybe that's unnecessary complication in the js16:09
*** enikanorov_ has quit IRC16:09
fungithe grey subway dots will make it pretty obvious anyway i think16:10
fungialso the eta is sort of meaningless for them16:10
marunjeblair, clarkb, sdague: so, our choices are a) set a bin path to testr b) require that hook scripts explicitly call a function?16:10
*** crc32 has joined #openstack-infra16:10
clarkbmarun: and I think a is a subset of b right?16:10
marunclarkb: hmmm, I hadn't thought so. I was thinking that a) was automatic post-processing as we're already doing but setting a bin path via a configuration option16:11
*** jkraj has joined #openstack-infra16:11
jeblairmarun: what's doing the post-processing?16:12
marunjeblair: currently, d-g cleanup.16:12
jeblairoh i thought it only did it for tempest16:12
marunjeblair: or, do you mean executable?16:12
marunjeblair: it did, until I refactored :)16:12
marunjeblair: but it's just a function call, so whether it happens in the post-hook or in cleanup doesn't really matter16:13
jeblairthis is when you learn that my mental model of openstack-infra is kept up to date entirely by code reviews :)16:13
marunjeblair: heh16:13
fungimarun: worth noting we explicitly call testr from an existing (tox-created) venv for post-processing of tox-based jobs, but outside of tox's execution... so running it directly from a venv isn't too unusual for us in other contexts16:13
marunfungi: is that run-tox.sh in project-config?16:13
*** jerryz has quit IRC16:14
clarkbfungi: is ATC based only on changes made to master?16:14
marunfungi: looks like it, ok16:14
clarkbfungi: looking at the EOL icehouse thread and supposedly you don't get atc for stable branch maintenance?16:14
jeblairmarun: ok, so the choices you presented were basically between two ways of providing info to that function -- a) variables b) arguments ?16:14
fungimarun: yep16:14
marunjeblair: hmmm... maybe we should be using run-tox.sh instead?16:15
marunjeblair: it already runs out of a venv16:15
jeblairmarun: if that would work, then maybe so.16:15
marunoh, I remember the sticking point16:15
*** ssam2 has joined #openstack-infra16:15
clarkbmarun: that would be a c) option :) the only thing that would have to be understood there is log scping16:15
*** dustins_ has quit IRC16:15
fungiclarkb: it's based on gerrit owners of changes merged in a specific time period to projects counted in the official set16:15
clarkboh sudo checking too16:15
marunwe'd have to make that configurale16:16
marunso that the sudo-requiring functional jobs wouldn't trip that check16:16
clarkbfungi: so any change within the window regardless of branch (feature/, master, stable/) should be included right?16:16
fungiclarkb: yep see https://git.openstack.org/cgit/openstack-infra/system-config/tree/tools/atc/email_stats.py16:16
*** sarob has joined #openstack-infra16:16
fungiclarkb: the challenge for stable branch maintenance is that the people may be 100% reviewer and 0% change owner16:17
marunclarkb: also, the permissions that tox runs as would need to be configurable iirc16:17
fungiclarkb: hence adding zul to the extra-atcs list in openstack/governance recently16:17
marunI'm not sure anymore why we need to run as different users though :/16:17
clarkbfungi: right because they are backports so author isn't reset16:17
fungiclarkb: well, the stable branch managers are likely not doing the backports16:18
jeblairmarun: in devstack it's so that we can revoke sudo from the user after setup (to ensure rootwrap, etc, are correctly tested)16:18
clarkbmarun: to exercise root wrap irc16:18
fungiclarkb: gerrit owner is set based on who uploads the change i think, but if you're only reviewing backports and not uploading backports, you don't show up in the set of change owners16:18
jeblair(but retain sudo for the devstack-gate script itself)16:19
clarkbfungi: http://lists.openstack.org/pipermail/openstack-dev/2015-February/056457.html is the specific thread entry16:19
marunah, ok.  that makes sense.  so run-tox.sh would have to able to change the user16:19
fungiclarkb: thanks, been behind on e-mail this morning16:19
clarkbfungi: in this particular case the claim appears to be "I have pushed backports but get no recognition"16:19
clarkbfungi: which may be true to stackalytics but should't be for ATC rolls16:19
marunjeblair, clarkb: Ok, I'll propose a patch for project-config that updates run-tox.sh to support what we need for functional testing.16:19
clarkbI have also read reed's ask.o.o answer and don't know how that conclusion was reached16:19
clarkbmarun: kk16:20
*** mpaolino has quit IRC16:20
marunand if that gets accepted I'll revert the d-g processing to target only tempest16:20
marunclarkb, fungi, jeblair: thank you!16:20
* reed on a call ... 16:20
jeblairmarun, clarkb: perhaps what we really need is a refactor of the run-tox script to use functions for these tasks; so we could have unit test and functional test variants of the scripts16:20
clarkbreed: ya its fine we can discuss after16:20
*** ChuckC has quit IRC16:21
jeblairmarun: (these scripts are basically ancient; most of the code dates from when we had 6 projects)16:21
jeblairmaybe 416:21
*** jgallard__ has quit IRC16:21
*** ChuckC has joined #openstack-infra16:21
*** sarob has quit IRC16:21
*** jgallard__ has joined #openstack-infra16:21
*** crc32 has quit IRC16:22
jeblairmarun: and btw, sorry for muddling things, but thank you very much for digging into this :)16:22
*** esmute has quit IRC16:23
*** yfried_ is now known as yfried|afk16:24
*** NobodyCam has quit IRC16:24
*** juice has quit IRC16:24
*** carl_baldwin has joined #openstack-infra16:24
*** abhirc_ has joined #openstack-infra16:24
*** BadCub has quit IRC16:24
*** hashar has quit IRC16:24
*** vipul has quit IRC16:25
*** NobodyCam has joined #openstack-infra16:25
*** juice has joined #openstack-infra16:26
*** esmute has joined #openstack-infra16:26
jeblairfungi: the precise apply job in 152508 has succeeded (it failed in previous runs without crd)16:26
*** vipul has joined #openstack-infra16:26
*** ChuckC_ has joined #openstack-infra16:26
*** BadCub has joined #openstack-infra16:26
*** hashar has joined #openstack-infra16:26
*** yfried|afk is now known as yfried_16:26
fungii think that's a good sign it's working as designed16:27
*** abhirc has quit IRC16:27
*** HeOS has quit IRC16:28
clarkbfungi: jeblair: did that depend on zuul cloner to get everything set up properly?16:28
clarkbeg g-g-p wouldn't have worked right?16:28
marunjeblair: I'll be happy to refactor as part of the work.  :)16:28
fungiclarkb: well, it depends on having working integration test jobs16:28
fungiclarkb: so devstack-gate or zuul-cloner based currently i think16:29
clarkbfungi: ah right its the integration not git set up that really matters16:29
*** chlong has quit IRC16:30
*** yamahata has quit IRC16:30
*** ChuckC has quit IRC16:30
jeblairfungi: right16:31
*** che-arne has quit IRC16:33
*** scheuran has quit IRC16:34
SpamapSclarkb: reading backscroll.. no I have not booted the image I made just yet16:34
*** koolhead17 has quit IRC16:35
*** unicell1 has joined #openstack-infra16:36
*** e0ne is now known as e0ne_16:36
*** hashar has quit IRC16:37
*** cdent_ has joined #openstack-infra16:37
*** unicell has quit IRC16:37
*** unicell1 has quit IRC16:39
*** e0ne_ is now known as e0ne16:39
*** cdent has quit IRC16:40
*** cdent_ has quit IRC16:40
*** cdent has joined #openstack-infra16:41
*** ashleighfarnham has joined #openstack-infra16:42
*** juice has quit IRC16:43
*** NobodyCam has quit IRC16:43
*** jesusaurus has quit IRC16:43
*** esmute has quit IRC16:43
*** dtantsur is now known as dtantsur|afk16:43
*** yfried_ is now known as yfried|afk16:43
*** ayoung_sleep is now known as ayoung16:43
*** BadCub has quit IRC16:44
*** vipul has quit IRC16:44
clarkbreed: to recap there is a claim (on a thread about EOLing icehouse) that one potential reason there is little interest in stable branch maintenance is that doing so does not grant you ATC status16:44
reedI saw that thread, I need to gather some details16:44
clarkbreed: fungi has refuted this here and my reading of the ATC generation script seems to confirm. Probably best if you or fungi respond to the list with a definitive answer16:44
*** esmute has joined #openstack-infra16:45
reedclarkb, that's my thought too, that he's wrong and fungi is right16:45
*** BadCub has joined #openstack-infra16:45
*** NobodyCam has joined #openstack-infra16:45
clarkbreed: I mentioned you because our ask.o.o answer about atc stuff was linked as evidence that only master counts but I cannot get that out of the ask thread when I read it16:45
*** juice has joined #openstack-infra16:45
reedclarkb, thanks, I thought my English failed there16:45
fungireed: clarkb: i've just replied on that thread and will reply on the ask.o.o entry as well16:46
*** vipul has joined #openstack-infra16:46
clarkbfungi: awesome thanks a bunch16:46
reedthe other thing he says wrong is that analytics looks at trunk only... not sure about stackalytics but activity.o.o does all activity in the whole repo16:46
*** jesusaurus has joined #openstack-infra16:46
reedwill reply in a bit16:46
clarkbreed: I think the analytics thing is a slight issue. basically when you backport a change the original author retains authorship16:46
* reed stuck in some administrativia still16:46
clarkbreed: so the backporter isn't credited as author. They are the change owner and committer which is different and if analytics only focuses on the author that will happen16:47
reedclarkb, that's a more interesting issue indeed, like counting co-author16:47
*** doude has quit IRC16:47
*** shashankhegde has joined #openstack-infra16:48
*** bdpayne has joined #openstack-infra16:49
fungiit's annoying that askbot comments are limited and you can't follow up to your comment with a second comment to increase that limit16:49
*** ssam2 has quit IRC16:49
*** tiswanso has quit IRC16:49
openstackgerritMerged openstack/requirements: Update gabbi to 0.6.0  https://review.openstack.org/15330116:50
fungilength-limited i mean16:50
*** gyee has joined #openstack-infra16:51
*** tqtran has joined #openstack-infra16:51
reedfungi, the idea is that you provide an answer instead or (with enough karma) fix someone else's answer16:51
*** tiswanso_ has joined #openstack-infra16:51
reedcomments are useless16:51
pc_mclarkb: Can you review https://review.openstack.org/#/c/154487/? This is to restore an accidentally reverted change.16:52
fungireed: agreed, except when people misinterpret comments into other contexts and there's not enough space to provide clear rebuttals to them16:52
reedlet's clarify the answer then16:52
openstackgerritDerek Higgins proposed openstack-infra/system-config: Set hp1 max-node to 0  https://review.openstack.org/15456116:53
fungireed: i think the answer is perfectly clear, and the comment thread which ensued after it is where misunderstandings are arising from16:53
reedexactly my point: comments are useless :)16:53
reeddelete them :)16:53
openstackgerritDerek Higgins proposed openstack-infra/system-config: Set hp1 max-servers: to 0  https://review.openstack.org/15456116:53
*** yfried|afk is now known as yfried_16:53
*** bswartz has quit IRC16:53
clarkbpc_m: done. How did those changes not conflict? or did they and the rebase was not proper?16:54
fungireed: i'm pretty sure that all of kevin's misunderstandings from that are not based at all on the answer, but instead on the commentary dialogue (no doubt he googled until he found some discussion which sounded vague enough to support his assumption)16:54
derekhHi, could somebody take a look at https://review.openstack.org/154561 to stop using hp1 while its failing tests16:54
*** shashankhegde has quit IRC16:55
*** arxcruz has quit IRC16:55
clarkbyolanda: still around? I left a couple comments on 13740116:55
*** ihrachyshka has quit IRC16:55
pc_mclarkb: Thanks! Not sure.16:55
pc_mdougwig: ^^16:55
fungiclarkb: i think the rebase reverted the conflict rather than accommodate it16:55
clarkbfungi: gotcha16:56
*** afazekas has quit IRC16:56
*** coolsvap_ is now known as coolsvap16:56
openstackgerritClark Boylan proposed openstack-infra/puppet-project_config: Provide way to specify repo ref  https://review.openstack.org/15430016:58
clarkbjeblair: ^ that should make puppet lint happy16:58
jeblairfungi, mrmartin: 152508 passes tests :)16:59
reedfungi, I edited the answer to include your comment16:59
*** otter768 has joined #openstack-infra16:59
*** luqas has quit IRC16:59
jd__knock knock! hello there – could we fill this https://review.openstack.org/#/admin/groups/600,members group with oslo-core at least? cc dims__ dhellmann17:00
*** sarob has joined #openstack-infra17:00
fungireed: wfm. hopefully that will help17:00
dims__you beat me jd__17:00
fungijd__: in the pas we've just added dhellmann and let him set it up however he wants from there17:00
fungijd__: dims__: which i've now done17:01
*** bswartz has joined #openstack-infra17:01
*** ssam2 has joined #openstack-infra17:02
clarkbmordred: https://review.openstack.org/#/c/154304/ https://review.openstack.org/#/c/154305/ and https://review.openstack.org/154300 :)17:02
jd__fungi: thanks17:03
*** otter768 has quit IRC17:03
clarkbmordred: the change to the puppet-ansible repo is the one I am least sure about17:03
*** koolhead17 has joined #openstack-infra17:03
mordredclarkb: yah - I have some thoughts but am in a meeting - I promise - responding is next up17:04
dims__fungi: thanks!17:05
*** e0ne has quit IRC17:06
*** nijaba_ is now known as nijaba17:06
*** yamahata has joined #openstack-infra17:06
openstackgerritMerged openstack-infra/project-config: Restore lost changes to neutron-vpnaas  https://review.openstack.org/15448717:06
clarkbnibalizer: can you explain how the thing in 153823 makes testing harder for non rooters?17:06
*** jistr has quit IRC17:07
*** jcoufal has quit IRC17:07
clarkbnibalizer: is it because puppetmaster is set? When I test locally I always use puppet apply17:07
clarkbnibalizer: so not sure I understand the issue there17:07
clarkbderekh: +217:08
derekhclarkb, thanks17:08
*** yamahata has quit IRC17:09
*** gokrokve_ has joined #openstack-infra17:09
nibalizerclarkb: it creates a lock between site.pp and that file17:09
openstackgerritMerged openstack-infra/system-config: Zuul status: Use a grey icon for non-live changes  https://review.openstack.org/15423217:09
nibalizerits better now that its certname instead of fqdn17:09
clarkbnibalizer: I still do not follow17:09
clarkbnibalizer: a lock?17:09
*** yamahata has joined #openstack-infra17:09
*** MaxV has quit IRC17:09
clarkbnibalizer: I don't doubt there is something that makes live difficult I just have no understanding of it with my use of puppet apply17:09
mordredjust so that everyone is aware - the only thing worse than systemd is a system where SOME things are in systemd and SOME things are not17:10
nibalizeryea puppet apply works17:10
*** ChuckC has joined #openstack-infra17:10
*** MaxV has joined #openstack-infra17:10
nibalizeri prefer to test with a master17:10
nibalizerbecause then i can have my master set up and massage code there, then spin up and down clients as I want17:11
clarkbassuming you have a local master how does certname make life difficult? you can use the same certnames that we use17:11
clarkbthen it should just work (tm) right?17:11
nibalizerya i think you're right, there might not actually be a problem now17:12
nibalizermy problems are all from when that was fqdn17:12
*** gokrokve has quit IRC17:12
nibalizeri honestly haven't spun up my infra testing environment in some time17:13
nibalizerbeen busy with other stuff17:13
*** Chuck__ has joined #openstack-infra17:13
*** ChuckC_ has quit IRC17:14
nibalizeralso that line is ... weird17:14
nibalizeri guess its okay, but its asserting what is already true, if that makes sense?17:14
*** MaxV has quit IRC17:15
dhellmannfungi: could you add me to oslo-versionedobjects-release so I can update that one, too, please? https://review.openstack.org/#/admin/groups/601,members17:15
*** dustins has joined #openstack-infra17:15
fungidhellmann: done17:15
*** jkraj has quit IRC17:15
dhellmannfungi: thanks!17:16
*** ChuckC has quit IRC17:17
*** kgiusti has quit IRC17:19
*** hashar has joined #openstack-infra17:19
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Add rally-ci group with label-Verified permission  https://review.openstack.org/15453217:20
*** amotoki has quit IRC17:21
*** jpich has quit IRC17:22
*** unicell has joined #openstack-infra17:22
*** wznoinsk_ has joined #openstack-infra17:22
*** coolsvap is now known as coolsvap_17:23
*** juice has quit IRC17:23
*** wznoinsk is now known as wznoinski__17:25
*** esmute has quit IRC17:25
*** wznoinski__ is now known as wznoinsk__17:25
*** esmute has joined #openstack-infra17:25
asselin_anyone seen this? not able to start nodepool: http://paste.openstack.org/show/170829/17:25
*** bswartz has quit IRC17:25
*** juice has joined #openstack-infra17:25
*** wznoinsk_ is now known as wznoinsk17:27
clarkbnibalizer ya the first time we run we pass the certname as an arg iirc17:28
clarkbso its true but not otherwise configured. this configures it17:28
clarkbasselin_ upgrade your version of lockfile17:29
*** mpaolino has joined #openstack-infra17:29
*** wznoinsk has quit IRC17:29
*** jedimike has quit IRC17:29
mrmartinjeblair: 152508 is testing together with dependent patch?17:30
pleia2good morning17:30
openstackgerritMerged openstack-infra/storyboard: Fix notifications tests docstrings referencing map_resource method  https://review.openstack.org/15440217:31
*** wznoinsk has joined #openstack-infra17:31
*** annegentle has quit IRC17:32
asselin_clarkb, thanks17:33
clarkbasselin_ fix is in review. we were depending on lockfile instead of having pytjon-daemon pull it in17:33
clarkbthis confused pip so fix is remove lockfile from our dep list17:33
*** tiswanso_ has quit IRC17:33
asselin_ok i see17:34
*** sputnik13 has joined #openstack-infra17:34
*** wznoinsk has quit IRC17:34
*** ildikov has quit IRC17:34
*** wznoinsk has joined #openstack-infra17:34
*** tiswanso has joined #openstack-infra17:34
*** yfried_ is now known as yfried|afk17:34
*** jlibosva has quit IRC17:35
jeblairmrmartin: yep!17:35
*** ianw has quit IRC17:39
*** emagana has joined #openstack-infra17:40
fungijust to bring over the discussion from #openstack-qa, it looks like we will need to manually merge stable/juno global requirements pin 151770 since it won't pass grenade jobs without stable/juno grenade fix 153702 (which in turn depends on the reqs pin being in place)17:41
fungimtreinish: jogo: ^17:41
fungiany objection to me exercising gerrit superpowers to merge 151770 now?17:41
*** lnxnut_ has joined #openstack-infra17:41
*** emagana_ has joined #openstack-infra17:41
fungiadam_g: ^17:41
*** bswartz has joined #openstack-infra17:42
*** mpaolino has quit IRC17:42
*** esmute has quit IRC17:42
*** BadCub has quit IRC17:43
*** vipul has quit IRC17:43
*** juice has quit IRC17:43
*** jesusaurus has quit IRC17:44
mtreinishfungi: I support you excercising you powers17:44
*** esmute has joined #openstack-infra17:44
* mtreinish shields his eyes17:44
*** lnxnut has quit IRC17:44
*** shashankhegde has joined #openstack-infra17:44
*** juice has joined #openstack-infra17:44
*** emagana has quit IRC17:45
*** BadCub has joined #openstack-infra17:45
* fungi tries to use his powers only for good, less often for personal amusement17:45
*** vipul has joined #openstack-infra17:45
*** koolhead_ has joined #openstack-infra17:46
*** ghostpl has quit IRC17:46
*** jesusaurus has joined #openstack-infra17:46
*** Ryan_Lane has joined #openstack-infra17:46
*** ashleighfarnham has quit IRC17:46
*** Chuck__ is now known as ChuckC17:48
*** koolhead17 has quit IRC17:48
*** ashleighfarnham has joined #openstack-infra17:49
clarkbfungi the superhero we deserve17:49
clarkbor is it we dont deserve? english is hard17:50
*** e0ne has joined #openstack-infra17:51
openstackgerritMerged openstack-infra/project-config: gnocchi: install the git repo  https://review.openstack.org/15439817:52
*** melwitt has joined #openstack-infra17:52
fungihearing no objections, i don the cape17:53
*** avozza is now known as zz_avozza17:54
fungimtreinish: jogo: adam_g: done. should be safe to put 153702 through normally now17:54
jogofungi: thanks17:55
*** dhritishikhar_ has quit IRC17:55
*** MaxV has joined #openstack-infra17:56
*** bhuvan has joined #openstack-infra17:56
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Removed unsupported python versions from setup.cfg  https://review.openstack.org/15458217:58
*** changbl has joined #openstack-infra17:58
*** derekh has quit IRC17:59
*** wznoinsk_mobl has joined #openstack-infra18:01
*** dannywilson has joined #openstack-infra18:01
*** dannywilson has quit IRC18:02
*** signed8bit has quit IRC18:02
*** dannywilson has joined #openstack-infra18:02
*** _nadya_ has quit IRC18:03
david-lylejeblair, clarkb or fungi, some time ago the previous owner of the openstack-horizon channel added a bunch of flags on that room, I was not so lucky, can one of you please give me the operator status at least18:04
openstackgerritMerged openstack-infra/project-config: Adds networking-hyperv in StackForge  https://review.openstack.org/15369118:04
david-lyle*flags for you on that room18:04
*** yfried|afk is now known as yfried_18:04
*** sdake has quit IRC18:04
*** wznoinsk has quit IRC18:04
*** ashleighfarnham has quit IRC18:05
fungidavid-lyle: sure, freenode confirms you're logged in, so i'm pretty sure i'm talking to the real you ;)18:05
*** shashankhegde has quit IRC18:05
david-lylefungi: most of the time18:06
*** patrickeast has joined #openstack-infra18:06
david-lylefungi: thank you18:06
*** gokrokve_ has quit IRC18:07
fungidavid-lyle: i added the same flags for you that our general irc operators have on that channel18:08
fungidavid-lyle: though you had a couple of other flags already (O and V) which i left on as well18:09
openstackgerritLance Bragstad proposed openstack/requirements: Add python-keyczar to global-requirements.py  https://review.openstack.org/15459018:09
david-lylefungi, perfect. 3 releases in, I can change the channel topic, progress as promised18:09
*** harlowja has joined #openstack-infra18:10
fungicoming along nicely! ;)18:10
*** kgiusti has joined #openstack-infra18:11
*** wznoinsk_mobl has quit IRC18:11
*** hashar has quit IRC18:11
*** andreykurilin_ has quit IRC18:12
*** thedodd has quit IRC18:13
*** abhirc_ has quit IRC18:14
*** abhirc has joined #openstack-infra18:14
*** tsg has joined #openstack-infra18:14
*** jaypipes has joined #openstack-infra18:15
*** ihrachyshka has joined #openstack-infra18:15
*** dalgaaf has quit IRC18:17
*** achanda has joined #openstack-infra18:18
*** vigneshvar has joined #openstack-infra18:19
*** thedodd has joined #openstack-infra18:19
clarkbfungi k9 exploded on one of your messages to the eol threads. so weird18:20
fungiasploded how?18:20
* fungi tries to stick to plain text e-mail... i don't even think i've crypto-signed any messages to that thread18:21
clarkb(no subject) and no content but apparently enough header to associate with the thread18:21
*** openstackgerrit has quit IRC18:22
*** openstackgerrit has joined #openstack-infra18:22
*** esmute has quit IRC18:23
*** thedodd has quit IRC18:24
*** juice has quit IRC18:24
*** yfried_ is now known as yfried|afk18:24
*** thedodd has joined #openstack-infra18:25
*** Ala has quit IRC18:26
mordredjeblair, clarkb: ok. I have ubuntu working on rackspace but can't get centos working. for nodepool-dib that's obviously not a big deal18:26
*** pelix has quit IRC18:26
mordredHOWEVER - the reason I can't get it working is systemd18:26
mordredso there is a problem to solve at some point if we want to add fedora or debian nodes18:26
mordredthat said - centos7 has a weird hybrid thing going on18:26
*** esmute has joined #openstack-infra18:26
mordredso it's quite possible that it'll be less bonghits on debian/fedora18:26
*** juice has joined #openstack-infra18:26
mordredso I might try them next18:27
mordred(I don't want to declare that it's usable knowing that there is a looming broken thing)18:27
greghaynesI know some of the tripleo rhers were doing centos7 work on dib...18:27
greghaynesso in theory it should work18:27
*** hdd has quit IRC18:28
*** jp_at_hp has quit IRC18:29
SpamapSI think it's more that centos7/rhel7's networking works very differently than fedora/debian's networking with systemd.18:29
greghaynesaye, fun18:29
mordredgreghaynes: this is also about getting network info from the rax config-drive ... so it's an extra special beasty18:30
mordredcentos7 itself is working just fine18:30
mordredalso - centos7 does not have "ifdown -a; ifup -a"18:30
mordredyou have to specify interfaces18:30
*** crc32 has joined #openstack-infra18:31
greghaynesIve been running into an error when making nodepool trusty images where prepare_tempest_testrepository.py fails with chown: cannot access ‘/opt/git/openstack/tempest/preseed-streams’: No such file or directory18:31
greghaynesAIUI preseed streams is a testr thing... theres a severe lack of debug info though18:31
*** shashankhegde has joined #openstack-infra18:32
SpamapSmordred: 'service networking restart' doesn't work?18:32
SpamapSthat used to be a thing anyway18:32
*** jgallard__ has quit IRC18:33
*** tsg has quit IRC18:35
openstackgerritMORITA Kazutaka proposed openstack-infra/project-config: Add test job for sheepdog devstack plugin  https://review.openstack.org/15460518:36
*** crc32 has quit IRC18:38
openstackgerritgordon chung proposed openstack/requirements: add ceilometermiddleware  https://review.openstack.org/15460718:38
clarkbgreghaynes mtreinish18:39
pleia2clarkb: so I'm back to trying to figure out what the intent is for all these devstack-gate jobs, what is the goal here, vote on the -icehouse but not voting on -juno? back in september we were voting on icehouse but not havana: https://review.openstack.org/#/c/122885/ (see Jenkins' check build on patchset 2)18:40
*** yfried|afk is now known as yfried_18:40
fungiclarkb: the three messages i sent related to those threads are http://lists.openstack.org/pipermail/openstack-dev/2015-February/056465.html 056472.html and 056476.html any idea which one of those is broken in k9?18:40
* greghaynes summons an mtreinish18:40
clarkbpleia2 I think tempest wants them voting and d-g does not18:40
clarkbpleia2 so likely we need 2 sets of jobs18:41
pleia2clarkb: for both icehouse and juno, or?18:41
clarkbfungi the one about stable tempest helping defcore18:41
clarkbpleia2 ya for both branches18:41
pleia2ok, thanks, I'll see where this goes and change it again if we learn differently18:42
fungigreghaynes: make sure you didn't accidentally run it once as root and pollute the ownership of stuff in your cache18:42
*** ssam2 has quit IRC18:43
*** BadCub has quit IRC18:43
fungigreghaynes: we were getting errors which looked basically like that some of the time which turned out to be local cache ownership issues18:43
greghaynesfungi: this is as the result of a nodepool dib image build, so I hope tats not an issue ;)18:43
*** vipul has quit IRC18:43
*** NobodyCam has quit IRC18:43
*** BadCub has joined #openstack-infra18:44
fungigreghaynes: if someone previously did a 'sudo nodepool image-create foo' then parts of the cache could be root-owned and inaccessible to the nodepool user which nodepool is trying that as when run from its periodic scheduler18:44
*** jesusaurus has quit IRC18:44
*** esmute has quit IRC18:44
greghaynesaye :(18:44
*** vipul has joined #openstack-infra18:44
*** NobodyCam has joined #openstack-infra18:44
fungigreghaynes: try to chown -R nodepool:nodepool /your/cache/dir18:44
*** Sukhdev has joined #openstack-infra18:45
fungiand see if that fixes it18:45
pc_mclarkb: ping18:45
*** esmute has joined #openstack-infra18:45
greghayneswow, there is definitely things in the cache not owned by me18:45
greghaynesthats a bad sign18:45
pc_mclarkb: Had a question on coverage testing... Is there a way to collect coverage info for dsvm-functional modules too?18:45
*** mgariepy has left #openstack-infra18:46
*** gokrokve has joined #openstack-infra18:47
pc_mclarkb: I ran "tox -e cover" and don't have an omit clause. The functional modules all show 0 lines coverage.18:47
clarkbpc_m run your tests and services under coverage18:47
clarkbtox -ecover is purely for whatever is run by tox -epy2718:48
pc_mclarkb: Any pointers on how that is done? link?18:48
clarkbpc_m http://nedbatchelder.com/code/coverage/ is the coverage docs18:49
pc_mclarkb: thanks!18:49
clarkbbut basically instead of python somescript you run coverage somescript18:49
fungipc_m: you might want to engage #openstack-qa but in general the topic of trying to get coverage from functional/integration tests has come up many times, several solutions have been tried with a variety of trade-offs, and is generally non-trivial18:49
pc_mfungi: Ah, good to know. I was wondering how to run the tests for those functional modules.18:50
*** markus_z has quit IRC18:51
pc_mfungi: Is there any documentation on the methods that you know of?18:51
clarkbI am going to miss first bit of meeting looks like. errand is running long18:52
fungipc_m: for unit tests it's easier because execution happens in a sort of rarified environment, but threading is a major challenge18:52
fungipc_m: looks like there may be some helpful hints in https://bitbucket.org/ned/coveragepy/issue/14918:54
pc_mfungi: cool18:55
fungipc_m: anyway my memory's fuzzy but i know the qa team have discussed it in prior summit sessions and some of them likely recall the specific challenges18:56
*** dmsimard has quit IRC18:56
anteayamorning jhesketh18:57
mtreinishgreghaynes: what's up?18:58
fungipleia2: have a change proposed yet for adding your user to openstack_project::base and our global site manifest?18:59
*** tsg has joined #openstack-infra18:59
*** _nadya_ has joined #openstack-infra18:59
morganfainberganteaya, FYI our meeting ran short - i know you use us as a reminder for -infra18:59
*** ianw has joined #openstack-infra18:59
morganfainberganteaya, ;)18:59
fungimeeting time18:59
fungiin 10 secs18:59
anteayamorganfainberg: :) I saw you earlier19:00
*** otter768 has joined #openstack-infra19:00
*** emagana_ has quit IRC19:00
anteayamorganfainberg: and thanks19:00
*** wznoinsk_mobl has joined #openstack-infra19:00
pleia2fungi: nope!19:00
*** gyee has quit IRC19:00
openstackgerritKhai Do proposed openstack-infra/system-config: Add the luanchpadlib OS package to review-dev.o.o  https://review.openstack.org/15461219:00
*** esmute has quit IRC19:01
fungipleia2: i guess that's the next item for your to do list ;)19:01
*** tiswanso has quit IRC19:01
pleia2fungi: sure thing :)19:01
mtreinishgreghaynes: oh, you're hitting issues with the preseed-testrepository script in nodepool19:01
mtreinishdo you have a stack trace19:01
greghaynesmtreinish: hey, yep19:02
greghaynesmtreinish: no stack trace19:02
*** vipul has quit IRC19:02
*** NobodyCam has quit IRC19:02
*** BadCub has quit IRC19:02
*** tiswanso_ has joined #openstack-infra19:02
greghaynesmtreinish: it fails on chowning that dir because it doesnt exist, the prepare_tempest_repository script I assume is what should make it?19:02
*** dims__ has quit IRC19:02
*** dims__ has joined #openstack-infra19:03
*** ashleighfarnham has joined #openstack-infra19:03
mtreinishgreghaynes: yeah the script makes it: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/prepare_tempest_testrepository.py#n5219:03
*** esmute has joined #openstack-infra19:03
*** NobodyCam has joined #openstack-infra19:03
*** ivar-lazzaro has joined #openstack-infra19:03
*** dims_ has joined #openstack-infra19:04
greghaynesah! I notice there is not checking of return code there19:04
* greghaynes adds this19:04
*** BadCub has joined #openstack-infra19:04
*** bradjones has joined #openstack-infra19:04
*** ashleighfarnham has quit IRC19:04
*** otter768 has quit IRC19:04
*** vipul has joined #openstack-infra19:05
*** ivar-lazzaro has quit IRC19:05
*** tobiash_ has joined #openstack-infra19:06
*** ivar-lazzaro has joined #openstack-infra19:06
*** tobiash has quit IRC19:06
fungipsh, error handling... who needs that?!?19:06
mordredfungi: NOBODY19:06
*** wznoinsk has joined #openstack-infra19:07
*** dims__ has quit IRC19:07
mtreinishgreghaynes: is there a return code for os.mkdir, I thought it just raised an IOError19:08
greghaynesyou may be right, is that all that is happening with that call?19:08
*** dmorita has joined #openstack-infra19:09
greghaynesanyhow, well find out momentarially when my build fails :)19:09
openstackgerritSteve Martinelli proposed openstack-infra/devstack-gate: Add oslo.policy to devstack-vm-gate-wrap.sh  https://review.openstack.org/15461519:09
*** _nadya_ has quit IRC19:09
*** ashleighfarnham has joined #openstack-infra19:09
*** wznoinsk_mobl has quit IRC19:10
sdaguejeblair: so I a funny thing happens with the depends on - http://status.openstack.org/zuul/19:10
sdaguesee 154216,3 and 154575,2 in that list19:11
*** EmilienM is now known as EmilienM|afk19:11
*** jesusaurus has joined #openstack-infra19:11
fungisdague: i think https://review.openstack.org/154232 hasn't taken effect yet to make that a little more obvious19:12
yolandaclarkb, i answered you as well19:12
yolandai want to talk with you about the testing19:12
*** tiswanso_ has quit IRC19:12
*** shashankhegde has quit IRC19:13
sdaguefungi: so, not quite. 154575,2 shows up as both a live and non-live change19:13
fungisdague: oh, actually after i force-reloaded my cache of the javascript for that updated19:13
fungisdague: yeah, it's both a live and non-live change19:14
sdagueshouldn't it only be one?19:14
fungisdague: it's being tested itself as a live change, but also appears as a non-live change applied while testing 154216,319:14
sdagueok, I guess I assumed that it would only show up once19:15
fungisdague: i don't think the goal was to collapse those, but maybe that's a good improvement to consider19:15
jeblairsdague: yeah, that might be slightly confusing from ux, but is sort of comforting when you think about what might happen if the parent fails; at least this makes it clear that no matter the outcome, the child will still be tested19:15
jheskethsdague: http://logs.openstack.org/92/154292/1/check/check-dg-tempest-dsvm-full/ddceb0d/index.html19:16
jheskethsdague: so I think the most obvious thing that is missing is the documentation19:16
sdaguejeblair: yep, I get it now. I might try to mock up some different visual metaphor for this so that it doesn't confuse people19:16
*** tiswanso_ has joined #openstack-infra19:16
sdaguejhesketh: are all those changes coming from swift?19:17
sdaguelike all the logs?19:17
jeblairyeah.  also, it looks like grey dots are in now, so if you don't have those, you should reload19:17
jheskethsdague: the link I sent you is since index.html isn't stored locally. Once you browse to a log, you'll probably need to append ?source=swift19:17
greghaynesmtreinish: success! now I have a traceback :)19:17
openstackgerritIan Wienand proposed openstack-infra/nodepool: Add generate-log-config command  https://review.openstack.org/15390419:17
mtreinishgreghaynes: what's it say?19:18
*** dims__ has joined #openstack-infra19:18
*** shashankhegde has joined #openstack-infra19:18
*** mrunge has quit IRC19:18
greghaynesmtreinish: http://paste.ubuntu.com/10162632/19:19
sdaguejhesketh: is the index static or or dynamically generated, I forget?19:20
jheskethsdague: static19:20
*** elliottk has quit IRC19:20
sdaguedo you have an instance of - http://logs.openstack.org/92/154292/1/check/check-dg-tempest-dsvm-full/ddceb0d/logs/  that's off swift?19:21
*** dims_ has quit IRC19:21
greghaynesmtreinish: gonna need some debugging, but I at least know where to look now19:22
mtreinishgreghaynes: isn't that because of the paren around stats, out? if it was a tuple response you wouldn't need them19:22
mtreinishalso that looks like the script is out of date19:22
mtreinishbecause it's trying to run testr as part of the image create (which doesn't work for a number of reasons)19:23
jheskethsdague: yes, append index.html: http://logs.openstack.org/92/154292/1/check/check-dg-tempest-dsvm-full/ddceb0d/logs/index.html19:23
*** esmute has quit IRC19:23
mtreinishgreghaynes: https://review.openstack.org/#/c/151732/19:23
greghaynesyou are correct, I am out of date19:24
jheskethsdague: which will happen automatically if the local copy isn't there. Also will be slightly faster if you could please see my comments on https://review.openstack.org/#/c/141260/\19:24
clarkbyolanda we should build images in tests the same way we build in prod19:24
patrickeasthas anyone run into problems with python-daemon 2.0.5? i updated my zuul and nodepool nodes today and ran into errors like http://paste.openstack.org/show/170892/, i saw this change https://github.com/openstack-infra/zuul/commit/7e11bd727606afa5d55cddfa07439ae646e0f330 and tried to roll it back to 2.0.4 and things work better now19:25
patrickeastis this a known issue?19:25
yolandaclarkb, what do you mean? about the image testing spec?19:25
*** esmute has joined #openstack-infra19:26
sdaguejhesketh: ok, so feedback. I find the icons extremely helpful in the apache view, the text content type just feels busy. K / M vs. KiB MiB would be nice to fix. And the size column needs right justification because it's a list of numbers.19:27
clarkbyolanda just that we shouldnt special case the fake image builds that way19:27
asselin__patrickeast, <clarkb> asselin_ upgrade your version of lockfile19:28
*** andreykurilin_ has joined #openstack-infra19:28
asselin__patrickeast, see the scrollback19:28
clarkbyolanda can you have the tests poll the db?19:28
yolandaah, clarkb, i know, i added a comment, because if i remove builds from the taskmanager, then tests fail19:28
yolandadidn't want to update the testing behaviour19:28
yolandaso that's why i wanted feedback19:28
jheskethsdague: okay, that shouldn't be too difficult19:28
patrickeastasselin__: awesome, thanks!19:29
yolandaso it's fine for me to poll database, but then it will be just an special case for it? or shall we update tests generally?19:29
clarkbyes tests will have to be updated19:29
sdaguejhesketh: the help text, might be nice to add, though honestly I don't know how many folks are really using it19:29
sdagueso I'd say let people complain19:29
fungipatrickeast: asselin__: proposed fix for that is https://review.openstack.org/14825219:30
fungiwhich should cause daemon to pull in an appropriately new lockfile release on its own19:30
sdagueor... can we enable server side includes and just pull it in that way?19:30
yolandaclarkb, ok.. so hall we create a different change for that? not sure if we want to mess it19:30
yolandafirst update tests to poll database instead of relying on taskmanager in an independent change19:30
patrickeastfungi: oo nice19:30
clarkbno I think it should be part of the change that changes image building19:31
jheskethsdague: I think it makes more sense to include the help text staticly so that as the logs/docs change the old text is still relevent19:31
mtreinishsdague: ah, all that time I spent decoding the filenames is going to disappear...19:31
clarkbsince they are couples19:31
asselin__fungi, wierd, my issue was with nodepool...perhaps another similar fix is needed?19:31
*** andreykurilin_ has quit IRC19:31
patrickeasti saw the same problem with nodepool and zuul19:31
yolandaclarkb , ok, i'm fine with it.. i try to send changes in smallest commits as possible, but this is related19:31
fungiasselin__: yep. also jenkins-job-builder. i think we patched it in jjb already but not sure about nodepool19:31
sdaguejhesketh: I guess, part of the reason it was nice to have it dynamic is that it let us be helpful with older logs as well19:31
yolandalet's see if i have some time this week to work on it19:31
sdaguewhen we realized it was less clear19:31
asselin__fungi, we're going to get a merge conflict fyi. not sure the status of this: https://review.openstack.org/#/c/149336/19:32
*** Guest92220 has joined #openstack-infra19:32
jheskethsdague: sure, but the older logs have it now and will continue to since they'll still be fetched from disk. Newer logs will all have the help text if we turn it on soon enough19:32
sdagueright, but that does get updated, and typically when it does, we want to post date it19:33
sdaguealso, it means we can use the same text files as on the static server19:33
sdagueunless we have a mechanism to bulk rebuild the index files19:33
fungiasselin__: indeed. one or the other can rebase when the time comes19:34
*** samueldmq is now known as samueldmq-away19:34
jheskethsdague: what if the log format changes and we want to update the help text in a way that doesn't apply to older logs?19:35
jheskethmy feeling is that by the time we update the help text people aren't interested in older logs anyway19:35
*** maxbit has joined #openstack-infra19:40
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Make gate-ceilometer-python34 only run on master branch  https://review.openstack.org/15463119:41
mriedemfungi: mtreinish: ^19:41
mtreinishmriedem: +119:42
clarkbyolanda was this a problem with earlier patchsets?19:42
yolandano, saw it in the latest one19:43
*** NobodyCam has quit IRC19:43
sdaguejhesketh: honestly, that documentation tends to lag by months behind file changes19:43
sdagueso the opposite has always been true19:43
*** BadCub has quit IRC19:44
yolandait was waiting forever for image to be created, and if i moved it to task manager it was finding it properly19:44
*** NobodyCam has joined #openstack-infra19:44
clarkbyolanda maybe we should sort that out? perhaps that indicates older patchsets might work better?19:44
*** BadCub has joined #openstack-infra19:44
yolandaclarkb, didn't did so much, lack of time so i did a small rebase... but this was quite a significant change, a new method added, so it may affect test behaviour in some way19:45
*** VijayTripathi has joined #openstack-infra19:45
jheskethsdague: fair, but the real fix there would be not to lag so much? (I know, much easier said than done)19:46
yolandabut you see, tests passed in previous changes, and i didn't update tests19:46
mtreinishjhesketh: yeah I mean jogo and I only went through and updated things after a ML thread about how the help was wrong19:46
mtreinishit doesn't really get updated otherwise19:46
*** dprince has quit IRC19:47
clarkbyolanda right thats why I am wondering if older versions might be better? I will run some local testing in a bit19:47
yolandaclarkb, nice, sorry to don't help so much but this week is impossible for me19:48
clarkbits ok just making sure I understand what is going on :)19:48
yolandai'd like to be more involved in nodepool, so please share what you get19:48
clarkbwill do19:48
sdagueso is this just a red herring - http://logs.openstack.org/94/152994/6/check/check-devstack-dsvm-updown/eaba588/console.html#_2015-02-10_17_16_36_476 ?19:52
openstackgerritMerged openstack-infra/reviewstats: Add tchaypo as core  https://review.openstack.org/15033219:53
mtreinishsdague: the job passed? was that not supposed to happen?19:54
openstackgerritMerged openstack-infra/reviewstats: Update Heat's core team  https://review.openstack.org/15075019:54
sdaguemtreinish: the job passed19:54
sdaguebut that output looks like it would be an error19:55
sdagueI'm curious if we just need to fix a d-g thing19:55
*** baoli has quit IRC19:57
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: Ajust gating jobs for ec2-api project  https://review.openstack.org/15464019:59
*** Guest92220 is now known as annegentle20:01
tristanCHello folks, I wanted to talk during the meeting about that 'include: layout-file' feature presented here: https://review.openstack.org/#/c/152290/. What is the way forward to get that feature accepted ?20:01
*** vipul has quit IRC20:01
jeblairtristanC: wait for someone to review it20:01
jheskethtristanC: I just had a look and will review it shortly20:01
tristanCoh alright, great :)20:01
*** NobodyCam has quit IRC20:01
jheskethtristanC: quick question though (possibly also directed at jeblair), would it be better to have a conf.d type directory rather than having to specify includes?20:01
*** jesusaurus has quit IRC20:01
*** BadCub has quit IRC20:02
*** esmute has quit IRC20:02
anteayano tc meeting?20:02
clarkbalso there is a spec for that20:02
clarkbso the change should hopefully implement the spec20:02
jheskethclarkb: a spec for splitting zuul's conf?20:02
clarkbjhesketh: yes20:02
clarkblet me get a link20:03
jheskethoh, I must have missed that one20:03
tristanCclarkb: oups, I didn't knew that...20:03
*** dprince has joined #openstack-infra20:03
tristanCjhesketh: yes sure, I just went the naive approach and extended the 'include' directive...20:03
fungiis the spec20:03
*** esmute has joined #openstack-infra20:03
*** BadCub has joined #openstack-infra20:03
jheskethspeaking of specs, there is a new version of bower for horizon sutff https://review.openstack.org/#/c/154297/20:04
*** NobodyCam has joined #openstack-infra20:04
*** bhuvan has quit IRC20:04
*** vipul has joined #openstack-infra20:04
clarkbits merged so it just needs to be implemented20:04
*** jesusaurus has joined #openstack-infra20:04
jheskethyep and has my preference for a conf dir rather than specific files20:04
*** rfolco has quit IRC20:05
*** aysyd has quit IRC20:05
*** yfried_ is now known as yfried|afk20:05
tristanCfungi: thanks for the link, good to know there is an infra-specs repository :)20:06
fungitristanC: we're not as disorganized and impulsive as some people might claim ;)20:06
fungiwell, i am, but the rest are sensible folk20:07
*** Rockyg has joined #openstack-infra20:07
tristanCalright, I can rework the proposed review to accomodate a conf.d like collection of files20:07
tristanCthough, while trying to refactor the layout.yaml of project-config, I found out a lot of dependencies, e.g. triple-o project adding a pipeline that is used in most project20:09
fungiclarkb: jeblair: mordred: zaro: looking at http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=9 before i go firing up the trusty replacement for review.o.o, i'm thinking we're good for sticking with the 30g ram flavor we're already using, but if we want to resize now is the time to discuss it20:09
clarkbfungi: yes I agree, don't forget the ssd volume for git repos too20:09
clarkbthough maybe we can just detach, then reattach to new node?20:10
clarkbI personally like rsync for simplicity though20:10
* sc68cal scratches his head at http://paste.openstack.org/show/170926/20:10
sc68calodd failure installing tempest via devstack..20:10
*** bhuvan has joined #openstack-infra20:12
clarkbsc68cal: maybe you don't have sudo rights with that user?20:12
sc68calnope, it's a devstack run so the stack user has the sudo privs20:13
*** shashankhegde has quit IRC20:13
krtaylorfungi, did we reach a conclusion on review.o.o upgrade date?20:14
*** bswartz has quit IRC20:14
clarkbsc68cal: devstack-gate removes the stack user's sudo rights20:14
clarkbsc68cal: so if running d-g twice it would fail like that iirc. Just pointing out there are scenarios where this likely won't work20:14
sc68calah you know what - I think it's related to I78d51f04ed01da4ce8aa0e127be028f969d3b4f8 just merged20:14
*** ldnunes has quit IRC20:15
jeblairasselin__: what's your best guess for how long the rule update would take at hp?20:15
*** achanda has quit IRC20:15
asselin__jeblair, 2 months20:15
jeblairasselin__: (it seemed like you were saying the 2 month figure involved an error that had to be corrected last time)20:15
asselin__jeblair, cody-somerville would know best20:15
pleia2is it incidental or intentional that root admins are also storyboard admins?20:15
jeblairpleia2: intentional20:15
*** wznoinsk has quit IRC20:16
fungiasselin__: make cody-somerville personally change the firewall rules ;)20:16
jeblairpleia2: they are also gerrit admins, we'll fix that soon20:16
*** wznoinsk_mobl has joined #openstack-infra20:16
pleia2jeblair: thanks20:16
asselin__jeblair, it my case it was 2 months + 1 month to fix the fat fingered mistake20:16
jeblaircody-somerville: ?20:16
cody-somervilleWhats needed? :)20:16
asselin__that only affected by site (wrong ip)20:16
clarkbyolanda: so my local tests don't seem to be returning. I assume this is the behavior your saw with images not going ready20:17
jeblaircody-somerville: gerrit's ip will be changing.  what's the minimum turnaround time for hp to upgrade firewall egress rules?20:17
yolandayes, same one20:17
*** wznoinsk_mobl has quit IRC20:17
*** cnesa has joined #openstack-infra20:17
yolandawhen you test the change with testr directly you see the issue, that is just watiing for getimage20:17
*** ashleighfarnham has quit IRC20:17
cody-somervilleminimum? It'll depend on queue length. And they have scheduled times for deploys. So I'd say it'll probably take a month to two months.20:18
clarkbyolanda: I wonder if that incidicates that the actual daemon won't see the image go ready either20:18
*** bswartz has joined #openstack-infra20:18
yolandaclarkb, but the actual daemon will do real tests20:19
clarkbyolanda: ya I need to read up on that portion of the code20:19
*** esmute has quit IRC20:19
yolandareal checks, i mean20:19
*** _nadya_ has joined #openstack-infra20:19
*** jesusaurus has quit IRC20:21
*** achanda has joined #openstack-infra20:21
*** vipul has quit IRC20:21
*** BadCub has quit IRC20:21
*** NobodyCam has quit IRC20:21
*** changbl has quit IRC20:21
jeblairclarkb, fungi, mordred: what do you think about 1 month, so schedule for mar 21?20:22
clarkbyolanda: I think this doesn't work because UploadImageTask needs to have its main() method called20:22
clarkbjeblair: that weekend also works for me20:22
clarkbjeblair: It is a bit frustrating that our ability to progress is impeded by arbitrary corporate rules in $locations though20:22
jeblairi could be convinced that 1 month is a good notice period.  honestly, i still feel like 2 months is us bending backwards to enable hp's dysfunction.20:22
*** NobodyCam has joined #openstack-infra20:22
clarkbjeblair: yes I would -1 2 months20:22
clarkb1 month seems to be reasonable ish20:23
*** zz_avozza is now known as avozza20:23
*** esmute has joined #openstack-infra20:23
*** BadCub has joined #openstack-infra20:23
*** vipul has joined #openstack-infra20:23
clarkbyolanda: I think we should go back to the patchset 5 type change20:23
mordredI'm fine with that - and just for the record, I agree - I think it's crazy that this is a conversation20:23
clarkbyolanda: since that doesn't treat the image upload as a Task object which requires the task managers20:23
yolandaclarkb, looking20:24
mordredI mainly just want to make sure that the folks who work at places as broken as HP is don't get screwed in the process20:24
fungijeblair: looks like that's right on the heels of feature freeze, so may be quiet-ish20:24
clarkbmordred: and the majority of them won't be. https works fine for pushing code and reviewing code20:24
fungijeblair: possibly the best choice that close to release20:24
*** jesusaurus has joined #openstack-infra20:25
fungijeblair: and i'm around that weekend as well20:25
mordredclarkb: k20:25
clarkbmordred: the only people that need to do crazyness are folks like asselin__ that either have to do silly proxying or wait for firewall updates (or both) to run third party CIs with zuul20:25
clarkbmordred: we know this works because it is how you get around China's firewall20:25
jeblairyeah, i feel like 1 month is a perfectly defensible time frame for that kind of update20:25
jeblairso it's a little painful for us, and a little painful for third parties20:25
jeblairwhich is the nature of all good compromises :)20:25
asselin__correct, only impacts thirdparty who use zuul20:26
fungias long as feature freeze doesn't get delayed for some reason we should be fine with that date i think20:26
fungiasselin__: only impacts third parties who have their zuul server stuck behind obnoxious egress filters they can't control20:26
yolandaclarkb, so you mean getting rid of the uploadimagetask object?20:26
*** r1chardj0n3s_afk is now known as r1chardj0n3s20:26
anteayaI should be around march 2120:26
yolandathis was created on future patches20:27
*** signed8bit has joined #openstack-infra20:27
asselin__fungi, correct.20:27
jeblairalso, we should be pretty clear that an ip change can happen at any time20:27
jeblairif there is some problem with the server, we may have to move to a new ip with little or no notice20:28
clarkbyolanda: yup20:28
mordredit IS a cloud, after all20:28
asselin__for everyone else on our team, we used corkscrew20:28
clarkbyolanda: I commented on the change20:28
anteayajeblair: agreed folks should know that20:28
jeblairso, happy to do some work to avoid breaking people with scheduled work, but we absolutely can't promise that there will always be notice20:28
yolandaclarkb , nice, i like your proposal20:29
mordredwenlock: ^^ you guys have magical pony powers - the above conversation might be of interest20:29
mordredwenlock: maybe someone could get 29418 blanket-outbound approved ...20:29
*** sparkycollier has joined #openstack-infra20:29
wenlockmordred, was just reading that, not sure which stream of conv though, trying to find the begining20:29
wenlockcan you summarize?20:29
asselin__this is what we used for 'normal' gerrit usage: http://paste.openstack.org/show/170931/20:30
mordredwenlock: we're moving the gerrit server so it's IP is going to change20:30
mordredwenlock: and HP has stupid firewall rules20:30
asselin__if we can find a working solution for zuul as clarkb says exists, that would be fine with me20:30
*** maxbit has quit IRC20:30
mordredwenlock: I think that sums it up20:30
wenlockahh yes,mordred, i know someone that knows someone that can help us fix that in our network20:30
wenlockwe did it before20:30
clarkbasselin__: with https that shouldnt be necessary, you just need an http proxy and a different gerrit remote (one that uses https)20:30
yolandaclarkb, i'll file a change tomororw, first thing in the morning :)20:30
yolandai want to have this updated20:30
mordredwenlock: awesome - I'll be sure to ping you when we have the new IP20:30
clarkbyolanda: sounds good, thank you20:31
wenlockbottom line, its just an ncs request, osanchez had set that up for us20:31
wenlockmordred sure, i'll forward to osanchez to help us kick that20:31
*** amrith is now known as _amrith_20:31
asselin__clarkb, sorry, not following20:31
yolandaoh, see that caching of floating ips got merged, i didn't realize :)20:31
yolandai need to sync nodepool this week , want to have these patches20:31
clarkbasselin__: gerrit 2.8 accepts changes over https20:31
clarkbasselin__: so you don't need corkscrew and all that. You just need to be able to https out which is super common in corp envs20:31
wenlockohh thats nice, clarkb, is git-review smart enough to make that switch?20:32
clarkbwenlock: if you change your gerrit remote20:32
asselin__clarkb, ok nice.20:32
asselin__clarkb, and for zuul, we can connect vis https?20:32
clarkbasselin__: no zuul requires ssh stream events20:32
wenlockclarkb not in the .gitreview file?20:32
clarkbwenlock: no because that would require changing all .gitreview files and using https across the board which I will personally -2 in projects I have the power to do that in :)20:33
wenlockic, so it stays as a personal choice20:33
clarkbI don't want to have to deal with gerrit https passwords20:33
clarkbI think ssh as the default is best20:34
clarkbhttps is then useable for those that it is easier for20:34
wenlockahh, i thought it would use pub keys, eek if it doesn't20:34
clarkbit uses the gerrit http password which you can get in your settings pages20:35
clarkbsame thing used for REST api access20:35
*** EmilienM|afk is now known as EmilienM20:35
fungiyeah, if gerrit https supported x.509 client certs for auth it would make some of this saner20:35
clarkbfungi: not sure I agree, I would have a hard enough time deal with x.509 certs for my self20:35
clarkbI can only imagine how much support that would require for the general use bas20:36
clarkbkeystone devs and fungi would be the only group with reliable push access :)20:36
wenlockclarkb ive been seeing that alot in other api's using https:// protocols though..20:36
fungiwell, it's no more complicated than pubkey auth for ssh, just not as well-known20:36
wenlockclarkb, docker api example20:36
clarkbfungi: it is in that openssl is the worst set of tools for a user ever20:36
clarkbfungi: even worse than socat20:36
morganfainberghm. keystone.20:36
clarkbfungi: ssh-keygen on the other hand is simple and easy20:37
stevemarclarkb, keystone you say20:37
morganfainbergclarkb, i heard keystone20:37
sc68calYeah i'm still trying to figure out why tempest is failing to install (http://paste.openstack.org/show/170926/) - it's a brand new VM, never run devstack before on it20:37
clarkbha I highlighted all of keystone :)20:37
morganfainbergstevemar, ^^20:37
r1chardj0n3sjhesketh: it seems things are busy in infra-land ;)20:37
morganfainbergclient certs are probably not fun for most people to deal with20:38
clarkbsc68cal: can you try running that command directly?20:38
clarkbmorganfainberg: exactly20:38
clarkbI always end up reading docs, then getting sad that the docs are terrible, then reading more docs, eventually giving in and just running commands until something works, but it doesn't work so I go back to the terrible docs20:39
jheskethr1chardj0n3s: things are always busy in infra-land20:39
clarkbthen heartbleed happens and it all makes sense20:39
funginew ip addresses for trusty review.o.o replacement: 2001:4800:7818:102:be76:4eff:fe05:9b1220:39
*** ashleighfarnham has joined #openstack-infra20:39
fungihas someone already started a draft announcement?20:39
sc68calclarkb: yeah, it's the "tox -e venv" command inside the tempest git dir20:40
clarkbfungi: I had not, maybe I should do that but also lunch20:40
*** tsg_ has joined #openstack-infra20:40
fungii can get it going20:40
sc68calrun by devstack near the end20:40
clarkbsc68cal: that doesn't look like a tox comamdn, it is installing globally not in a tox env20:40
*** ildikov has joined #openstack-infra20:40
clarkboh I see tox in the command that is weird20:40
wenlockfungi, so review.openstack.org will be, any others on 29418 ???20:40
anteayafungi: they should be able to access review-dev should they not?20:41
wenlockfungi btw, what was the old ip?20:41
fungiwenlock: we have a review-dev which is useful for testing things on if you want, 2001:4800:7818:104:be76:4eff:fe05:a76320:41
clarkbsc68cal: I do not know why a tox venv python would be updating /usr/local20:41
wenlockfungi, perfect, ty20:42
clarkbwenlock: host review.openstack.org :)20:42
fungiwenlock: old (current) review.openstack.org is 2001:4800:7815:104:3bc3:d7f6:ff03:bf5d20:42
fungialso in dns20:42
sc68calclarkb: neither do i20:42
wenlockfungi , clarkb ty, i'll see if i can dig up those old request20:42
*** tsg has quit IRC20:42
wenlockand have things updated on our end20:42
*** MaxV has quit IRC20:42
*** VijayTripathi has quit IRC20:42
*** MaxV has joined #openstack-infra20:43
*** achanda has quit IRC20:43
wenlockmordred, no email needed if that info is good20:43
*** claudiub_ has quit IRC20:43
*** Sukhdev has quit IRC20:44
*** timo has joined #openstack-infra20:45
clarkbwenlock: we have to send email for everyone else20:45
*** timo is now known as Guest6841720:45
wenlockclarkb, yep, i was referring to the message he was sending me.20:46
wenlockon the info fungi just relayed20:46
*** gokrokve has quit IRC20:46
*** gokrokve has joined #openstack-infra20:46
sc68calclarkb: reverting 9f1612e7afd9de56bc70d5c5b1c63ad5181d84ab fixes the issue - https://review.openstack.org/#/c/150928/20:47
clarkbsc68cal: so that was tested, interesting that its breaking for you20:48
*** MaxV has quit IRC20:48
clarkbsc68cal: and this is a fresh host right?20:48
sc68calclarkb: https://bugs.launchpad.net/devstack/+bug/142048020:48
openstackLaunchpad bug 1420480 in devstack "DevStack fails to install tempest" [Undecided,New]20:48
clarkbsc68cal: its a tempest bug if reverting that change fixes it20:48
*** mestery has joined #openstack-infra20:49
wenlockfungi, do you have dates yet for when those will cut over?20:49
sc68calclarkb: ok - going to do a full run with the revert in my VM and see if it completes the run20:50
wenlockfungi, the new ip's that is20:50
sc68calclarkb: believe me, I nuked the VM 3 times and ran devstack and got this error each time20:50
clarkbsc68cal: sure just pointing out you filed the bug in devstack but it would be tempest if tempest broke it20:50
sc68calit's possible that some other change is triggering it20:50
*** gokrokve has quit IRC20:51
clarkbsc68cal: also what version of pip and setuptools are ou using? devstack should update both to latest but maybe that doesn't happen for some reason?20:51
*** mestery has quit IRC20:51
sc68calclarkb: pip 6.0.8 from /usr/local/lib/python2.7/dist-packages (python 2.7)20:51
*** Guest68417 has quit IRC20:52
clarkbya thats up to date so no problem there20:52
sc68calI had clean runs this morning :-\20:52
sc68calerr... that sounds wrong20:52
anteayawenlock: well march 21 is being offered as an option with no clear acceptance that I have seen20:53
anteayawenlock: so dates to be determined20:53
clarkbsc68cal: also ubuntu 14.04? fedora 21? I am assuming a debuntu version based ont he /usr/local stuff20:53
*** _amrith_ is now known as amrith20:53
fungiwenlock: i'm presently drafting the announcement so we can reach a consensus on the schedule20:53
anteayawenlock: part of the date selection process was understanding how long big corps needed to update20:54
*** shashankhegde has joined #openstack-infra20:54
*** nicodemos has quit IRC20:54
wenlockanteaya last time we did it it took us 3 months, but i anticipate updates to take less than 30 days20:55
*** crc32 has joined #openstack-infra20:55
*** CIBuilds has joined #openstack-infra20:55
anteayawenlock: that is useful information to incorprate into out date selection, thank you20:55
wenlockanteaya i need to get the request going though to find out what the window will be, let me see if i can find a specific SLA they provide20:55
anteayawenlock: awesome20:55
fungidraft announcement is at https://etherpad.openstack.org/p/nMchefon2n20:58
fungii had some stuff in there about using the gerrit rest api with git-review but ripped it back out because i don't know of any concise documentation we can point people at for the current release20:58
*** mriedem has quit IRC20:59
*** BadCub has quit IRC20:59
*** esmute has quit IRC20:59
*** jesusaurus has quit IRC20:59
*** vipul has quit IRC20:59
*** NobodyCam has quit IRC20:59
clarkbfungi: +2 I have a mail list thread on the https stuff20:59
clarkbfungi: let me get a link to that and you can just refer to it21:00
fungiclarkb: awesome--thanks!21:00
clarkbfungi: http://lists.openstack.org/pipermail/openstack-dev/2014-September/045383.html thread starts there21:00
*** otter768 has joined #openstack-infra21:01
*** BobBall is now known as BobBall_AWOL21:01
clarkbhttp://lists.openstack.org/pipermail/openstack-dev/2014-September/045385.html has my directions21:01
clarkband ZZelle has info on using a git credential manager21:01
*** VijayTripathi has joined #openstack-infra21:01
*** mriedem has joined #openstack-infra21:02
clarkbsc68cal: fwiw I can run `tox -evenv -- ls` in an up to date tempest repo just fine21:02
clarkbsc68cal: so possibly that devstack is doing something funny21:03
sc68caldunno - it's totally not touched21:03
sc68cal5fc64f1fcba9a971e3c7f9e6100fd84d10678425 is the sha of devstack21:03
clarkband I really need to go get lunch now. so back in a bit21:03
sc68calclarkb: no problem21:04
*** sdake has joined #openstack-infra21:04
openstackgerritKhai Do proposed openstack-infra/project-config: Setup project to run jobs on jenkins-dev.o.o  https://review.openstack.org/15373921:05
*** otter768 has quit IRC21:05
jeblairfungi: made suggested addition21:06
jeblairfungi: lgtm21:06
*** baoli has joined #openstack-infra21:07
jeblairand i do think we pretty much have consensus on mar 21 at this point21:07
fungijeblair: thanks--that helps21:07
*** esmute has joined #openstack-infra21:08
wenlockanteaya, i found the answer in our faq from network connectivity services.   Normal request are 90 days, expedited request are 30 days.21:08
*** Alexandra_ has joined #openstack-infra21:10
jeblairso, the good (?) news is that we have sufficient time to file an expedited request.21:10
wenlockjeblair, hehe yep21:10
jeblairthe bad news is that if we ever had a problem with the gerrit server that resulted in an immediate ip change, hp would get a one month vacation from having to work on openstack21:10
wenlockit also helps if we can use the same request number as we did the first time, so i have to get that from my team who filed it21:10
fungiokay, if nobody else has any feedback on the announcement at https://etherpad.openstack.org/p/nMchefon2n i'll go ahead and send it21:10
*** CIBuilds is now known as Tim_O21:11
wenlockjeblair, would be nice if it was a pool of 5 ip's or something or 1 ip in each zone that it could possibly live in21:11
*** vipul has joined #openstack-infra21:11
wenlockfungi, wondering if there were backup ip's included, would be nice for future issues.21:11
jeblairwenlock: yes it would, but rax doesn't have floating ips, and we can't use their load-balancer product with non-http things :(21:12
fungiwenlock: lots of things would be nice, but our service providers assign ip addresses when we build a new instance21:12
*** BadCub has joined #openstack-infra21:12
openstackgerritKhai Do proposed openstack-infra/project-config: Setup a project to test zuul-dev.o.o periodic pipeline  https://review.openstack.org/15466721:12
wenlockfungi, jeblair, gotcha21:12
*** NobodyCam has joined #openstack-infra21:12
*** esmute has quit IRC21:12
fungiwenlock: it's this new-fangled "cloud" concept you might have heard of already ;)21:12
*** jesusaurus has joined #openstack-infra21:12
wenlockfungi, im starting to get familiar with it....funny enough that was the same issue raised on other teams we're doing pilots with21:13
fungiwenlock: great to know there's some consensus around pain points at least21:13
wenlockold networks suck21:13
*** amrith is now known as _amrith_21:14
*** esmute has joined #openstack-infra21:14
openstackgerritKhai Do proposed openstack-infra/project-config: Setup a project to test zuul-dev.o.o periodic pipeline  https://review.openstack.org/15466721:14
*** stevemar has quit IRC21:14
*** sparkycollier has quit IRC21:15
*** marun has quit IRC21:16
fungiokay, announcement sent21:18
*** radez_g0n3 is now known as radez21:20
*** Hefeweizen has quit IRC21:21
*** achanda has joined #openstack-infra21:21
anteayafungi: thanks21:21
*** bswartz has quit IRC21:28
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add Elizabeth K. Joseph to servers and storyboard  https://review.openstack.org/15467221:30
*** emagana has joined #openstack-infra21:32
zarojeblair: was wondering, where is the gerrit trove db? rax or hp?21:32
fungizaro: rackspace dfw region21:33
*** rlandy has quit IRC21:33
fungizaro: same provider and region as the server connecting to it21:33
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add Elizabeth K. Joseph to servers and storyboard  https://review.openstack.org/15467221:33
*** Alexandra_ has quit IRC21:33
*** jsavak has joined #openstack-infra21:35
zaroI just asked SlickNik about the connection timeout and he says that rax might be using something in front of the trove db, repose?, and there's a timeout setting there as well.21:36
*** dmsimard_away has joined #openstack-infra21:37
*** dmsimard_away is now known as dmsimard21:37
*** joesavak has quit IRC21:38
*** mestery has joined #openstack-infra21:39
*** baoli has quit IRC21:40
*** baoli has joined #openstack-infra21:40
*** _nadya_ has quit IRC21:41
fungizaro: ooh, interesting21:42
fungithat sounds entirely possible21:42
zaroknow anybody we can ask about that?21:43
anteayawould JayF be someone we could ask?21:44
fungijroll: phschwartz: do either of you happen to know why can tell us about possible timeouts for established sockets to trove databases?21:44
jeblairfungi, clarkb, sdague: https://etherpad.openstack.org/p/Kukz1pmRkY21:44
jeblairanteaya: ^21:44
*** dannywilson has quit IRC21:44
jeblairlemme know if that looks like it's ready for the -dev list.21:45
*** dannywilson has joined #openstack-infra21:45
jrollfungi: uhhhhh21:45
JayFI don't know things about trove?21:45
*** Sukhdev has joined #openstack-infra21:45
jrollfungi: can you clarify? I can try to round someone up21:45
mtreinishjeblair: does it detect if you did a circular dependency to know not to do that?21:46
jrollfungi: is this a problem or a question you have?21:46
jeblairmtreinish: see section "Cycles" :)21:46
SlickNikzaro / fungi: I'm trying to see if I can reach out to some of the RAX folks who help operate trove their public cloud to see if I can find out some more about it as well.21:47
fungijroll: we have server instances in dfw using cloud databases in dfw and see the connections to them time out in a matter of a few minutes21:47
*** ociuhandu has quit IRC21:47
mtreinishjeblair: heh, didn't get that far21:47
anteayajeblair: yes was reading, nicely communicated21:47
jeblairmtreinish: i did also inclued "don't create a cycle" in the tldr :)21:47
jrollfungi: lemme ask around, can you put in a support ticket while you're at it?21:48
jeblairmtreinish: i should put it in the tltldrdr21:48
mtreinishjeblair: that's why I asked. You said don't do it, but I was curious why21:48
anteayajeblair: in the case of a patch series that is tested together in check and then one of the dependent patches is removed, what happens?21:48
mtreinisherr curios what would happen if I didn't listen21:48
jeblairoh good, hopefully it will encourage people to keep reading then :)21:48
anteayajeblair: dependencies is removed21:48
fungijroll: i think i tried previously, but was unable to reproduce the behavior using the command-line mysqlclient21:49
jeblairanteaya: still asking about in check?  so like if one is abandoned while something depending on it is testing?21:49
*** bswartz has joined #openstack-infra21:49
jeblairanteaya: or were you asking what happens in gate?21:49
fungijroll: though we've had a couple database-using applications written in entirely unrelated frameworks display similar behavior21:50
jrollfungi: hrm, weird. is there traffic moving over it when it times out? it does connect at all, correct?21:50
zarofungi: remember it only repros for us when connectionpool=true21:50
jeblairanteaya: (regardless, the answer is "nothing happens in check" and "it is removed in gate")21:50
jrollfungi: zaro: maybe a connection limit thing?21:50
jeblairzaro: connectionpool!=true means make a new connection each time21:50
*** gyee has joined #openstack-infra21:51
fungijroll: yeah, it behaves like an idle timeout, but mysqlclient reports that the trove instance is configured for default (many hours) idle timeout21:51
jeblairjroll: there are about 4 connections involved21:51
zarojroll: it only disconnect on after idling for about 5 mins.21:51
jrollfungi: right, I'm wondering if there's network gear killing it21:51
jrollI've seen this with F5s21:51
fungijroll: i'm assuming there probably is21:51
fungijroll: such as the f5 big-ip default 5-minute socket timeout, yeah21:51
jrollyeah... idk if this is something I can get folks to fix but I can try21:51
fungiused to manage dozens of them myself21:51
fungithat was one of the things we always had to bump for customers21:52
jrollfungi: right, that's what this looks like, I'll see if I can do something though it will likely need a ticket to get traction21:52
*** asselin_ has quit IRC21:52
anteayajeblair: was asking about the jenkins comment21:52
*** dannywilson has quit IRC21:53
fungianyway, i can try to write something up in a ticket, but i'm not sure how to describe a concise case to reproduce it21:53
*** dannywilson has joined #openstack-infra21:53
jrollfungi: just give what you have and we'll see how it goes21:53
anteayajeblair: to know if some patch was tested with another patch in one set of tests and if the dependency relationship is removed or changed to recheck it to test both together again21:53
anteayajeblair: and yes, figured it wouldn't work in gate but might be helpful at some point to have a comment with the tests to know if the testing environment (which patches it was tested wtih) has changed21:54
anteayaif I am making sense21:54
*** dkliban is now known as dkliban_afk21:54
jrollfungi: to be clear, pass me the ticket id when you have it21:55
jeblairanteaya: to remove or change the dependency relationship, you would have to upload a new version of the patch21:55
anteayajeblair: that is true21:55
jeblairanteaya: so that would be obvious.  if there's a transitive dependency relationship and something in the middle changes, then, yeah, the results of the thing on the end are now invalid21:56
*** iccha has joined #openstack-infra21:56
icchahey I hear some trouble with instances in dfw with mysql21:56
icchaanyone got an instance id?21:56
anteayaiccha: you are here21:56
anteayaiccha: I'm so glad21:56
jeblairanteaya: but then gerrit will say that it depends on an outdated patch.  so that should be apparent too.21:56
anteayaI'll look for that21:57
pleia2welcome iccha, zaro was looking into some trove issues21:57
anteayajeblair: I added a question to line 108 in the etherpad21:57
pleia2I think he'd be the one to chat with21:57
fungiiccha: i can get you the nova and trove instance uuids for one known to be causing a problem anyway21:57
anteayaor asked you a question21:57
jeblairanteaya: yeah, working out the timestamp now21:57
anteayajeblair: cool, thanks21:57
icchathanks fungi that ll be useful21:57
jrolliccha: you got this then?21:57
icchaya i pinged our ops and they have not heard about anything else in dfw, so having instance id might help22:00
fungiiccha: we're seeing it, for example, with established mysql sockets originating from nova instance 9d3ff96d-920d-4d4f-854e-7057d268795c to trove instance f8bc89f6-adfb-4e89-9312-ea38b8c63fb0 (both in dfw)22:00
fungiiccha: this has been going on for months, maybe a year, probably for at least as long as we've been using rackspace trove instances22:00
fungiwe already moved one service off of a trove instance and back onto a local mysql database on the server due to similar behavior22:01
*** crc32 has quit IRC22:01
fungii'm assuming this is something inherent in the design of the deployment22:02
ttxfun part about Liberty is... I ran a test poll with yesterday's ballot set and Lizard won22:02
*** dimtruck is now known as zz_dimtruck22:03
mtreinishttx: did some one write a bot for last min voting?22:03
ttxmtreinish: I checked for that22:03
fungiiccha: basically the common thread between the cases we've seen is that the client end doesn't know that the server has dropped its established socket, and then throws an error and has to reconnect. and this seems to happen after only a few (~3) minutes of inactivity on the socket22:03
ttxmtreinish: nothing obvious in the last line22:03
fungittx: mtreinish: people who love liberty also love procrastinating?22:04
ttxmtreinish: here was yesterday's test: http://civs.cs.cornell.edu/cgi-bin/results.pl?id=E_7d805614c0655c5d22:04
*** Hefeweizen has joined #openstack-infra22:04
sdaguefungi: sounds pretty 'merican22:04
mtreinishfungi: heh, probably22:04
*** e0ne has quit IRC22:05
mtreinishttx: yeah comparing that with the final set doesn't reall show anything obvious22:05
ttxlookign at trends on the poll, all I can say is that Lizard started strong but Liberty caught up22:05
* mtreinish inner conspiracy nut is disappointed22:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for disabling bad pipelines  https://review.openstack.org/15436722:06
sdaguemordred: http://logs.openstack.org/37/151037/2/check/check-requirements-integration-dsvm/087d1d7/console.html#_2015-02-10_21_46_35_337 ???22:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers  https://review.openstack.org/11899322:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add gerrit reviews into patchset approvals  https://review.openstack.org/9739022:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for negative requirements  https://review.openstack.org/10272622:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically  https://review.openstack.org/11953422:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept  https://review.openstack.org/12152822:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Call driver methods more dynamically  https://review.openstack.org/11953322:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for triggers  https://review.openstack.org/11953222:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for sources  https://review.openstack.org/11953122:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for reporters  https://review.openstack.org/11953022:06
sdagueerror in setup command: Error parsing /tmp/tmp.A8NOB41seo/projects/installceilometer/setup.cfg: ValueError: git history requires a target version of pbr.version.SemanticVersion(2014.2.3), but target version is pbr.version.SemanticVersion(2014.2.2)22:07
jheskethsdague: just wondering if you were able to respond to my comments on https://review.openstack.org/#/c/141260/ please22:07
mtreinishsdague: that's because after the most recent release no one bumped the version string on stable's setup.cfg22:07
mtreinishmriedem: has patches to fix it everywhere22:07
mtreinishfwiw, it's on that tracking etherpad as #222:07
sdaguemtreinish: oh, so all of stable juno is bust?22:07
sdaguefor a new reason22:07
mtreinishsdague: yeah pretty much22:07
mtreinishwhich is part of what we were raising on the list...22:08
*** dustins has quit IRC22:08
clarkbthough pbr enforcement of that is unreleased22:09
clarkbI think22:09
*** joesavak has joined #openstack-infra22:09
mtreinishclarkb: oh right it was reverted at one point22:10
*** crainte has joined #openstack-infra22:10
jeblairanyone have any further thoughts on the CRD email?  or shall i fire it off?22:10
crainteiccha '-')/22:10
*** bswartz has quit IRC22:11
clarkbjeblair I just realized it will break sdagues hopes for no tests on only commit changes22:11
clarkbbut I dont think that needs to be in the email22:11
jeblairclarkb: ?22:11
*** vipul has quit IRC22:11
sdagueclarkb: 0.10.7 right?22:11
fungijeblair: crd announcement lgtm22:11
fungisorry, slow reader and keep getting sucked into other broken things22:12
clarkbjeblair sdague wants/wanted to not test patches that only update the commit message22:12
icchafungi: crainte is our ops engineer who is looking currently into your issue22:12
anteayajeblair: lgtm too22:12
*** NobodyCam has quit IRC22:12
clarkbsdague no thats off a featyre branch without semver22:12
anteayajeblair: mostly this will just create a flood of questions but the email is clear enough that now I think I know what is going on22:12
sdagueclarkb: here is the freeze - http://logs.openstack.org/37/151037/2/check/check-requirements-integration-dsvm/087d1d7/logs/pip-freeze.txt.gz22:12
*** VijayTripathi1 has joined #openstack-infra22:12
*** jsavak has quit IRC22:12
*** MaxV has joined #openstack-infra22:13
fungiiccha: crainte: thanks. as much as anything we were curious whether this was an intentional part of the design and if so why such a short timeout was chosen22:13
anteayaso when they pop in here hopefully I can say something coherent and useful22:13
*** VijayTripathi has quit IRC22:13
sdagueclarkb: so I think the no tests on commit message change will just need to be smarter.... possibly zuul smarter22:13
craintefungi not that i'm aware of, typically you see that kind of timeout when you put the server behind a load balancer. That doesn't seem to be the case here though22:13
jeblairclarkb: i don't think it ends all hope of that.  but that isn't straightforward to start with.22:13
*** mrmartin has quit IRC22:13
sdaguejeblair: agreed22:13
zarojeblair: does depends-on require the full change id?22:13
jeblairzaro: yes22:14
jeblairzaro: probably22:14
jeblairzaro: maybe not but let's say it does22:14
*** NobodyCam has joined #openstack-infra22:14
sdaguelet's all pretend22:14
anteayait does for me22:14
fungicrainte: would a packet capture help at all?22:14
jeblairi'll put that in the email and send it off.22:14
craintefungi i'm watching traffic on the database instance now but there is a big lack of mysql traffic. trying to find your cloud server and capture traffic on that end as well22:15
craintefungi if you have anything to generate mysql traffic to the instance please do22:15
*** vipul has joined #openstack-infra22:15
fungicrainte: right, one of the reasons we suspect an idle timeout is because this is a fairly idle system. i can cause it to do things now22:15
*** VijayTripathi has joined #openstack-infra22:15
jrollfungi: are you getting a tcp reset?22:16
*** tiswanso_ has quit IRC22:16
jrollcrainte: is there an F5 in front of the dbaas stuff?22:16
fungijroll: i'm not sure in this particular case. i haven't fired up tcpdump to see if i can get it to happen while i'm watching22:16
*** thedodd has quit IRC22:17
*** VijayTripathi1 has quit IRC22:17
zarojeblair: ok.  lgtm22:17
icchajroll: no f522:17
jrolloh, interesting22:18
sdagueclarkb: oh, the install pbr from repo parts are still in there22:18
ekarlsohey guys, I have a git requirement in requirements.txt, how can I make devstack honor my requirement there and not the one in global reqs ?22:18
*** claudiub has joined #openstack-infra22:18
zarofungi: which trove db are we looking at?  review.o.o or review-dev.o,o?22:18
fungizaro: review-dev22:18
zarofungi: i set the connectionpool to false so it's not gonna timeout anymore.22:19
fungizaro: oh22:19
ekarlsoI've even added LIBS_FROM_GIT=kazoo22:19
zaroshall i change it back to true22:19
openstackgerritSean Dague proposed openstack/requirements: don't install pbr from upstream source  https://review.openstack.org/15468722:19
ekarlsowithout it being honored :(22:19
fungizaro: how easy would it be to put it back so that rackspace can troubleshoot?22:19
*** mriedem has quit IRC22:19
zaroeasy, doing now22:20
*** tiswanso has joined #openstack-infra22:20
fungizaro: thanks, i'll start tcpdump while it's up and running and then try to catch a timeout22:20
*** mriedem has joined #openstack-infra22:20
mriedemmtreinish: sdague: https://review.openstack.org/#/q/Ib8a29258d99de75b49a9b19aef36bb99bc5fcac0,n,z22:20
sdagueekarlso: LIBS_FROM_GIT only works for libraries that have git install paths in devstack22:21
mriedemmtreinish: sdague: the ceilometer one depends on https://review.openstack.org/#/c/154631/22:21
zarook, done.  it should timeout in about 3-5 mins22:21
ekarlsosdague: :'(22:21
sdaguemriedem: so we could just fix the tests22:21
ekarlsosdague: so no way to pull a lib from git =?22:21
sdagueekarlso: make it an external devstack plugin22:21
fungizaro: thanks, i'll start the tcpdump in a couple minutes and then try to interact with the site and see what we catch22:21
ekarlsosdague: eh, really ? :|22:21
*** unicell has quit IRC22:21
ekarlsosounds like a bit of crap atm22:22
craintefungi iccha zaro https://gist.github.com/crainte/abf7b44ad74da0bac342 :P22:22
anteayaekarlso: let's ensure our language supports a positive work environment22:23
fungicrainte: is that 120 seconds timeout for inactive sessions?22:23
zarocrainte: 120 was exactly what we were seeing.22:23
ekarlsoanteaya: ok, I meant it's really sad that I can't specify a requirement from git and have it installed..22:23
fungicrainte: when queried the mysqld running there claims to have default session timeouts (which i think are something like 12 hours)22:23
ekarlsowithout having to do semi-hacks to get it in there22:23
anteayaekarlso: and your clearly explained statement does so wonderfully22:24
*** crc32 has joined #openstack-infra22:24
anteayaekarlso: and yes requirements are difficult22:24
fungiekarlso: semi-hacks like releasing software>22:24
*** pc_m has quit IRC22:24
sdaguethe external devstack plugins also aren't semi-hacks22:24
*** tiswanso has quit IRC22:25
craintefungi zaro updated that gist22:25
zarofungi: i just saw the timeout22:25
fungizaro: yep, i'm looking at the capture now22:25
craintethe show variables does return 28800 but the show global returns the configured 12022:25
sdaguemriedem: this might fix it as well - https://review.openstack.org/#/c/154688/22:26
sdagueanyway, time to call it a night, see you folks in the morning22:26
fungicrainte: yep, tcp reset coming back from the database server/port22:26
*** stevemar has joined #openstack-infra22:26
fungicrainte: thanks--that explains our confusion and the behavior we're seeing22:27
fungijeblair: ^ https://gist.github.com/crainte/abf7b44ad74da0bac34222:27
jeblairfungi: following22:27
jeblairnot sure i understand the difference between 'show variables' and 'show global variables'22:27
*** mriedem has quit IRC22:27
fungicrainte: is the global wait_timeout configurable from the client side or is that immutable and set by rackspace?22:28
craintelooking it up before i make that suggestion :P22:29
*** dprince has quit IRC22:29
jeblair(or why the global scope would override the session scope)22:29
craintefungi looks like it is configurable via control panel/api22:30
jeblairi think that if we had a mysql lib client query that value it would show 120 in both places22:31
*** elliottk_ has joined #openstack-infra22:31
*** arosen has joined #openstack-infra22:31
jeblairbut the _interactive_ sessions get the default set from interactive_timout, so that's what we see as the session (non-global) value22:31
arosenHi, I keep hitting these error in my CI runs pretty frequently: ReadTimeoutError: HTTPConnectionPool(host='pypi.dfw.openstack.org', port=80): Read timed out.22:31
jeblairnon-interactive (eg, gerrit) sessions get their value initialized from global wait_timout22:31
*** cnesa has quit IRC22:31
arosenI was wondering if there was anything obvious I could do. Is there a better mirror or ..?22:32
fungithis explains why i also wasn't able to reproduce the idle timeout behavior with an idle mysqlclient cli session22:32
jeblairarosen: this is for your own third-party ci?22:32
arosenhttp://logs2.aaronorosen.com/88/153088/8/check/dsvm-tempest-full-congress-nodepool/cd8fd02/logs/devstacklog.txt.gz#_2015-02-10_22_29_15_134 <-- it works for a while22:32
arosenjeblair:  yup22:32
fungiarosen: use pypi.python.org instead22:32
jeblairarosen, fungi: ++22:33
arosenif I set that in /etc/nodepool/scripts/configure_mirrors.sh do i need to rebuild the image?22:33
dansmithjeblair: thanks for the extra awesome22:33
jeblairdansmith: you're welcome!22:34
fungicrainte: so i'm looking in the dashboard and it's not obvious where you would adjust these values, but presumably they can be set via the cli/api?22:34
mordredjeblair: yes to your above statement22:34
mordredsorry I wasn't here to chime in22:34
craintefungi correct, in the dashboard it's Databases > Configurations22:35
crainteyou basically build a set of default configurations and then apply that set to an instance22:35
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Update the description for grey icons on status pg  https://review.openstack.org/15469022:35
fungicrainte: ooh! i was looking at instances22:35
jeblairmordred: np; sounds like we understand the problem and hopefully ^^ might be able to fix it!22:35
fungicrainte: aha, so we can create a new db config and then apply that. thanks22:35
mordredjeblair: yay!22:36
jeblairmordred: i am, however, going to blame mysql for being slightly insane in this regard.22:36
mordredjeblair: well ... there is actually a good reason for it - but I agree it's not unconfusing22:36
*** NobodyCam has quit IRC22:37
elliottk_hi all. i could use some help from someone familiar with the JJB email-ext module. is there anyone here who is?22:37
*** NobodyCam has joined #openstack-infra22:38
zarosure, ask away22:38
elliottk_thanks, zaro22:38
fungijeblair: mordred: clarkb: zaro: so anyway, we probably want to create a custom config with the wait_timeout cranked up to a more reasonable value and then apply that named configuration to all our trove instances22:38
clarkb8 hours22:39
elliottk_i'm trying to configure my email-ext so that the failure trigger and success trigger sections have, perhaps, different recipients as well as different subject lines, for example22:40
mordredoh - wait - hang on - I may have been missing the context on the why here22:40
jeblairmordred: which? initializing one value from an identically named other value?  or selectively initializing it from a third, different value in certain circumstances?  :)22:40
dims__jeblair, folks, very cool. ("Depends-On")22:40
*** elliottk_ has left #openstack-infra22:40
mordredfungi: what was the base problem?22:40
*** elliottk_ has joined #openstack-infra22:40
mordredjeblair: how else are you going to justify highly-paid consultants?22:40
mordredfungi: this is for the trove connection from ... gerrit?22:40
jeblairmordred: you're right, there was a good reason :)22:40
fungimordred: yep22:41
jeblairfungi, clarkb: 8 hours (mysql default) sounds good.22:41
fungimordred: rackspace default for new trove instances seems to be 2 minutes idle timeout22:41
mordredok. it does not bother me as a setting for gerrit22:41
jeblairmordred: would it bother you for something else?22:41
jeblaireg, storyboard?22:42
*** mwagner_lap has quit IRC22:42
zaroi'm noticing that this little bit of puppet doesn't seem to work.  http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/gerrit.pp#n27422:42
craintefungi this might just be a mysql 5.1 default (for rackspace) I'll try to get it updated to a higher value. Our 5.6 instances seem to have 360022:42
mordredit would - often times in higher-volume (potentially higher numbers of connections) because of the way thread memory works - you want to reap and re-create rather than hold open for ages22:42
mordredfor instance, for web properties, having a wait_timeout of over 1 is almost never correct22:43
fungicrainte: interesting. i wonder if that changed in response to customer feedback ;)22:43
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Disable install tempest by default  https://review.openstack.org/15469422:43
zarothe source is a soft link, not an actual file.  does anybody know if that should work?22:43
mordredBUT - as with all tuning, it depends on usage22:43
jeblairfungi, crainte, mordred: i thought that mysql doc was saying the default was 8 hours22:43
mordredand the way gerrit is using this I would imagine it's fine22:43
mordredthe default is 8 hours22:43
mordredmysql is notorious for having bad defaults22:43
jeblairmordred: do you think we should have a lower value for storyboard?22:44
mordredthe real thing is that we noticed at all - gerrit shoudl be transparently reconnecting22:44
craintejeblair mysql default, but rackspace configs override it. we currently have 120s for mysql 5.1 and 3600s for mysql 5.622:44
jeblaircrainte: gotcha22:44
fungimordred: fwiw i strongly suspect this is what was happening to paste/lodgeit as well. it was using sqlalchemy (purportedly) and should have been automagically detecting and reconnecting via mysql ping functionality22:45
mordredjeblair: apps _should_ be designed such that a network interruption to the db does not cause a failed operation, but merely a reconnect - if they want to appropriately deal with HA systems and whatnot22:45
clarkbmordred we thought that about lodgeit too22:45
mordredyah - clearly neither is actually designed to dtrt here - and they're not high traffic enough - so I thin kthe wait_timeout is fine22:45
fungimordred: we were also seeing it from jeepyb's gerritdb module until i added an explicit mysql ping call to it22:46
mordredI'd like to think about it more for storyboard - but not right now - it's not an urgent setting22:46
elliottk_zaro: what i'm trying to do is use the email-ext module to allow me to have different recipients and different subjects, depending on if the job fails or succeeds.  i have not been able to figure out how to do this22:46
jeblairmordred: yeah, i'm not sure how forcing a disconnect rather than just expecting clients to cycle their own connections is preferable.  i'm looking forward to hearing war stories over beer that explain all.  :)22:46
elliottk_the config files of jenkins-ui-generated jobs have a deeper level of email notification than i have been able to create using jjb22:47
mordredjeblair: it's actually to deal with poorly-behaving clients :)22:47
elliottk_i can paste a couple config files if that would help22:47
*** mfink_ has quit IRC22:47
*** Rockyg has quit IRC22:47
mordredjeblair: but poorly behaving clients in a runaway web app that's getting slammed can up the memory pressure inside of mysql, cause all queries to get slower and death by thundering herd22:48
elliottk_zaro: or some screenshots, if that's even better22:48
mordredwe have no web apps that see that kind of traffic22:48
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes  https://review.openstack.org/14126022:48
mordredjeblair: the other thing is that you want to make sure that your app handles disconnects properly so that it can handle db HA motions properly22:49
fungimordred: jeblair: it's entirely likely that the rackspace default difference between mysql 5.1 and 5.6 (2 minutes vs an hour) is why it's only the things we put onto trove early seemed to be doing this but things we added later weren't (because they got 5.6)22:49
*** shashankhegde has quit IRC22:49
mordredthen - if your app _Does_ ... because MySQL connection cost is so low, but because the internal thread scheduling is so bad22:49
jeblairfungi: yeah22:49
mordredyou may want to trigger disconnects while idle _anyway_ so that the mysql thread scheduler can do a better job22:49
*** shashankhegde has joined #openstack-infra22:49
mordredit's one of those places where you really want all of the parts of the system to be resilient to get it to _really_ scream22:50
zaroelliottk_: the jjb email-ext module might need to be updated22:50
zaroelliottk_: are there any existing tests in that for it?22:50
elliottk_zaro: i'm not sure what you mean by "any existing tests"22:51
*** Tim_O has quit IRC22:51
elliottk_zaro: are you saying the jjb email-ext module may not current do what i need?22:51
zaroelliottk_: http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/tests/publishers/fixtures/email-ext001.yaml22:51
*** achanda has quit IRC22:52
*** esmute has quit IRC22:52
fungizaro: i've applied a 28800 second wait_timeout config to the review-dev db22:52
elliottk_zaro: yes, this is the same as the page i'm using:  http://ci.openstack.org/jenkins-job-builder/publishers.html22:52
*** hdd has joined #openstack-infra22:52
elliottk_but from what i'm seeing, it does not currently support having different info for different email triggers22:53
zarofungi: cool.  i assume we don't need to restart Gerrit.22:53
fungizaro: doesn't seem so22:53
*** timcline has quit IRC22:53
zarofungi: i'm using review-dev.o.o so will let you know if it makes a difference.22:53
fungizaro: i've used review-dev's ui again to get it to reconnect, so i'll check it again in 5-10 minutes and a few more times over the evening too22:54
*** niska has quit IRC22:54
*** camunoz_gone has quit IRC22:55
*** BadCub has quit IRC22:55
*** vipul has quit IRC22:55
jogomordred: what does this mean http://logs.openstack.org/80/154680/1/check/check-requirements-integration-dsvm/c983c3a/console.html#_2015-02-10_22_26_13_28322:55
*** NobodyCam has quit IRC22:56
*** achanda has joined #openstack-infra22:56
zaroelliottk_: different info?  what type of other info were you expecting?  i see failure, success, and aborted22:56
jogomordred: ohh never mind, I sdague fixued those22:56
jogojust need to poke a few patches22:56
jogoihrachyshka: https://review.openstack.org/#/q/Ib8a29258d99de75b49a9b19aef36bb99bc5fcac0,n,z are ready  Ithink22:57
*** niska has joined #openstack-infra22:57
*** camunoz_gone has joined #openstack-infra22:57
*** vipul has joined #openstack-infra22:57
*** esmute has joined #openstack-infra22:58
*** NobodyCam has joined #openstack-infra22:58
SlickNikiccha / crainte: Thanks for following up with the wait_timeout matter!22:59
*** dannywilson has quit IRC22:59
zaroelliottk_: i think you just need to have multiple '- email-ext' definitions.23:00
fungiSlickNik: iccha: crainte: definitely--huge thanks. didn't realize the global values could override the reported session values, nor that we could set them ourselves. this ought to help tremendously23:00
*** dannywilson has joined #openstack-infra23:01
*** dannywilson has quit IRC23:01
elliottk_zaro: oh! i hadn't thought of that. let me try23:01
zaroyeah thanks a bunch. that timeout was annoying the bleepers out of me.23:01
*** BadCub has joined #openstack-infra23:01
*** otter768 has joined #openstack-infra23:01
*** bknudson has quit IRC23:01
*** dmorita has quit IRC23:01
craintefungi SlickNik np. i have a PR in now to make the change when new instances are created, so hopefully this won't creep up again23:02
*** patrickeast_ has joined #openstack-infra23:02
*** patrickeast has quit IRC23:02
*** patrickeast_ is now known as patrickeast23:02
anteayaas a note, I'm afk for tomorrow running errands23:03
*** dannywil_ has joined #openstack-infra23:03
*** dannywil_ has quit IRC23:04
*** dannywilson has joined #openstack-infra23:04
fungiooh, there's a devstack change cross-repo-depending on a devstack-gate change in the check pipeline right now23:05
mordredfungi: NEAT23:06
*** otter768 has quit IRC23:06
dansmithI'm about to push up a 23 patch set across four projects23:06
*** reed has quit IRC23:06
dansmith(no not really)23:06
fungidansmith: i was about to not be at all surprised23:07
jeblairby pure coincidence, i'm about to leave for the day!23:07
*** Alexandra_ has joined #openstack-infra23:07
jeblair(oh, then, uh, not really)23:07
jogomordred: so I think this patch is old enough https://review.openstack.org/#/c/121251/123:07
*** kgiusti has quit IRC23:07
jogomordred: actually we may want to revisit what belongs in openstack/openstack23:08
*** hdd has quit IRC23:08
*** armax has quit IRC23:08
fungiyep, jogo seems to be taking crd for a spin with devstack change 154689 so that he can actually see whether it works with the devstack-gate change 154694 it's depending on23:08
jogonow that we have decoupled a lot of projects from the gate23:08
*** alexpilotti has joined #openstack-infra23:09
openstackgerritK Jonathan Harker proposed openstack-infra/jenkins-job-builder: The email publisher needs a list of recipients  https://review.openstack.org/15343223:10
*** alexpilotti has quit IRC23:10
*** baoli has quit IRC23:11
mordredjogo: uhm. yeah23:11
*** doug-fish has left #openstack-infra23:11
*** baoli has joined #openstack-infra23:11
elliottk_zaro: i don't believe the recommendation of having multiple email-ext definitions is going to solve my problem23:11
jogomordred: maybe make openstack/openstack everything under openstack23:12
jogoand not just 'integrated gate' openstack23:12
elliottk_zaro: i need one definition with, for example, a different subject for a success trigger and a different one for a failure trigger23:12
*** joesavak has quit IRC23:12
mordredjogo: well ... in the new world order - is there any benefit in that repo anymore?23:12
mordredjogo: (just asking - I don't know if I have an answer)23:12
lifelessdoes it have a definition even?23:12
mordredlifeless: right now it's everything that's in the integrated gate23:13
jogomordred: I think so, for tracking purposes23:13
elliottk_zaro: i'm not sure how to make it clear the problem i'm having.  i can upload some screenshots or config.xml snippets if that could help23:13
lifelessmordred: I know that23:13
*** alexpilotti has joined #openstack-infra23:13
jogomordred: to say opentack landed x patches this week23:13
lifelessmordred: the context was new world order...23:13
mordredlifeless: then why did you ask? :)23:13
* mordred trolls lifeless23:13
* lifeless trouts mordred 23:13
mordredjogo: nod23:13
* mordred llamas lifeless23:13
jogomordred: because I think this graph is neato https://github.com/openstack/openstack/graphs/commit-activity23:13
*** BadCub has quit IRC23:14
*** esmute has quit IRC23:14
mordredjogo: what if we just made you a graph like that out of graphite?23:14
jogomordred: having a single git tree is still useful for debugging things23:14
mtreinishmordred: would it be as pretty?23:14
mordredmtreinish: it would be prettier, as it wouldn't be tainted by that terrible URL :)23:14
*** vipul has quit IRC23:14
*** NobodyCam has quit IRC23:14
jogoeven if some were release based things and other are install from source23:14
mtreinishheh, fair enough :)23:15
jogomordred: but I would settle for a graphite graph23:15
*** NobodyCam has joined #openstack-infra23:16
*** esmute has joined #openstack-infra23:17
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Add networking-nec project to StackForge  https://review.openstack.org/15470523:17
openstackgerritElizabeth K. Joseph proposed openstack-infra/project-config: Make devstack-gate stable release tests non-voting  https://review.openstack.org/15470623:18
*** vipul has joined #openstack-infra23:19
clarkbmordred: its a handy way to track numbers if nothing else23:19
clarkbgah I was scrolled up23:19
*** shashankhegde has quit IRC23:20
* mordred hands clarkb a llama23:20
jogomordred: I'll propose a change to openstack/openstack and we can discuss it there23:21
clarkbmordred: I can report that slingtv espn seems to work pretty well. so there is no reason to have cable anymore23:21
mordredjogo: kk23:21
mordredclarkb: woot23:21
*** BadCub has joined #openstack-infra23:21
zarojeblair: was wondering why ZUUL_PIPELINE param is sent for silent pipeline but not for periodic pipeline?  is that something that's configurable?23:22
*** hdd has joined #openstack-infra23:22
jeblairzaro: it should always be sent23:22
zaro*i mean sent to jenkins23:22
zarojeblair: i don't see it in some of the pipelines.. https://jenkins-dev.openstack.org/job/gate-noop/23:23
zarojeblair: ohh wait, i do see it. it's just not in the jenkins desription for the job.23:24
*** crainte has quit IRC23:24
zarojeblair: jenkins build info is set by zuul as well right?  i guess that's the problem.  why is it different for different jobs?23:26
*** amitgandhinz has quit IRC23:27
openstackgerritMerged openstack-infra/project-config: Make gate-ceilometer-python34 only run on master branch  https://review.openstack.org/15463123:27
*** gyee has quit IRC23:28
*** iccha has left #openstack-infra23:28
*** tsg_ has quit IRC23:29
zarofungi: good sign, i haven't seen a timeout yet.23:30
fungizaro: same here23:31
*** esmute has quit IRC23:31
*** mestery has quit IRC23:31
*** BadCub has quit IRC23:33
*** NobodyCam has quit IRC23:33
*** vipul has quit IRC23:34
*** elliottk_ has left #openstack-infra23:34
*** mestery has joined #openstack-infra23:36
*** esmute has joined #openstack-infra23:37
*** marun has joined #openstack-infra23:37
*** vipul has joined #openstack-infra23:38
*** ihrachyshka has quit IRC23:38
*** chlong has joined #openstack-infra23:39
*** BadCub has joined #openstack-infra23:40
*** NobodyCam has joined #openstack-infra23:41
*** vigneshvar has quit IRC23:42
*** ociuhandu has joined #openstack-infra23:44
*** unicell has joined #openstack-infra23:44
jheskethmordred, jeblair: so, rethinking on the bower mirror stuff, I've rerolled https://review.openstack.org/#/c/154297/ if you get a chance :-) (no pressure)23:44
*** Sukhdev has quit IRC23:45
*** nelsnelson has quit IRC23:45
*** nelsnelson has joined #openstack-infra23:45
*** nelsnelson has quit IRC23:46
*** dims_ has joined #openstack-infra23:46
*** emagana has quit IRC23:46
openstackgerritMerged openstack-infra/system-config: Only start 1 subunit worker per node  https://review.openstack.org/15172523:46
*** emagana has joined #openstack-infra23:47
*** dims__ has quit IRC23:47
openstackgerritMerged openstack-infra/system-config: Update Jenkins doc to reference nodepool  https://review.openstack.org/14905123:49
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-elasticsearch: Make the number of shards and replicas tunable  https://review.openstack.org/15471123:49
*** sabeen2 has quit IRC23:50
*** dims_ has quit IRC23:50
*** bswartz has joined #openstack-infra23:51
*** emagana has quit IRC23:51
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full  https://review.openstack.org/13650423:53
*** adalbas has quit IRC23:54
*** cdent has quit IRC23:54
*** carl_baldwin has quit IRC23:56
*** camunoz_gone is now known as camunoz23:56
*** mestery has quit IRC23:57
*** Alexandra_ has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!