| *** akshai has quit IRC | 00:01 | |
| *** shakamunyi has joined #tripleo | 00:03 | |
| *** shakamunyi has quit IRC | 00:09 | |
| *** rhallisey has quit IRC | 00:14 | |
| *** bvandenh has quit IRC | 00:25 | |
| *** bvandenh has joined #tripleo | 00:25 | |
| *** shakamunyi has joined #tripleo | 00:31 | |
| *** thrash is now known as thrash|g0ne | 00:33 | |
| *** julim has quit IRC | 00:43 | |
| *** shakamunyi has quit IRC | 00:47 | |
| *** dmacpher has quit IRC | 00:53 | |
| *** tiswanso has joined #tripleo | 00:59 | |
| *** tiswanso has quit IRC | 00:59 | |
| *** tiswanso has joined #tripleo | 01:00 | |
| *** shakamunyi has joined #tripleo | 01:04 | |
| *** yamahata has joined #tripleo | 01:05 | |
| *** dtrainor has quit IRC | 01:41 | |
| *** akshai_ has quit IRC | 02:07 | |
| *** Marga_ has quit IRC | 02:11 | |
| *** apetrich has quit IRC | 02:43 | |
| *** ramishra has joined #tripleo | 02:53 | |
| *** coolsvap has quit IRC | 03:01 | |
| *** Marga_ has joined #tripleo | 03:01 | |
| *** Marga_ has quit IRC | 03:06 | |
| *** saneax_AFK is now known as saneax | 03:08 | |
| *** Marga_ has joined #tripleo | 03:19 | |
| *** coolsvap has joined #tripleo | 03:24 | |
| *** coolsvap_ has joined #tripleo | 03:25 | |
| *** coolsvap_ has quit IRC | 03:25 | |
| *** tiswanso has quit IRC | 03:38 | |
| *** links has joined #tripleo | 03:39 | |
| *** ccamacho has quit IRC | 03:41 | |
| *** ccamacho has joined #tripleo | 03:44 | |
| *** r-mibu has quit IRC | 03:49 | |
| *** cwolferh has quit IRC | 03:50 | |
| *** masco has joined #tripleo | 04:02 | |
| *** akuznetsov has joined #tripleo | 04:09 | |
| *** ibravo has quit IRC | 04:10 | |
| *** akuznetsov has quit IRC | 04:13 | |
| *** cwolferh has joined #tripleo | 04:14 | |
| *** r-mibu has joined #tripleo | 04:29 | |
| *** tzumainn has quit IRC | 04:30 | |
| *** ramishra has quit IRC | 04:52 | |
| *** ramishra has joined #tripleo | 04:57 | |
| *** numans has joined #tripleo | 05:09 | |
| *** ramishra has quit IRC | 05:11 | |
| *** numans has quit IRC | 05:22 | |
| *** numans has joined #tripleo | 05:36 | |
| *** jcoufal has joined #tripleo | 05:45 | |
| *** rcernin has joined #tripleo | 05:51 | |
| *** oshvartz has joined #tripleo | 05:56 | |
| *** cwolferh has quit IRC | 05:57 | |
| *** bvandenh has quit IRC | 06:09 | |
| *** leanderthal|afk is now known as leanderthal | 06:22 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 06:23 |
|---|---|---|
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with periodic jobs https://review.openstack.org/297038 | 06:23 |
| *** liverpooler has joined #tripleo | 06:24 | |
| *** mikelk has joined #tripleo | 06:24 | |
| *** mcornea has joined #tripleo | 06:25 | |
| *** liverpooler has quit IRC | 06:26 | |
| *** liverpooler has joined #tripleo | 06:26 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 06:27 |
| *** rlandy has quit IRC | 06:29 | |
| *** yamahata has quit IRC | 06:33 | |
| *** shivrao has quit IRC | 06:34 | |
| *** rdopiera has joined #tripleo | 06:51 | |
| *** ramishra has joined #tripleo | 06:52 | |
| *** rdopiera has quit IRC | 06:56 | |
| *** cwolferh has joined #tripleo | 06:58 | |
| *** shivrao has joined #tripleo | 06:59 | |
| *** rdopiera has joined #tripleo | 07:10 | |
| *** zoli|gone is now known as zoliXXL | 07:11 | |
| zoliXXL | good morning | 07:12 |
| bandini | mornin | 07:12 |
| *** florianf has joined #tripleo | 07:12 | |
| *** aufi has joined #tripleo | 07:19 | |
| *** jaosorior has joined #tripleo | 07:21 | |
| *** apetrich has joined #tripleo | 07:23 | |
| *** jpena|off is now known as jpena | 07:24 | |
| *** apetrich has quit IRC | 07:25 | |
| *** apetrich has joined #tripleo | 07:26 | |
| *** qasims has joined #tripleo | 07:26 | |
| *** Marga_ has quit IRC | 07:29 | |
| *** dmacpher has joined #tripleo | 07:34 | |
| *** pcaruana has joined #tripleo | 07:38 | |
| *** shivrao_ has joined #tripleo | 07:38 | |
| *** shivrao has quit IRC | 07:40 | |
| *** shivrao_ is now known as shivrao | 07:40 | |
| *** ifarkas has joined #tripleo | 07:44 | |
| *** jistr has joined #tripleo | 07:44 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Remove individual service certificates https://review.openstack.org/301493 | 07:44 |
| *** rasca has quit IRC | 07:46 | |
| *** jaosorior has quit IRC | 07:46 | |
| *** jaosorior has joined #tripleo | 07:47 | |
| *** rasca has joined #tripleo | 07:48 | |
| *** shardy has joined #tripleo | 07:48 | |
| *** dtantsur|afk is now known as dtantsur | 07:49 | |
| *** cwolferh has quit IRC | 07:51 | |
| *** paramite has joined #tripleo | 07:53 | |
| *** mbound has joined #tripleo | 07:54 | |
| *** ohamada has joined #tripleo | 07:54 | |
| *** sshnaidm has quit IRC | 07:55 | |
| *** mbound has quit IRC | 08:00 | |
| *** jistr has quit IRC | 08:01 | |
| *** jistr has joined #tripleo | 08:01 | |
| *** mbound has joined #tripleo | 08:06 | |
| *** gfidente has joined #tripleo | 08:08 | |
| *** gfidente has quit IRC | 08:08 | |
| *** gfidente has joined #tripleo | 08:08 | |
| *** shivrao has quit IRC | 08:10 | |
| *** jprovazn has joined #tripleo | 08:16 | |
| jaosorior | marios: Hey dude, when you have time can you check this commit out? https://review.openstack.org/#/c/299475/ It's a bit harder to digest :/ | 08:18 |
| marios | jaosorior: o/ willdo | 08:21 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add scripts to generate tripleo.org website https://review.openstack.org/300452 | 08:22 |
| *** ebalduf has quit IRC | 08:23 | |
| *** panda has quit IRC | 08:26 | |
| jistr | have we seen the IPv6 job pass on stable/liberty yet? | 08:26 |
| *** panda has joined #tripleo | 08:26 | |
| *** akrivoka has joined #tripleo | 08:27 | |
| jistr | i'm asking because maybe if we want to merge to s/l and we have non-ha + ha passed, we should merge maybe? | 08:27 |
| jistr | yeah i don't see a single /liberty green on -upgrades job in cistatus http://tripleo.org/cistatus.html | 08:28 |
| jistr | marios, gfidente: so should we merge this? https://review.openstack.org/#/c/301168/ | 08:28 |
| jistr | i don't think the recheck is going to do it | 08:29 |
| jistr | i mean it's not going to be greener | 08:29 |
| *** Marga_ has joined #tripleo | 08:29 | |
| * jistr checks what's the scoop with IPv6 jobs on liberty | 08:29 | |
| marios | jistr: not sure what the status of that upgrades job is gfidente will know more | 08:29 |
| *** sshnaidm has joined #tripleo | 08:30 | |
| jistr | currently i think upgrade doesn't test upgrades, it tests IPv6, right? | 08:30 |
| marios | jistr: nice having time to do reviews and such stuff isn't it :) /me bites tongue multiple times | 08:30 |
| jistr | hehe :) | 08:31 |
| jistr | well, we have a few things we need to get backported, so i'm looking mainly how to get us there | 08:31 |
| jistr | not sure if rechecks will cut it if we're talking about -upgrades job | 08:31 |
| rdopiera | hi there, I wonder if I could help with anything? | 08:31 |
| marios | rdopiera: sheep man o/ long time stranger | 08:32 |
| jistr | rdopiera: hi! | 08:32 |
| rdopiera | yeah, I was lost in the nova land | 08:33 |
| marios | rdopiera: upstream meeting is today if you want https://wiki.openstack.org/wiki/Meetings/TripleO | 08:33 |
| rdopiera | excellent | 08:33 |
| jistr | and our main problem is unstable CI i think | 08:34 |
| jistr | so any debugging on that front is always helpful | 08:34 |
| *** Marga_ has quit IRC | 08:34 | |
| marios | jistr: speaking of which the upgrades job on that review is failing on an attribute error http://logs.openstack.org/68/301168/1/check-tripleo/gate-tripleo-ci-f22-upgrades/48e1a0c/console.html#_2016-04-04_22_12_55_999 | 08:34 |
| jistr | marios: so it *does* try to do overcloud update http://logs.openstack.org/68/301168/1/check-tripleo/gate-tripleo-ci-f22-upgrades/48e1a0c/console.html#_2016-04-04_22_12_55_999 | 08:35 |
| jistr | yes | 08:35 |
| jistr | hehe | 08:35 |
| marios | :) | 08:35 |
| rdopiera | unstable ci, why does it always have to be unstable ci | 08:35 |
| jistr | ok that's not going to help on recheck. my 2 cents is that we should just merge to s/l without upgrades job green, until we see it pass for the first time | 08:35 |
| *** chlong has quit IRC | 08:35 | |
| jistr | gfidente, marios ^^ | 08:35 |
| marios | jistr: yeah, i mean, really we should make it non-voting until it does pass. | 08:38 |
| marios | jistr: we spoke briefly with gfidente about it last week so i know he is/was working on it but don't know what else he got pulled into etc | 08:39 |
| *** derekh has joined #tripleo | 08:39 | |
| jistr | yea... i saw it passing on master once or twice | 08:39 |
| jistr | but never on liberty yet | 08:39 |
| *** electrofelix has joined #tripleo | 08:40 | |
| *** dmacpher has quit IRC | 08:41 | |
| gfidente | jistr marios I think we can merge the undercloud fix for now | 08:41 |
| gfidente | I will be investigating the failures in the upgrades job | 08:41 |
| jistr | gfidente: +1 | 08:41 |
| gfidente | looks like the ha passing rate is also very low | 08:41 |
| jistr | i'd like to get the backport going, and this is a low risk change, esp. given that we've seen the other jobs pass | 08:41 |
| gfidente | so looks like it's time for some ci love | 08:41 |
| gfidente | jistr yeah it's also restoring what we used to do before anyway | 08:42 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: PLUMgrid Neutron integration https://review.openstack.org/299151 | 08:42 |
| jistr | gfidente: yea | 08:42 |
| jistr | thanks | 08:42 |
| gfidente | behavioUr | 08:43 |
| shardy | marios: Hey, I didn't see any objections on the ML re the exception so I landed ^^ | 08:44 |
| marios | o/ gfidente | 08:44 |
| shardy | the liberty backport looks fine too other than the upgrades job timing out | 08:44 |
| marios | shardy: yeah makes sense i wrote along the same lines revisited yesterday | 08:44 |
| shardy | so I guess we just have to wait on that passing | 08:44 |
| shardy | marios: cool, I was hoping for a more affirmative +1 re the exception, but I guess no objections will have to do ;) | 08:44 |
| marios | shardy: well, i one question/issue was 'who else would need an exemption' wrt integrations work... we can take that to mean no-one will feel hard done by cos they didn't get their thing landed into liberty | 08:46 |
| gfidente | shardy on the upgrade job | 08:46 |
| gfidente | have any idea why this | 08:47 |
| gfidente | http://logs.openstack.org/68/301168/1/check-tripleo/gate-tripleo-ci-f22-upgrades/48e1a0c/console.html#_2016-04-04_22_12_55_999 | 08:47 |
| gfidente | is happening only on the update attempt? | 08:47 |
| shardy | marios: Yeah, it's tricky but my take is folks had an entire cycle to get things landed into liberty | 08:47 |
| openstackgerrit | Merged openstack/instack-undercloud: Set mariadb open_files_limit to -1 (unlimited) https://review.openstack.org/301168 | 08:47 |
| shardy | and we have to draw the line due to the mitaka backport policy | 08:47 |
| gfidente | I checked the deploy_args / update_args and they are identical except for the -puppet-registry.yaml which is only given on the update attempt but it's the first environment passed | 08:47 |
| shardy | we can retain the ability to vote on any exceptional backports, but I'm thinking they will become increasingly less likely when we get beyond the Mitaka GA | 08:48 |
| *** dtantsur is now known as dtantsur|brb | 08:49 | |
| *** florianf has quit IRC | 08:50 | |
| shardy | gfidente: Hmm, yeah that looks wrong - have you managed to reproduce locally? | 08:52 |
| shardy | it should just be reading an output from OS::TripleO::Services | 08:52 |
| *** florianf has joined #tripleo | 08:53 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh for --overcloud_delete https://review.openstack.org/301087 | 08:54 |
| gfidente | shardy no I don't see it locally | 08:54 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh for --overcloud_delete https://review.openstack.org/301087 | 08:55 |
| *** Marga_ has joined #tripleo | 08:56 | |
| shardy | http://paste.openstack.org/show/492955/ | 08:56 |
| shardy | that's the undercloud heat error - I think we reverted a patch last week which fixes it, sec | 08:57 |
| openstackgerrit | Jiri Stransky proposed openstack/instack-undercloud: Set mariadb open_files_limit to -1 (unlimited) https://review.openstack.org/301518 | 08:57 |
| shardy | https://review.openstack.org/#/c/299112/ | 08:59 |
| shardy | gfidente: I suspect we're pinned to a heat version that doesn't have that revert ^^ | 08:59 |
| shardy | So unless the current pin has been recently promoted, I guess we'll need a tmprevert | 09:00 |
| openstackgerrit | Jiri Stransky proposed openstack/instack-undercloud: Set mariadb open_files_limit to -1 (unlimited) https://review.openstack.org/301521 | 09:00 |
| *** ramishra has quit IRC | 09:03 | |
| ccamacho | jistr, good morning! quick question, after thinking about how to update the commands for Heat in tripleo.sh without duplicating to much code... and adding backwards compatibility with Liberty... I have created a small snippet to redefine the openstack command, do you think is a good idea to do it in this way? | 09:04 |
| ccamacho | Im testing it with different combinations... | 09:05 |
| shardy | ccamacho: Morning! | 09:05 |
| ccamacho | shardy good morning! | 09:05 |
| shardy | ccamacho: The heat cli commands are deprecated for Mitaka, but I don't think it's urgent to switch to the openstack CLI | 09:05 |
| *** ramishra has joined #tripleo | 09:05 | |
| shardy | ccamacho: do you have a patch up already? | 09:05 |
| ccamacho | shardy, yeahp, this one: https://review.openstack.org/#/c/301087/ | 09:06 |
| shardy | ccamacho: Looks OK, but note re the TODO, we can't remove it after mitaka releases because tripleo.sh is now in the tripleo-ci repo, which needs to support all branches (not just master) | 09:10 |
| shardy | I suspect we'll support the old heat CLI for longer than we'll support e.g stable/liberty in TripleO, so the other option is just to delay this change until we EOL liberty | 09:11 |
| shardy | getting coverage of the new CLI interface is probably a good thing tho :) | 09:11 |
| *** Marga_ has quit IRC | 09:12 | |
| ccamacho | shardy, I saw it interesting as In my case I'm facing with to many tools doing the same and there is already code using the openstack CLI in tripleo.sh | 09:14 |
| shardy | Yeah, it could be argued we should have an openstack overcloud delete CLI as it's more consistent | 09:16 |
| *** mkovacik has joined #tripleo | 09:16 | |
| shardy | but then we'll have *another* way to delete the stack ;) | 09:16 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh for --overcloud_delete https://review.openstack.org/301087 | 09:17 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Enable galera replication for Mariadb 10.1 https://review.openstack.org/300706 | 09:18 |
| *** ramishra has quit IRC | 09:20 | |
| *** ramishra has joined #tripleo | 09:22 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh for --overcloud_delete https://review.openstack.org/301087 | 09:22 |
| *** florianf has quit IRC | 09:26 | |
| *** sambetts|afk is now known as sambetts | 09:34 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 09:40 |
| *** florianf has joined #tripleo | 09:41 | |
| jaosorior | shardy, marios, gfidente: Taking another thought at the skipping of the post-deployment https://review.openstack.org/#/c/250268 now that I think about it. It also generates the services passwords file... So when we switch to using puppet for the keystone endpoints... wouldn't we instead want an option that only skips that part? and not the password file generation for the post-deployment | 09:43 |
| *** akuznetsov has joined #tripleo | 09:44 | |
| *** coolsvap has quit IRC | 09:45 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 09:46 |
| shardy | jaosorior: are you sure - looking at the code we create the passwords in _update_parameters? | 09:46 |
| shardy | which makes sense because you need them before starting the deployment | 09:46 |
| shardy | it's true that in a future split-stack model that also may become optional | 09:47 |
| shardy | jaosorior: it seems we've got some gross global stuff which should mean e.g OVERCLOUD_ADMIN_PASSWORD matches the parameters | 09:49 |
| shardy | hopefully we can kill that when we move to puppet configuring keystone | 09:49 |
| jaosorior | shardy: nevermind. I got a bit confused by the fact that that dict is used by os-cloud-config | 09:49 |
| jaosorior | so what happens is that the service users are created there, and use those passwords passed by os-cloud-config | 09:49 |
| jaosorior | but that can hopefully be removed by passing it all to puppet | 09:50 |
| jaosorior | sooo | 09:50 |
| jaosorior | no problem | 09:50 |
| shardy | cool - yeah stuff like the utils.get_password in _keystone_init is confusion | 09:50 |
| shardy | confusing even | 09:50 |
| jaosorior | yeah dude | 09:50 |
| jaosorior | shardy: We really need to get rid of that crap X_x | 09:50 |
| *** coolsvap has joined #tripleo | 09:57 | |
| *** rasca has quit IRC | 10:00 | |
| shardy | derekh: Hey, what's the current status of current-tripleo promotion, is it automated via the periodic job now? | 10:00 |
| number80 | /buffer/bu19 | 10:00 |
| openstackgerrit | Merged openstack/instack-undercloud: Delete default flavors in Python https://review.openstack.org/286233 | 10:01 |
| number80 | sorry, irc client going nuts | 10:01 |
| derekh | shardy: it will be automated once this is merged https://review.openstack.org/#/c/296363/ | 10:01 |
| *** mgould has joined #tripleo | 10:01 | |
| shardy | derekh: Ok, looking | 10:02 |
| shardy | it seems we're pinned to a 2-week-old heat commit which lacks a revert that we need | 10:02 |
| derekh | shardy: but trunk hasn't passed for about 2 weeks, that has to be figured out befor an actual promote happens | 10:02 |
| shardy | derekh: Ok - think we need this before the upgrades job will work: | 10:03 |
| shardy | https://github.com/openstack/heat/commit/354f1bcef038340d95f7dc4bc29268c9efed642e | 10:03 |
| *** rasca has joined #tripleo | 10:03 | |
| shardy | ref the issue gfidente highlighted earlier | 10:03 |
| derekh | shardy: ok, I havn't looked at why trunk isn't working but we wont get a promote until it does | 10:04 |
| shardy | derekh: OK, I'll propose a tmprevert in the meantime, thanks! | 10:05 |
| derekh | shardy: a temprevert in heat ? not sure that will work, checking | 10:06 |
| shardy | derekh: can we not do tmpreverts in tripleo-ci anymore? | 10:07 |
| derekh | shardy: we havn't been able to in a long time(except for puppet modules), as we should no longer need to because we've pinned the packages and only move pins to a working repo | 10:09 |
| shardy | derekh: ack, yeah I just realized that :( | 10:09 |
| shardy | So we have to fix master and promote, or temporarily force heat to build from source | 10:10 |
| shardy | I incorrectly assumed we still had code that could automagically flip DIB_INSTALLTYPE and build a delorean package | 10:11 |
| derekh | shardy: Yup, either fix trunk and promote or we could add heat to the list of whitelisted packages so its pulled from /current | 10:11 |
| shardy | testing with it whitelisted would be a good plan anyway I guess, let me try that | 10:11 |
| derekh | shardy: but that will leave us open to heat regressions, maybe this is ok as a temporary solution but its only putting off the inevitable of fixing trunk | 10:12 |
| derekh | shardy: yup, worth testing out at least | 10:12 |
| shardy | Yeah, even if we don't merge it it'll at least give us some data re fixing master | 10:12 |
| shardy | I'm not sure the upgrades job can work at all without this revert tho | 10:13 |
| *** electrofelix has quit IRC | 10:14 | |
| *** electrofelix has joined #tripleo | 10:14 | |
| *** dtantsur|brb is now known as dtantsur | 10:16 | |
| shardy | derekh: it does raise an interesting question re the workflow around reverts | 10:16 |
| shardy | e.g this patch was reverted because of TripleO failures reported around deletes | 10:17 |
| derekh | shardy: looking at the temprevert function at the moment, I think we could make it work for packages again, it would just need to add the name of the project to ZUUL_CHANGES so that toci knows to build it, thats I think all thats missing | 10:17 |
| shardy | we reverted it, and there's no easy way for TripleO to actually get the fix | 10:17 |
| shardy | derekh: cool, that'd be a nice option to have | 10:17 |
| openstackgerrit | Miles Gould proposed openstack/tripleo-quickstart: Update URLs to reflect move to OpenStack org https://review.openstack.org/301249 | 10:17 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Test with latest master heat https://review.openstack.org/301554 | 10:18 |
| derekh | shardy: yup, we're not ontop of keeping track of the periodic job, we need to be aiming for a promote every night and if it doesn't we need to fix the problem so a promote happens the next night | 10:18 |
| derekh | shardy: we just shouldn't allow the promote to get more then a day or 2 behind | 10:19 |
| shardy | derekh: Yeah, topic for the meeting later I guess | 10:19 |
| derekh | shardy: yup | 10:19 |
| shardy | need to find some folks with bandwidth to help keep that rolling | 10:19 |
| derekh | exactly | 10:20 |
| jaosorior | Have you guys seen some errors like "Engine went down during stack create" ? While the pingtest is running | 10:26 |
| shardy | jaosorior: Yes, it normally means it got OOM killed on a controller without enough ram and zero swap | 10:28 |
| *** akuznetsov has quit IRC | 10:28 | |
| jaosorior | bummer | 10:28 |
| jaosorior | shardy: I remember we reduced the number of workers for the undercloud. Have we done such a thing for the overcloud nodes? | 10:28 |
| *** qasims has quit IRC | 10:31 | |
| derekh | jaosorior: you can check the journal log to see if OOM kill anything | 10:34 |
| shardy | jaosorior: In CI, yes, but not by default | 10:35 |
| shardy | https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/worker-config.yaml | 10:36 |
| jaosorior | shardy: yeah, in CI is fine. Was just wondering where that is being set. | 10:37 |
| jaosorior | cause I see the definition of that template. But I don't find where it's actually used in CI | 10:38 |
| shardy | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/deploy.sh#L76 | 10:41 |
| shardy | jaosorior: ^^ | 10:41 |
| jaosorior | shardy: Alright, thanks dude | 10:42 |
| *** electrofelix has quit IRC | 10:42 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder: Add clean-machine-id element and use it in the ironic-agent element https://review.openstack.org/301571 | 10:44 |
| *** electrofelix has joined #tripleo | 10:47 | |
| *** dshulyak has joined #tripleo | 10:50 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-tripleoclient: Use clean-machine-id element when building the overcloud image https://review.openstack.org/301574 | 10:53 |
| *** hewbrocca has quit IRC | 10:59 | |
| *** hewbrocca-afk has joined #tripleo | 10:59 | |
| *** hewbrocca-afk is now known as hewbrocca | 10:59 | |
| *** hewbrocca is now known as hewbrocca-afk | 11:00 | |
| *** hewbrocca-afk is now known as hewbrocca | 11:01 | |
| *** rhallisey has joined #tripleo | 11:04 | |
| *** chem has quit IRC | 11:04 | |
| *** chem has joined #tripleo | 11:06 | |
| *** Marga_ has joined #tripleo | 11:12 | |
| gfidente | I think we it's about time we track rechecks appending a bug? | 11:13 |
| jistr | shardy: hi, we hit a problem where we need to distinguish between stack-create and stack-update again. I'm thinking the easiest way to solve it might be to have tripleo-common/tripleoclient pass a specific parameter "StackOperation" that would be set to "CREATE" or "UPDATE", and listen for that parameter in t-h-t. (We cannot use the 'action:' conditional on resources, because we're interested in create/update status of the *top* stack, not | 11:17 |
| jistr | the immediate parent stack of the resource.) | 11:17 |
| *** Marga_ has quit IRC | 11:17 | |
| shardy | jistr: can't you just look at DeployIdentifier and store it? | 11:18 |
| shardy | if it's not stored on the node, it's a create | 11:18 |
| shardy | otherwise, it's an update? | 11:18 |
| *** akuznetsov has joined #tripleo | 11:18 | |
| jistr | shardy: hmm like create another file in /var/lib/tripleo, similarly as we do with package manifests? yeah i think we could do that | 11:19 |
| gfidente | I was thinking to move that in the shared functions | 11:19 |
| *** nico_auv has joined #tripleo | 11:19 | |
| jistr | shardy: except it's not going to work on the first update after introducing this mechanism | 11:19 |
| shardy | jistr: yeah, that may be the simplest solution | 11:19 |
| shardy | jistr: Hmm, yeah I'll have to think if there's any way we can work around that | 11:20 |
| jistr | shardy: because existing deployments won't have deployment identifier stored, even though stack-create has happened for them | 11:20 |
| shardy | jistr: well it is stored *somewhere* | 11:20 |
| shardy | but obtaining it is likely to be hacky | 11:20 |
| * shardy drops off for lunch, bbiab | 11:20 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Nova needs the proper volumes to use Cinder https://review.openstack.org/294908 | 11:21 |
| *** florianf has quit IRC | 11:22 | |
| jistr | shardy: true, it's stored in hiera. But IIRC it's stored in hiera even for deployments that are about to be created, so that won't help to distinguish between create and update i think. (Unless we put someting even before the deployment that writes out the hiera file. That might do it.) | 11:23 |
| *** mbound has quit IRC | 11:24 | |
| jistr | shardy: also such mechanism would be node-local, e.g. on controller replacement it would resolve to 'create' on one controller and to 'update' on other controllers. That might be desirable in different scenarios but i don't think it's very good for the one we're solving (post-puppet restart trigger). | 11:26 |
| *** nico_auv has quit IRC | 11:26 | |
| *** akuznetsov has quit IRC | 11:28 | |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Nova needs the proper volumes to use Cinder https://review.openstack.org/301590 | 11:29 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Nova needs the proper volumes to use Cinder https://review.openstack.org/301592 | 11:30 |
| openstackgerrit | Miles Gould proposed openstack/python-tripleoclient: Use Ironic API v1.11 to support ENROLL state https://review.openstack.org/272206 | 11:30 |
| *** rodrigods has quit IRC | 11:32 | |
| *** rodrigods has joined #tripleo | 11:32 | |
| *** trown|outtypewww is now known as trown | 11:35 | |
| *** ibravo has joined #tripleo | 11:36 | |
| *** thrash|g0ne is now known as thrash | 11:36 | |
| jaosorior | trown: Hey dude, got some time to check this CR out? https://review.openstack.org/#/c/299475/7 | 11:36 |
| trown | derekh: is it possible for me to get the sqlite db that backs http://tripleo.org/cistatus.html | 11:36 |
| trown | jaosorior: looking | 11:36 |
| derekh | trown: yes, gimme a minute | 11:37 |
| trown | k I will review jaosorior patch in the meantime :) | 11:37 |
| *** florianf has joined #tripleo | 11:38 | |
| *** mbound has joined #tripleo | 11:39 | |
| *** mbound has quit IRC | 11:39 | |
| jaosorior | thanks dude :D | 11:39 |
| *** mbound has joined #tripleo | 11:40 | |
| *** mbound has quit IRC | 11:40 | |
| derekh | trown: goodsquishy.com/downloads/tripleo-jobs.db | 11:41 |
| derekh | trown: the periodic job is a separate file but I guess that the one you want? | 11:42 |
| derekh | *jobs | 11:42 |
| trown | derekh: ya, I wrote a compare jobs function to show pass rate of 1 job conditoned on another job passing | 11:42 |
| trown | I think that is a reasonable approximation of real pass rate | 11:42 |
| derekh | trown: yup | 11:43 |
| trown | but running the get_data function only gives a small recent sample of data | 11:43 |
| derekh | tripleo-jobs.py used to also generate a report for recent data but we took it out at some stage | 11:44 |
| trown | derekh: is there any unique identifier that links jobs run in the same ci run? I am having to fudge the calculation of P(A) AND P(B) | 11:45 |
| derekh | trown: you'll have to keep an eye out for jobs changing names in that db file (depending on how far back in time you go) | 11:45 |
| trown | by just taking min(P(A),P(B)) | 11:45 |
| trown | ya really only interested in making a 30day,15day,7day,1day table | 11:46 |
| derekh | trown: let me check, iirc we have the ZUUL id in there somewhere | 11:46 |
| derekh | trown: the zuul_ref colume on the table should be common for jobs triggered at the same time | 11:48 |
| trown | derekh: ah right, I could use that with a time window and get all jobs run on a patchset | 11:49 |
| trown | I was using that, but without the time window it gives all jobs for a review | 11:50 |
| *** weshay has joined #tripleo | 11:50 | |
| derekh | trown: you shouldn't need a time window, it should be uniq to the trigger (I think) | 11:51 |
| trown | oh right, I think I was mistaking that last field for change-id, but I see it is Z* | 11:52 |
| derekh | trown: occasionally you'll get multiple zuul_ref's for the same trigger, that where zuul restarted a job e.g. if it lost its ssh connection to the jenkins slave | 11:52 |
| trown | but the other ones would not have SUCCESS or FAILURE, so that would be fine | 11:53 |
| *** MaxPC has joined #tripleo | 11:54 | |
| derekh | trown: for example is you look at http://tripleo.org/cistatus.html right now you'll see some entries have multiple jobs against them, looks like a jenkins went down or something | 11:54 |
| derekh | trown: yup, filtering by SUCCESS/FAILURE should get you what you want | 11:54 |
| trown | ya, for stats we don't want ABORTED or None, I also have it filtered to remove jobs that only run the nonha job and changes to tripleo-ci since those skew things too | 11:55 |
| derekh | trown: seems about right | 11:57 |
| *** dprince has joined #tripleo | 11:57 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Include Gnocchi endpoints in enable-tls https://review.openstack.org/301157 | 11:57 |
| derekh | trown: also filter out every second jobs with FAILURE ;-) | 11:57 |
| trown | ? | 11:57 |
| derekh | trown: just a joke, never mind me | 11:58 |
| trown | lol, got ya, 20% -> 40% just like that :) | 11:58 |
| *** masco has quit IRC | 12:01 | |
| rook-lappy | shardy: i see that jenkins gave that patch a +1 (network validation patch I submitted).. Once that gets through we can look at patching up the gw | 12:03 |
| jistr | heya, trying to look into some of the CI fails | 12:04 |
| jistr | is it expected that we don't get host info tarballs from jobs that timed out? | 12:04 |
| jistr | e.g. this one http://logs.openstack.org/69/300569/1/check-tripleo/gate-tripleo-ci-f22-ha/09831a4/ | 12:04 |
| jistr | has no undercloud/overcloud logs | 12:05 |
| jaosorior | jistr: I think that's related. this one also timed out http://logs.openstack.org/99/292799/4/check-tripleo/gate-tripleo-ci-f22-ha/d18683f/ and it only has the logs for one of the controllers and the undercloud | 12:07 |
| shardy | jistr: Ah I thought we only set DeployIdentifier on update but I see that's not actually the case | 12:07 |
| shardy | I guess we can have a DeployAction parameter boolean based on stack_is_new easily enough | 12:08 |
| shardy | it's just seems a bit weird given that we should already know the action based on the API call | 12:08 |
| shardy | just setting DeployAction=stack_is_new as a parameter default would probably solve your problem tho I think? | 12:08 |
| *** electrofelix has quit IRC | 12:08 | |
| jistr | shardy: yea that's what i suggested above, just called it StackOperation :) | 12:09 |
| jistr | but DeployAction wfm too :) | 12:09 |
| shardy | jistr: Aha, either wfm ;) | 12:09 |
| *** mbound has joined #tripleo | 12:11 | |
| jistr | jaosorior: looks like the one you linked timed out while collecting the logs after a green test. Talk about bad luck :D | 12:13 |
| trown | well statistically speaking... average luck :p | 12:13 |
| jistr | ok so i guess it *can* currently expected that we don't have logs from jobs that timed out | 12:14 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Plans Dropdown in Deployment Plan page https://review.openstack.org/295882 | 12:14 |
| jistr | and we have timeouts that don't seem to work | 12:14 |
| jistr | http://logs.openstack.org/69/300569/1/check-tripleo/gate-tripleo-ci-f22-ha/09831a4/console.html#_2016-04-05_09_35_05_916 | 12:14 |
| jistr | that one was supposed to wait_for 3 minutes | 12:14 |
| jistr | but it waited ~28 minutes | 12:14 |
| trown | ya, I thought we tried to solve that by pre-timing out so we always get logs | 12:14 |
| jistr | ok... i'll see if i can fix that timeout ^^ | 12:15 |
| jistr | there's something wrong with the deployed overcloud, looks like crm_resource --wait never finishes waiting. We need to really kill it after 3 minutes, otherwise we'll never get logs from those stuck HA jobs. | 12:16 |
| *** pradk has quit IRC | 12:17 | |
| *** eggmaster has joined #tripleo | 12:17 | |
| *** pradk has joined #tripleo | 12:17 | |
| jaosorior | jistr: well... knowing that it's green. +2? https://review.openstack.org/#/c/292799/ :P | 12:18 |
| jistr | trown: ah yea. wait_for is for polling, but what we need there is to timeout a blocking call. i'll submit a fix | 12:18 |
| *** sshnaidm has quit IRC | 12:19 | |
| *** egafford has joined #tripleo | 12:20 | |
| *** nico_auv has joined #tripleo | 12:22 | |
| *** jayg|g0n3 is now known as jayg | 12:23 | |
| *** snecklifter has quit IRC | 12:25 | |
| *** panda has quit IRC | 12:26 | |
| *** panda has joined #tripleo | 12:26 | |
| jistr | jaosorior: any potential for mitaka/liberty backport on that one? wondering about what it might do with updates/upgrades | 12:27 |
| jaosorior | jistr: Which one? The one I pointed you to? | 12:28 |
| jaosorior | jistr: the /etc/hosts format change? | 12:28 |
| *** aufi has quit IRC | 12:28 | |
| jistr | jaosorior: yes | 12:28 |
| jaosorior | jistr: Those names are not used anywhere yet | 12:29 |
| jaosorior | jistr: Which is what I was asking gfidente about. | 12:29 |
| jistr | jaosorior: ack, thanks. +2 | 12:30 |
| jaosorior | jistr: Thanks for checking it out dude | 12:31 |
| jaosorior | jistr: How acquainted are you with the python-tripleoclient? Got a question regarding that | 12:31 |
| jistr | jaosorior: not much, but you can post the question and maybe thrash can chime in if i don't know | 12:32 |
| thrash | jaosorior: go for it. | 12:32 |
| *** sshnaidm has joined #tripleo | 12:34 | |
| jaosorior | jistr, thrash: in the postconfig, if the KeystoneAdminVip and the KeystoneInternalVip are not set as outputs, it seems that it uses the public endpoint... why is that? Is that something that might actually happen? Cause in this CR I'm removing them https://review.openstack.org/#/c/299475/7/tripleoclient/v1/overcloud_deploy.py | 12:34 |
| *** snecklifter has joined #tripleo | 12:34 | |
| jaosorior | I mean, removing that setting; not the output | 12:34 |
| *** akshai has joined #tripleo | 12:35 | |
| *** paramite is now known as paramite|afk | 12:35 | |
| *** rlandy has joined #tripleo | 12:37 | |
| *** paramite|afk is now known as paramite | 12:38 | |
| *** liverpooler has quit IRC | 12:39 | |
| thrash | jaosorior: let me look, but that is probably fine. | 12:39 |
| jaosorior | trown and I were wondering what's up with that O_O | 12:39 |
| thrash | that code should probably be removed in N anyway. I thought we deprecated all of that. | 12:40 |
| thrash | all of that should move to puppet. | 12:40 |
| *** jpena is now known as jpena|lunch | 12:41 | |
| jaosorior | thrash: That is the case. But if support for configurable ports is needed (haven't checked the bz but I think it is) then this can be backported | 12:41 |
| thrash | jaosorior: I'll comment on the patch where you and trown were discussing. | 12:42 |
| jaosorior | alrighty | 12:42 |
| thrash | jaosorior: rather than type it twice. :) | 12:43 |
| thrash | jaosorior: commented... | 12:43 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Fix timeout on crm_resource --wait https://review.openstack.org/301624 | 12:43 |
| jistr | trown, derekh: hoping this should help ^ | 12:44 |
| jaosorior | thrash: Thanks dude | 12:45 |
| openstackgerrit | Cyril Lopez proposed openstack/tripleo-docs: The control placement is needed more information to aviod a no valid host found https://review.openstack.org/301185 | 12:49 |
| *** julim has joined #tripleo | 12:52 | |
| *** sshnaidm has quit IRC | 12:53 | |
| hewbrocca | dprince: Would you be able to have a look at an os-net-config blocker | 12:58 |
| hewbrocca | dsneddon: is on PTO today | 12:59 |
| hewbrocca | and it's kind of an emergency... | 12:59 |
| hewbrocca | http://bugzilla.redhat.com/1323717 | 12:59 |
| openstack | hewbrocca: Error: Error getting bugzilla.redhat.com bug #1323717: NotPermitted | 12:59 |
| hewbrocca | feh | 12:59 |
| *** ibravo has quit IRC | 12:59 | |
| *** morazi has joined #tripleo | 13:04 | |
| *** sshnaidm has joined #tripleo | 13:08 | |
| *** wendar_ has joined #tripleo | 13:14 | |
| *** mgagne_ has joined #tripleo | 13:20 | |
| *** rhallisey has quit IRC | 13:21 | |
| *** gfidente has quit IRC | 13:21 | |
| *** rbrady has quit IRC | 13:21 | |
| *** jdob has quit IRC | 13:21 | |
| *** rook has quit IRC | 13:21 | |
| *** uncurlhalo has quit IRC | 13:21 | |
| *** ayoung has quit IRC | 13:21 | |
| *** dsneddon has quit IRC | 13:21 | |
| *** mgagne has quit IRC | 13:21 | |
| *** wendar has quit IRC | 13:21 | |
| *** saneax has quit IRC | 13:21 | |
| *** al has quit IRC | 13:21 | |
| *** jprovazn has quit IRC | 13:21 | |
| *** mcornea has quit IRC | 13:21 | |
| *** jcoufal has quit IRC | 13:21 | |
| *** bhunter has quit IRC | 13:21 | |
| *** marios has quit IRC | 13:21 | |
| *** pblaho has quit IRC | 13:21 | |
| *** pleia2 has quit IRC | 13:21 | |
| *** therve has quit IRC | 13:21 | |
| *** mburned has quit IRC | 13:21 | |
| *** florianf has quit IRC | 13:21 | |
| *** shardy has quit IRC | 13:21 | |
| *** Ryjedo has quit IRC | 13:21 | |
| *** tserong has quit IRC | 13:21 | |
| *** yuanying has quit IRC | 13:21 | |
| *** Erming__ has quit IRC | 13:21 | |
| *** NobodyCam has quit IRC | 13:21 | |
| *** isq_ has quit IRC | 13:21 | |
| *** fabbione has quit IRC | 13:21 | |
| *** sirushti has quit IRC | 13:21 | |
| *** morazi has quit IRC | 13:21 | |
| *** rlandy has quit IRC | 13:21 | |
| *** egafford has quit IRC | 13:21 | |
| *** mbound has quit IRC | 13:21 | |
| *** ohamada has quit IRC | 13:21 | |
| *** ifarkas has quit IRC | 13:21 | |
| *** rcernin has quit IRC | 13:21 | |
| *** thrash has quit IRC | 13:21 | |
| *** larsks has quit IRC | 13:21 | |
| *** hewbrocca has quit IRC | 13:21 | |
| *** coolsvap has quit IRC | 13:21 | |
| *** shakamunyi has quit IRC | 13:21 | |
| *** bswartz has quit IRC | 13:21 | |
| *** marcusvrn_ has quit IRC | 13:21 | |
| *** hjensas has quit IRC | 13:21 | |
| *** rwsu has quit IRC | 13:21 | |
| *** spredzy has quit IRC | 13:21 | |
| *** afazekas has quit IRC | 13:21 | |
| *** jroll has quit IRC | 13:21 | |
| *** trown has quit IRC | 13:21 | |
| *** ianw has quit IRC | 13:21 | |
| *** EmilienM has quit IRC | 13:21 | |
| *** mandre has quit IRC | 13:21 | |
| *** StevenK has quit IRC | 13:21 | |
| *** MaxPC has quit IRC | 13:21 | |
| *** weshay has quit IRC | 13:21 | |
| *** trozet has quit IRC | 13:21 | |
| *** tremble has quit IRC | 13:21 | |
| *** jtomasek has quit IRC | 13:21 | |
| *** zoliXXL has quit IRC | 13:21 | |
| *** kbyrne has quit IRC | 13:21 | |
| *** onovy has quit IRC | 13:21 | |
| *** larstobi has quit IRC | 13:21 | |
| *** adam_g has quit IRC | 13:21 | |
| *** Nakato has quit IRC | 13:21 | |
| *** dobson has quit IRC | 13:21 | |
| *** tristanC has quit IRC | 13:21 | |
| *** cmyster has quit IRC | 13:21 | |
| *** akshai has quit IRC | 13:21 | |
| *** nico_auv has quit IRC | 13:21 | |
| *** eggmaster has quit IRC | 13:21 | |
| *** dprince has quit IRC | 13:21 | |
| *** dshulyak has quit IRC | 13:21 | |
| *** rasca has quit IRC | 13:21 | |
| *** ramishra has quit IRC | 13:21 | |
| *** pcaruana has quit IRC | 13:21 | |
| *** rook-lappy has quit IRC | 13:21 | |
| *** openstackgerrit has quit IRC | 13:21 | |
| *** beagles has quit IRC | 13:21 | |
| *** pino|work has quit IRC | 13:21 | |
| *** pradk has quit IRC | 13:21 | |
| *** lazy_prince has quit IRC | 13:21 | |
| *** dmsimard has quit IRC | 13:21 | |
| *** zaneb has quit IRC | 13:21 | |
| *** mordred has quit IRC | 13:21 | |
| *** jpeeler has quit IRC | 13:21 | |
| *** andreaf has quit IRC | 13:21 | |
| *** HenryG has quit IRC | 13:21 | |
| *** julim has quit IRC | 13:21 | |
| *** jaosorior has quit IRC | 13:21 | |
| *** numans has quit IRC | 13:21 | |
| *** shadower has quit IRC | 13:21 | |
| *** leanderthal has quit IRC | 13:21 | |
| *** stevebaker has quit IRC | 13:21 | |
| *** SpamapS has quit IRC | 13:21 | |
| *** zigo has quit IRC | 13:21 | |
| *** bkero has quit IRC | 13:21 | |
| *** Hazelesque has quit IRC | 13:21 | |
| *** ChanServ has quit IRC | 13:21 | |
| *** mgould has quit IRC | 13:21 | |
| *** mkovacik has quit IRC | 13:21 | |
| *** paramite has quit IRC | 13:21 | |
| *** rdopiera has quit IRC | 13:21 | |
| *** david-lyle has quit IRC | 13:21 | |
| *** lifeless has quit IRC | 13:22 | |
| *** lynxman has quit IRC | 13:22 | |
| *** leifmadsen has quit IRC | 13:22 | |
| *** r-mibu has quit IRC | 13:22 | |
| *** social has quit IRC | 13:22 | |
| *** SlickNik has quit IRC | 13:22 | |
| *** igorbelikov has quit IRC | 13:22 | |
| *** jpena|lunch has quit IRC | 13:22 | |
| *** dtantsur has quit IRC | 13:22 | |
| *** Slower has quit IRC | 13:22 | |
| *** Ng has quit IRC | 13:22 | |
| *** tdasilva has quit IRC | 13:22 | |
| *** quack_quack has quit IRC | 13:22 | |
| *** CaptTofu has quit IRC | 13:22 | |
| *** rhefner has quit IRC | 13:22 | |
| *** cinerama has quit IRC | 13:22 | |
| *** panda has quit IRC | 13:22 | |
| *** rodrigods has quit IRC | 13:22 | |
| *** jistr has quit IRC | 13:22 | |
| *** mikelk has quit IRC | 13:22 | |
| *** ccamacho has quit IRC | 13:22 | |
| *** lucasagomes has quit IRC | 13:22 | |
| *** psanchez has quit IRC | 13:22 | |
| *** Tristitia has quit IRC | 13:22 | |
| *** bnemec has quit IRC | 13:22 | |
| *** crinkle has quit IRC | 13:22 | |
| *** rickflare has quit IRC | 13:22 | |
| *** sshnaidm has quit IRC | 13:22 | |
| *** snecklifter has quit IRC | 13:22 | |
| *** chem has quit IRC | 13:22 | |
| *** akrivoka has quit IRC | 13:22 | |
| *** oshvartz has quit IRC | 13:22 | |
| *** Goneri has quit IRC | 13:22 | |
| *** greghaynes has quit IRC | 13:22 | |
| *** athomas has quit IRC | 13:22 | |
| *** ryansb has quit IRC | 13:22 | |
| *** colonwq has quit IRC | 13:22 | |
| *** dweaver has quit IRC | 13:22 | |
| *** sambetts has quit IRC | 13:22 | |
| *** slagle has quit IRC | 13:22 | |
| *** derekh has quit IRC | 13:22 | |
| *** number80 has quit IRC | 13:22 | |
| *** jrist has quit IRC | 13:22 | |
| *** michchap has quit IRC | 13:22 | |
| *** adarazs has quit IRC | 13:22 | |
| *** nijaba has quit IRC | 13:22 | |
| *** lmiccini has quit IRC | 13:22 | |
| *** shanky has quit IRC | 13:22 | |
| *** george_goh has quit IRC | 13:22 | |
| *** bandini has quit IRC | 13:22 | |
| *** jayg has quit IRC | 13:22 | |
| *** ansiwen has quit IRC | 13:22 | |
| *** telnoratti has quit IRC | 13:22 | |
| *** jidar has quit IRC | 13:22 | |
| *** matbu has quit IRC | 13:22 | |
| *** saneax has joined #tripleo | 13:24 | |
| *** Marga_ has joined #tripleo | 13:24 | |
| *** sshnaidm has joined #tripleo | 13:24 | |
| *** morazi has joined #tripleo | 13:24 | |
| *** julim has joined #tripleo | 13:24 | |
| *** rlandy has joined #tripleo | 13:24 | |
| *** akshai has joined #tripleo | 13:24 | |
| *** snecklifter has joined #tripleo | 13:24 | |
| *** panda has joined #tripleo | 13:24 | |
| *** nico_auv has joined #tripleo | 13:24 | |
| *** egafford has joined #tripleo | 13:24 | |
| *** pradk has joined #tripleo | 13:24 | |
| *** eggmaster has joined #tripleo | 13:24 | |
| *** mbound has joined #tripleo | 13:24 | |
| *** dprince has joined #tripleo | 13:24 | |
| *** MaxPC has joined #tripleo | 13:24 | |
| *** weshay has joined #tripleo | 13:24 | |
| *** florianf has joined #tripleo | 13:24 | |
| *** rodrigods has joined #tripleo | 13:24 | |
| *** chem has joined #tripleo | 13:24 | |
| *** hewbrocca has joined #tripleo | 13:24 | |
| *** dshulyak has joined #tripleo | 13:24 | |
| *** rasca has joined #tripleo | 13:24 | |
| *** mgould has joined #tripleo | 13:24 | |
| *** coolsvap has joined #tripleo | 13:24 | |
| *** ramishra has joined #tripleo | 13:24 | |
| *** mkovacik has joined #tripleo | 13:24 | |
| *** derekh has joined #tripleo | 13:24 | |
| *** akrivoka has joined #tripleo | 13:24 | |
| *** jprovazn has joined #tripleo | 13:24 | |
| *** jistr has joined #tripleo | 13:24 | |
| *** ohamada has joined #tripleo | 13:24 | |
| *** paramite has joined #tripleo | 13:24 | |
| *** shardy has joined #tripleo | 13:24 | |
| *** jaosorior has joined #tripleo | 13:24 | |
| *** ifarkas has joined #tripleo | 13:24 | |
| *** pcaruana has joined #tripleo | 13:24 | |
| *** rdopiera has joined #tripleo | 13:24 | |
| *** mcornea has joined #tripleo | 13:24 | |
| *** mikelk has joined #tripleo | 13:24 | |
| *** oshvartz has joined #tripleo | 13:24 | |
| *** rcernin has joined #tripleo | 13:24 | |
| *** jcoufal has joined #tripleo | 13:24 | |
| *** numans has joined #tripleo | 13:24 | |
| *** r-mibu has joined #tripleo | 13:24 | |
| *** ccamacho has joined #tripleo | 13:24 | |
| *** shakamunyi has joined #tripleo | 13:24 | |
| *** rook-lappy has joined #tripleo | 13:24 | |
| *** bhunter has joined #tripleo | 13:24 | |
| *** quack_quack has joined #tripleo | 13:24 | |
| *** david-lyle has joined #tripleo | 13:24 | |
| *** Goneri has joined #tripleo | 13:24 | |
| *** Ryjedo has joined #tripleo | 13:24 | |
| *** trozet has joined #tripleo | 13:24 | |
| *** number80 has joined #tripleo | 13:24 | |
| *** greghaynes has joined #tripleo | 13:24 | |
| *** bswartz has joined #tripleo | 13:24 | |
| *** marcusvrn_ has joined #tripleo | 13:24 | |
| *** tremble has joined #tripleo | 13:24 | |
| *** openstackgerrit has joined #tripleo | 13:24 | |
| *** marios has joined #tripleo | 13:24 | |
| *** athomas has joined #tripleo | 13:24 | |
| *** pblaho has joined #tripleo | 13:24 | |
| *** tserong has joined #tripleo | 13:24 | |
| *** ryansb has joined #tripleo | 13:24 | |
| *** hjensas has joined #tripleo | 13:24 | |
| *** jtomasek has joined #tripleo | 13:24 | |
| *** jrist has joined #tripleo | 13:24 | |
| *** rwsu has joined #tripleo | 13:24 | |
| *** zoliXXL has joined #tripleo | 13:24 | |
| *** sambetts has joined #tripleo | 13:24 | |
| *** mburned has joined #tripleo | 13:24 | |
| *** kbyrne has joined #tripleo | 13:24 | |
| *** thrash has joined #tripleo | 13:24 | |
| *** yuanying has joined #tripleo | 13:24 | |
| *** tristanC has joined #tripleo | 13:24 | |
| *** colonwq has joined #tripleo | 13:24 | |
| *** lazy_prince has joined #tripleo | 13:24 | |
| *** michchap has joined #tripleo | 13:24 | |
| *** beagles has joined #tripleo | 13:24 | |
| *** onovy has joined #tripleo | 13:24 | |
| *** social has joined #tripleo | 13:24 | |
| *** pleia2 has joined #tripleo | 13:24 | |
| *** SlickNik has joined #tripleo | 13:24 | |
| *** larstobi has joined #tripleo | 13:24 | |
| *** lucasagomes has joined #tripleo | 13:24 | |
| *** spredzy has joined #tripleo | 13:24 | |
| *** shadower has joined #tripleo | 13:24 | |
| *** psanchez has joined #tripleo | 13:24 | |
| *** Erming__ has joined #tripleo | 13:24 | |
| *** igorbelikov has joined #tripleo | 13:24 | |
| *** dweaver has joined #tripleo | 13:24 | |
| *** adam_g has joined #tripleo | 13:24 | |
| *** dmsimard has joined #tripleo | 13:24 | |
| *** Nakato has joined #tripleo | 13:24 | |
| *** adarazs has joined #tripleo | 13:24 | |
| *** zaneb has joined #tripleo | 13:24 | |
| *** lifeless has joined #tripleo | 13:24 | |
| *** nijaba has joined #tripleo | 13:24 | |
| *** jpena|lunch has joined #tripleo | 13:24 | |
| *** lmiccini has joined #tripleo | 13:24 | |
| *** NobodyCam has joined #tripleo | 13:24 | |
| *** afazekas has joined #tripleo | 13:24 | |
| *** shanky has joined #tripleo | 13:24 | |
| *** Tristitia has joined #tripleo | 13:24 | |
| *** dtantsur has joined #tripleo | 13:24 | |
| *** Slower has joined #tripleo | 13:24 | |
| *** lynxman has joined #tripleo | 13:24 | |
| *** Ng has joined #tripleo | 13:24 | |
| *** therve has joined #tripleo | 13:24 | |
| *** matbu has joined #tripleo | 13:24 | |
| *** dobson has joined #tripleo | 13:24 | |
| *** mordred has joined #tripleo | 13:24 | |
| *** george_goh has joined #tripleo | 13:24 | |
| *** bnemec has joined #tripleo | 13:24 | |
| *** tdasilva has joined #tripleo | 13:24 | |
| *** jpeeler has joined #tripleo | 13:24 | |
| *** cmyster has joined #tripleo | 13:24 | |
| *** bandini has joined #tripleo | 13:24 | |
| *** jayg has joined #tripleo | 13:24 | |
| *** ansiwen has joined #tripleo | 13:24 | |
| *** andreaf has joined #tripleo | 13:24 | |
| *** leanderthal has joined #tripleo | 13:24 | |
| *** larsks has joined #tripleo | 13:24 | |
| *** pino|work has joined #tripleo | 13:24 | |
| *** ianw has joined #tripleo | 13:24 | |
| *** jroll has joined #tripleo | 13:24 | |
| *** trown has joined #tripleo | 13:24 | |
| *** sirushti has joined #tripleo | 13:24 | |
| *** fabbione has joined #tripleo | 13:24 | |
| *** isq_ has joined #tripleo | 13:24 | |
| *** EmilienM has joined #tripleo | 13:24 | |
| *** CaptTofu has joined #tripleo | 13:24 | |
| *** telnoratti has joined #tripleo | 13:24 | |
| *** rhefner has joined #tripleo | 13:24 | |
| *** stevebaker has joined #tripleo | 13:24 | |
| *** slagle has joined #tripleo | 13:24 | |
| *** HenryG has joined #tripleo | 13:24 | |
| *** mandre has joined #tripleo | 13:24 | |
| *** leifmadsen has joined #tripleo | 13:24 | |
| *** bkero has joined #tripleo | 13:24 | |
| *** StevenK has joined #tripleo | 13:24 | |
| *** wolfe.freenode.net sets mode: +v Ng | 13:24 | |
| *** SpamapS has joined #tripleo | 13:24 | |
| *** crinkle has joined #tripleo | 13:24 | |
| *** zigo has joined #tripleo | 13:24 | |
| *** jidar has joined #tripleo | 13:24 | |
| *** rickflare has joined #tripleo | 13:24 | |
| *** Hazelesque has joined #tripleo | 13:24 | |
| *** cinerama has joined #tripleo | 13:24 | |
| *** ChanServ has joined #tripleo | 13:24 | |
| *** wolfe.freenode.net sets mode: +o ChanServ | 13:24 | |
| *** tiswanso has joined #tripleo | 13:24 | |
| *** saneax has quit IRC | 13:24 | |
| *** saneax has joined #tripleo | 13:24 | |
| *** openstackstatus has quit IRC | 13:24 | |
| *** mgagne_ has quit IRC | 13:24 | |
| *** mgagne_ has joined #tripleo | 13:24 | |
| *** al has joined #tripleo | 13:25 | |
| *** dsneddon has joined #tripleo | 13:27 | |
| *** zoliXXL is now known as zoli|mtg | 13:27 | |
| *** sshnaidm has quit IRC | 13:27 | |
| *** gfidente has joined #tripleo | 13:28 | |
| *** openstackstatus has joined #tripleo | 13:28 | |
| *** ChanServ sets mode: +v openstackstatus | 13:28 | |
| *** uncurlhalo has joined #tripleo | 13:28 | |
| *** jdob has joined #tripleo | 13:28 | |
| *** ayoung has joined #tripleo | 13:29 | |
| *** rhallisey has joined #tripleo | 13:29 | |
| *** rbrady has joined #tripleo | 13:29 | |
| *** mgould has quit IRC | 13:29 | |
| *** mgould has joined #tripleo | 13:32 | |
| *** apetrich has quit IRC | 13:34 | |
| *** links has quit IRC | 13:35 | |
| *** trozet has quit IRC | 13:37 | |
| *** trozet has joined #tripleo | 13:37 | |
| *** tzumainn has joined #tripleo | 13:38 | |
| gfidente | ayoung you around? | 13:39 |
| slagle | EmilienM: you merged this with failing CI: https://review.openstack.org/#/c/259568/ | 13:40 |
| slagle | and it's a real failure | 13:40 |
| slagle | the patch breaks updates and deletes | 13:40 |
| dprince | michchap: I'm going to try and wire up your puppet-tripleo neutron work today. Nice job on that (complex) set of services | 13:41 |
| *** saneax is now known as saneax_AFK | 13:41 | |
| michchap | dprince: ta. I need to add plumgrid since it came after I wrote them | 13:41 |
| michchap | dprince: but otherwise I think it's looking ok | 13:42 |
| *** sshnaidm has joined #tripleo | 13:42 | |
| dprince | michchap: cool | 13:42 |
| dprince | slagle: is there a bug for the delete issue? | 13:43 |
| dprince | slagle: we can revert if needed but this patch is kind of important to land | 13:43 |
| dprince | I see it | 13:44 |
| dprince | https://bugs.launchpad.net/tripleo/+bug/1566259 | 13:44 |
| openstack | Launchpad bug 1566259 in tripleo "Upgrades job failing with ControllerServiceChain config_settings error" [Critical,Triaged] | 13:44 |
| slagle | dprince: yea, it's the same error on delete | 13:44 |
| gfidente | that is the upgrade job testing heat | 13:45 |
| *** dtrainor has joined #tripleo | 13:45 | |
| dprince | slagle: okay, yeah I'm using heat master and I'm not seeing any delete issues | 13:45 |
| numans | i am seeing the same error while deleting an overcloud.. | 13:48 |
| numans | openstack stack delete overcloud --yes | 13:48 |
| numans | ERROR: The Referenced Attribute (ControllerServiceChain config_settings) is incorrect. | 13:48 |
| *** dtrainor has quit IRC | 13:48 | |
| *** dtrainor has joined #tripleo | 13:49 | |
| shardy | dprince: AFAICT that's because we're using a 2-week old heat in CI | 13:50 |
| *** sshnaidm has quit IRC | 13:50 | |
| shardy | we already reverted the heat patch, but we're using the buggy one still | 13:50 |
| hewbrocca | can we move the pin for Heat to latest? | 13:50 |
| *** jpena|lunch is now known as jpena | 13:51 | |
| shardy | https://review.openstack.org/#/c/301554/ | 13:51 |
| dprince | shardy: +2 on your patch to bump the pin | 13:51 |
| shardy | hewbrocca: I was trying that here | 13:51 |
| hewbrocca | *nod* | 13:51 |
| dprince | shardy: but it still isn't passing for other reasons? | 13:51 |
| shardy | dprince: the upgrades job just took too long | 13:52 |
| dprince | shardy: yeah, it looks like the upgrade was totally fine. | 13:52 |
| shardy | and the nonha job failed due to a glance error | 13:52 |
| shardy | dprince: we were discussing this earlier as a topic for the meeting | 13:53 |
| shardy | we really need to figure out who has bandwidth to constantly fix things when the periodic job fails | 13:54 |
| jistr | hmm would be nicer if we could bump delorean repo pin instead of always pulling latest heat, no? IIRC we used to pull latest ironic and heat, but we stopped doing that because we were broken often. Or do i not recall this correctly? | 13:54 |
| shardy | e.g we've not promoted for 2 weeks, so how can we improve that latency | 13:54 |
| * jistr searches logs | 13:54 | |
| hewbrocca | shardy: I need to get you more folks to help :) | 13:54 |
| dprince | shardy: this is the pain w/ not being a gate :/ | 13:54 |
| shardy | jistr: Yes, well volunteered! :D | 13:54 |
| jistr | ehm :D | 13:55 |
| dprince | jistr: way to step up dude :) | 13:55 |
| *** mbound has quit IRC | 13:55 | |
| ayoung | gfidente, I'm more linear than round | 13:56 |
| *** lblanchard has joined #tripleo | 13:58 | |
| shardy | dprince: Hey did you already start an etherpad with session ideas for Austin? | 13:58 |
| shardy | I was about to start one so we can discuss it in the meeting | 13:59 |
| dprince | shardy: start one | 13:59 |
| shardy | https://etherpad.openstack.org/p/newton-tripleo-sessions | 13:59 |
| shardy | thanks | 13:59 |
| dprince | meeting time | 14:00 |
| *** sshnaidm has joined #tripleo | 14:04 | |
| *** liverpooler has joined #tripleo | 14:05 | |
| *** liverpooler has quit IRC | 14:05 | |
| *** liverpooler has joined #tripleo | 14:05 | |
| *** zoli|mtg is now known as zoli|bb | 14:10 | |
| *** zoli|bb is now known as zoli|brb | 14:10 | |
| *** apetrich has joined #tripleo | 14:11 | |
| *** mbound has joined #tripleo | 14:15 | |
| *** jprovazn has quit IRC | 14:17 | |
| sshnaidm | what is "wall time"? | 14:17 |
| *** ramishra has quit IRC | 14:18 | |
| trown | total job time | 14:18 |
| sshnaidm | trown, thanks | 14:18 |
| *** paramite has quit IRC | 14:19 | |
| openstackgerrit | Miles Gould proposed openstack/tripleo-quickstart: Update URLs to reflect move to OpenStack org https://review.openstack.org/301249 | 14:20 |
| *** Marga_ has quit IRC | 14:20 | |
| dmsimard | derekh: o/ can you let me know when you have a bit of time to talk about the tripleo symlink ? | 14:21 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic https://review.openstack.org/299537 | 14:21 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538 | 14:21 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Upload the ironic-python-agent images to cache https://review.openstack.org/301698 | 14:21 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Use the cached ironic-python-agent images https://review.openstack.org/301699 | 14:21 |
| *** julim has quit IRC | 14:21 | |
| *** rook has joined #tripleo | 14:21 | |
| derekh | dmsimard: yup, lets do it after the tripleo meeting (on at the moment) | 14:21 |
| *** ramishra has joined #tripleo | 14:22 | |
| openstackgerrit | Antoine Musso proposed openstack/diskimage-builder: dib-run-parts: make cp to target root more robust https://review.openstack.org/219453 | 14:27 |
| *** dshulyak has quit IRC | 14:29 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder: Add clean-machine-id element to remove /etc/machine-id https://review.openstack.org/301571 | 14:30 |
| hewbrocca | dprince: For what it's worth, the os-net-config bug almost certainly has to be related to | 14:30 |
| *** aufi has joined #tripleo | 14:30 | |
| hewbrocca | https://review.openstack.org/#/c/290224/ | 14:30 |
| openstackgerrit | Antoine Musso proposed openstack/diskimage-builder: dib-run-parts: make cp to target root more robust https://review.openstack.org/219453 | 14:30 |
| dprince | hewbrocca: yeah, could be. This was something I had wanted to kick the tires on a bit more :/ | 14:31 |
| hewbrocca | Yeah | 14:31 |
| hewbrocca | I remember... | 14:32 |
| hewbrocca | We could just revert this one and the following one that depends on it | 14:32 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Remove images and flavors tabs https://review.openstack.org/301720 | 14:36 |
| *** ibravo has joined #tripleo | 14:42 | |
| *** dshulyak has joined #tripleo | 14:45 | |
| rook | shardy -- is there something wrong with the patch that is holding up the +1 to workflow? | 14:48 |
| *** myoung has joined #tripleo | 14:49 | |
| pradk | gfidente, we're trying to debug an ipv6 issue in ci with adarazs and we seem to be hitting a dead end .. can you help ? | 14:50 |
| adarazs | gfidente: looks like the different services can't reach the mysql server during dbsync: http://logs.openstack.org/32/252032/39/check-tripleo/gate-tripleo-ci-f22-upgrades/3dda1df/console.html#_2016-04-05_10_12_51_937 | 14:51 |
| adarazs | which is a bit weird. | 14:51 |
| pradk | yea it fails with No route to host .. seems like a networking issue | 14:52 |
| *** ibravo has quit IRC | 14:52 | |
| shardy | rook: Only that it's not passed all the voting CI jobs | 14:52 |
| *** ibravo has joined #tripleo | 14:53 | |
| shardy | which is most likely related to a current critical bug impacting the upgrades job | 14:53 |
| *** mgould has quit IRC | 14:53 | |
| rook | ugh | 14:53 |
| rook | shardy i tried recheck -- but nada | 14:55 |
| rook | guess i need someone with +2 to recheck? | 14:56 |
| *** mgould has joined #tripleo | 14:56 | |
| pradk | shardy, whats the critical bug impacting upgrade jobs? is it same as what adarazs and I and talking above? | 14:57 |
| *** chem has quit IRC | 14:57 | |
| *** zoli|brb is now known as zoli|wfh | 14:57 | |
| *** chem has joined #tripleo | 14:59 | |
| *** derekh is now known as derekh_brb | 14:59 | |
| dprince | shardy: totally fine with changing the agenda up BTW | 15:00 |
| shardy | dprince: Cool, I've been thinking we could use a little more time for one-off items for a while | 15:00 |
| dprince | shardy: I tried it before... I think some of the categories are likely go come up on a weekly basis | 15:00 |
| dprince | shardy: like trown's item from last week. I was just going to go through those first | 15:01 |
| shardy | dprince: ack, I guess we can maybe just figure out ways to timebox certain topics then | 15:01 |
| dprince | shardy: essentially, anything that didn't get addressed previously goes first next week | 15:01 |
| shardy | in the heat meeting, we mostly focus on one-off items people want to discuss | 15:01 |
| shardy | so there's less focus on specs/bugs/ci etc | 15:01 |
| shardy | but we do have reasons for increased focus on those obviously | 15:02 |
| dprince | shardy: yeah, it is a trade off | 15:02 |
| trown | I think bugs/CI is taking alot of the meeting time, and could pretty easily be split into another meeting | 15:02 |
| trown | with a report (short) back to the main meeting | 15:02 |
| shardy | trown: Yeah, if we can get folks with time to commit to it then the CI sub-team is a great idea | 15:02 |
| *** dtantsur is now known as dtantsur|brb | 15:03 | |
| trown | well, I think everyone agrees we need folks commited to it :) meeting or not | 15:03 |
| trown | though we have always agreed on that I think | 15:04 |
| shardy | Yeah, the message I am hearing is folks mostly don't have time to commit to it | 15:04 |
| shardy | so that's the first problem to solve | 15:04 |
| dprince | shardy: yeah, if we can't land patches cause it is broken it is the thing to focus on | 15:04 |
| *** pcaruana has quit IRC | 15:06 | |
| shardy | pradk: not the same issue AFAICS | 15:06 |
| *** derekh_brb is now known as derekh | 15:06 | |
| *** lblanchard has quit IRC | 15:07 | |
| pradk | shardy, hmm ok, we're a bit lost on what could be the problem with mysql not reachable here .. http://logs.openstack.org/32/252032/39/check-tripleo/gate-tripleo-ci-f22-upgrades/3dda1df/console.html#_2016-04-05_10_12_51_937 | 15:09 |
| pradk | shardy, you have any suggestions? | 15:09 |
| *** jprovazn has joined #tripleo | 15:10 | |
| shardy | pradk: what do the overcloud logs say? | 15:10 |
| pradk | shardy, No route to host | 15:10 |
| pradk | shardy, so seems like mysql is not reachable when ipv6 is enabled but not exactly sure why.. nothing realy sticks out in the logs | 15:12 |
| *** links has joined #tripleo | 15:14 | |
| gfidente | pradk adarazs though I have seen overcloud deployed successfully in ci | 15:18 |
| gfidente | pradk adarazs where is that you're having the error? | 15:18 |
| pradk | gfidente, in upgrade ci job http://logs.openstack.org/32/252032/39/check-tripleo/gate-tripleo-ci-f22-upgrades/3dda1df/console.html#_2016-04-05_10_12_51_937 | 15:19 |
| adarazs | gfidente: according to the logs it's resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0] | 15:19 |
| pradk | gfidente, fails with DBConnectionError: (pymysql.err.OperationalError) (2003, \"Can't connect to MySQL server on 'fd00:fd00:fd00:2000::11' ([Errno 113] No route to host)\") | 15:20 |
| adarazs | well, it has like a zillion errors, but most of them is this. :) | 15:20 |
| *** aufi has quit IRC | 15:20 | |
| pradk | yea looks like services cant connect to mysql when running db sync.. so each services tries and ends up with the above error | 15:21 |
| gfidente | I think this might be oom | 15:21 |
| gfidente | it's single node deployment | 15:21 |
| pradk | hmm may be .. though i dont see anything related to memory in the logs | 15:23 |
| pradk | ha and nonha jobs finish deploy without issues | 15:24 |
| *** links has quit IRC | 15:25 | |
| openstackgerrit | Ishant Tyagi proposed openstack/os-collect-config: Add insecure option to the cfn collector https://review.openstack.org/284725 | 15:25 |
| *** mikelk has quit IRC | 15:25 | |
| *** mkovacik has quit IRC | 15:26 | |
| *** bvandenh has joined #tripleo | 15:31 | |
| *** Marga_ has joined #tripleo | 15:38 | |
| *** Marga_ has quit IRC | 15:38 | |
| *** Marga_ has joined #tripleo | 15:38 | |
| *** Marga_ has quit IRC | 15:39 | |
| rook | shardy so, whats the gist of the CI issue? It isn't something specific with my patch is it? | 15:39 |
| *** Marga_ has joined #tripleo | 15:39 | |
| *** dtantsur|brb is now known as dtantsur | 15:39 | |
| shardy | rook: we're experiencing a variety of CI issues atm, the upgrades job in your case just took too long (because the CI environment is slooow atm) so the infra timeout killed it | 15:41 |
| shardy | so it's not related to your patch, sorry for the inconvenience | 15:42 |
| shardy | rook: there are other actual errors too, where folks are continually rechecking because some problems aren't deterministic | 15:42 |
| shardy | this all contibutes to further overloading the already slow environment unfortunately | 15:43 |
| shardy | rook: over the las two runs your patch passed all the voting jobs, so it's probably fine to merge it | 15:44 |
| *** jaosorior has quit IRC | 15:46 | |
| *** leanderthal is now known as leanderthal|afk | 15:46 | |
| *** jaosorior has joined #tripleo | 15:47 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Ping retry https://review.openstack.org/298919 | 15:48 |
| *** apetrich has quit IRC | 15:48 | |
| rook | shardy: ah ok. | 15:50 |
| rook | thanks shardy | 15:51 |
| *** oshvartz has quit IRC | 15:55 | |
| hewbrocca | dprince: Ugh... we can't revert those os-net-config patches unfortunately | 15:55 |
| hewbrocca | we're going to have to fix it | 15:56 |
| bnemec | dtantsur: Regarding a backport of https://review.openstack.org/#/c/298439/ , I'm not sure. I would like to, but I'm not sure whether it qualifies as a bug fix or not. | 15:56 |
| dprince | hewbrocca: yeah. Those were added for midonet or something right? | 15:56 |
| hewbrocca | Nuage :( | 15:56 |
| dtantsur | bnemec, it might fix booting from multipath volume, no? | 15:57 |
| bnemec | On the one hand, there's no reason for this to not have worked in the past, so you could argue it's a bug. | 15:57 |
| *** cwolferh has joined #tripleo | 15:57 | |
| hewbrocca | among others | 15:57 |
| bnemec | On the other, it's not something we explicitly supported in the past, so it's kind of a new feature too. | 15:57 |
| bnemec | dtantsur: I guess I can propose the backports and if someone disagrees with them they can -1. :-) | 15:58 |
| dtantsur | bnemec, not rebuilding the ramdisk might also break hardware support for people, if they need to insert a module.. | 15:58 |
| dtantsur | though I don't have proofs for this | 15:58 |
| *** yamahata has joined #tripleo | 15:59 | |
| bnemec | dtantsur: Yeah, I think it's a good thing to be doing anyway. I'll propose the backports, if someone objects we can discuss it on the review. | 15:59 |
| snecklifter | bnemec: any chance you can review https://review.openstack.org/#/c/291243/ again? | 16:01 |
| *** jpeeler has quit IRC | 16:06 | |
| *** yamahata has quit IRC | 16:13 | |
| *** apetrich has joined #tripleo | 16:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Filter for local nodes in check_resource function https://review.openstack.org/300494 | 16:14 |
| zoli|wfh | see you tomorrow | 16:15 |
| *** jcoufal has quit IRC | 16:17 | |
| *** derekh has quit IRC | 16:19 | |
| jistr | trown: this https://review.openstack.org/#/c/301624/ has already worked for the "bad case" -- it timed out (probably stuck) and we got the overcloud logs \o/ we'll need to verify the "good case" too before we merge | 16:20 |
| trown | jistr: saw that, nice one | 16:21 |
| *** jpeeler has joined #tripleo | 16:23 | |
| *** jpeeler has quit IRC | 16:23 | |
| *** jpeeler has joined #tripleo | 16:23 | |
| *** apetrich has quit IRC | 16:25 | |
| *** panda has quit IRC | 16:26 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Map gnocchi vip to haproxy_listen_bind_param https://review.openstack.org/301798 | 16:26 |
| *** panda has joined #tripleo | 16:27 | |
| openstackgerrit | Christopher Brown proposed openstack/os-net-config: Use interface operstate to determine nic status https://review.openstack.org/291243 | 16:27 |
| snecklifter | bnemec: thanks for review, have amended as per your comment | 16:29 |
| *** sambetts is now known as sambetts|afk | 16:29 | |
| openstackgerrit | Dmitry Ilyin proposed openstack/puppet-pacemaker: Merge with fuel-infra/puppet-pacemaker https://review.openstack.org/296440 | 16:33 |
| hewbrocca | dprince: nice work on coming up with a patch | 16:33 |
| dprince | hewbrocca: yep, working on it now. | 16:34 |
| dprince | hewbrocca: give me a few and I think I'll have it | 16:35 |
| hewbrocca | rock | 16:35 |
| *** trown is now known as trown|lunch | 16:43 | |
| jistr | so analyzing the first HA logs we got from the HA job hang problem, services have a bunch of "WSREP has not yet prepared node for application use" errors when trying to talk to DB | 16:43 |
| jistr | googling says this can be a symptom of Galera cluster split | 16:43 |
| jistr | we do check for clustercheck pass on deployment though | 16:43 |
| jistr | after we merge this we'll get more logs for HA hanged jobs https://review.openstack.org/#/c/301624 | 16:45 |
| jistr | so far i just rechecked it in hope we can get a green pass on that too | 16:45 |
| *** wendar_ is now known as wendar | 16:45 | |
| *** jpena is now known as jpena|off | 16:46 | |
| *** tiswanso has quit IRC | 16:48 | |
| *** mbound has quit IRC | 16:48 | |
| *** tiswanso has joined #tripleo | 16:49 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Filter for local nodes in check_resource function https://review.openstack.org/301824 | 16:49 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Filter for local nodes in check_resource function https://review.openstack.org/301825 | 16:50 |
| *** nico_auv has quit IRC | 16:51 | |
| jistr | hmm this is also interesting http://fpaste.org/349948/87504214/ | 16:51 |
| *** apetrich has joined #tripleo | 16:51 | |
| jistr | specifically the VIP monitor and *start* failures | 16:51 |
| jistr | could be network issues, or maybe also generally things being very overloaded? | 16:52 |
| jistr | locally i've never had a failure where a VIP resource would fail to start | 16:52 |
| *** sshnaidm has quit IRC | 16:53 | |
| *** lblanchard has joined #tripleo | 16:53 | |
| openstackgerrit | Dan Prince proposed openstack/os-net-config: Add MASTER=bond SLAVE=yes to linux bond interfaces https://review.openstack.org/301827 | 16:53 |
| dprince | hewbrocca: ^^ | 16:53 |
| dprince | mburned: ^^ | 16:54 |
| mburned | dprince: ack, thanks | 16:54 |
| gfidente | jistr is it a single job you're looking at? | 16:56 |
| *** eil397 has joined #tripleo | 16:56 | |
| gfidente | jistr the saddest thing is that we actually get some greens https://review.openstack.org/#/c/300494/5 | 16:57 |
| jistr | gfidente: yes, we don't have logs for more jobs so far. There's a bug in tripleo-ci that these issues hang the whole CI job (waiting for crm_resource --wait indefinitely) and it then gets killed by the "main CI timeout", which results in not collecting machine logs. We need to merge the patch i linked above ^^ to get more logs for the hangs. | 16:58 |
| gfidente | yes I saw taht | 16:58 |
| *** akuznetsov has joined #tripleo | 16:59 | |
| *** sanjay__u has joined #tripleo | 16:59 | |
| *** akuznetsov has quit IRC | 16:59 | |
| hewbrocca | dprince: ROCK thanks | 16:59 |
| *** ifarkas has quit IRC | 17:00 | |
| gfidente | but there is all sort of strange things, this seems to be failing when creating the admin roles | 17:00 |
| gfidente | Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Command: 'openstack [\"token\", \"issue\", [\"--format\", \"value\"]]' has been running for more then 20 seconds! | 17:00 |
| gfidente | http://logs.openstack.org/24/301624/1/check-tripleo/gate-tripleo-ci-f22-ha/585ad4c/console.html#_2016-04-05_15_43_39_281 | 17:00 |
| gfidente | as if things were just very slow | 17:01 |
| *** lblanchard has quit IRC | 17:02 | |
| openstackgerrit | Dan Prince proposed openstack/os-net-config: Add MASTER=bond SLAVE=yes to linux bond interfaces https://review.openstack.org/301827 | 17:02 |
| jistr | gfidente: yea, also corosync log looks bad http://paste.fedoraproject.org/349953/14598757/ | 17:04 |
| jistr | "Corosync main process was not scheduled for 3309.2935 ms" heh | 17:04 |
| gfidente | yeah it's full of those messages | 17:04 |
| jistr | yeah several things point to general sluggishness being the cause of the CI failures | 17:05 |
| jistr | cc trown|lunch ^^ | 17:05 |
| gfidente | jistr so it seems corosync is keeps reforming the cluster every time the token goes above the threshold | 17:06 |
| *** oshvartz has joined #tripleo | 17:08 | |
| jistr | yea | 17:09 |
| jistr | maybe we could try to increase it as the message suggests :) | 17:09 |
| *** masco has joined #tripleo | 17:10 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Wait 20 minutes for tenant stack https://review.openstack.org/301832 | 17:11 |
| jistr | gfidente: i will submit a wip patch for now to test the token timeout increase approach | 17:12 |
| *** sanjay__u has quit IRC | 17:12 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Wait 20 minutes for tenant stack https://review.openstack.org/301832 | 17:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Apply os-net-config with a script instead of element https://review.openstack.org/271450 | 17:12 |
| *** apetrich has quit IRC | 17:13 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: [WIP] Increase corosync token timeout https://review.openstack.org/301836 | 17:15 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Apply os-net-config with a script instead of element https://review.openstack.org/271450 | 17:15 |
| *** weshay has quit IRC | 17:17 | |
| jistr | gfidente++ thanks for the help. WIP patch above ^^, we'll see | 17:20 |
| *** mgould has quit IRC | 17:21 | |
| *** Marga_ has quit IRC | 17:22 | |
| *** dshulyak has quit IRC | 17:23 | |
| *** jistr has quit IRC | 17:24 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Reload haproxy after injecting certs w/o pcmk too https://review.openstack.org/301840 | 17:26 |
| *** akuznetsov has joined #tripleo | 17:28 | |
| *** shardy has quit IRC | 17:34 | |
| *** ramishra has quit IRC | 17:34 | |
| *** weshay has joined #tripleo | 17:36 | |
| *** shivrao has joined #tripleo | 17:38 | |
| *** jdob has quit IRC | 17:41 | |
| *** coolsvap has quit IRC | 17:41 | |
| *** jdob has joined #tripleo | 17:41 | |
| *** coolsvap has joined #tripleo | 17:42 | |
| *** davidlenwell has quit IRC | 17:42 | |
| *** mcornea has quit IRC | 17:46 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: TEST do not merge https://review.openstack.org/301849 | 17:47 |
| *** julim has joined #tripleo | 17:48 | |
| *** davidlenwell has joined #tripleo | 17:49 | |
| *** ChanServ sets mode: +v davidlenwell | 17:49 | |
| *** bvandenh has quit IRC | 17:52 | |
| *** jdob has quit IRC | 17:53 | |
| *** jdob has joined #tripleo | 17:53 | |
| *** Marga_ has joined #tripleo | 17:53 | |
| *** mkovacik has joined #tripleo | 17:54 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Remove centos-cloud-repo element https://review.openstack.org/301851 | 17:54 |
| *** coolsvap has quit IRC | 17:54 | |
| *** sshnaidm has joined #tripleo | 17:59 | |
| *** shivrao has quit IRC | 18:00 | |
| *** lblanchard has joined #tripleo | 18:01 | |
| openstackgerrit | Dan Prince proposed openstack/os-net-config: Add MASTER=bond SLAVE=yes to linux bond interfaces https://review.openstack.org/301827 | 18:01 |
| *** trown|lunch is now known as trown | 18:02 | |
| *** ohamada has quit IRC | 18:03 | |
| *** masco has quit IRC | 18:03 | |
| *** bvandenh has joined #tripleo | 18:06 | |
| *** dtantsur is now known as dtantsur|afk | 18:08 | |
| *** akuznetsov has quit IRC | 18:13 | |
| Goneri | hi, I maintain a python lib that I use to do TripleO deployment. At this point, It's called python-rdo-m-helper and I'm about to rename it python-tripleo-helper. https://github.com/redhat-cip/python-rdo-m-helper | 18:19 |
| *** paramite has joined #tripleo | 18:23 | |
| *** bvandenh has quit IRC | 18:24 | |
| *** dshulyak has joined #tripleo | 18:24 | |
| openstackgerrit | Merged openstack/tripleo-docs: Streamline the basic deployment flow https://review.openstack.org/259197 | 18:25 |
| openstackgerrit | Dan Prince proposed openstack/os-net-config: Bump hacking in test-requirements.txt https://review.openstack.org/301863 | 18:28 |
| dprince | bnemec: is that hacking error ^^^ something you've seen yet? | 18:29 |
| dprince | bnemec: https://bugs.launchpad.net/tripleo/+bug/1566463 | 18:29 |
| openstack | Launchpad bug 1566463 in tripleo "os-net-config: hacking fails to install" [High,New] - Assigned to Dan Prince (dan-prince) | 18:29 |
| *** akuznetsov has joined #tripleo | 18:30 | |
| *** egafford has quit IRC | 18:36 | |
| dprince | slagle: you wanna have a look at this so that if it passes we can land it | 18:37 |
| dprince | slagle: https://review.openstack.org/#/c/301554/ | 18:37 |
| dprince | slagle: I'm actually wondering if we need both patches (yours and Steve's) in order to get somewhat more stable results | 18:37 |
| dprince | slagle: and should we just go on and land them both if we see any of the jobs passing | 18:38 |
| *** akuznetsov has quit IRC | 18:38 | |
| slagle | dprince: +2'd. i'm fine to push them through if we wanted to do that | 18:39 |
| slagle | i havent seen a single patch pass all 3 jobs yet this week | 18:39 |
| dprince | slagle: last 3 pass I saw was my heat patch from this weekend to t-h-t (that I abandoned) | 18:40 |
| *** rcernin has quit IRC | 18:41 | |
| dprince | slagle: okay, landed them both | 18:41 |
| dprince | slagle: lets see what that does | 18:41 |
| dprince | slagle: we are now at risk for other heat breakages though | 18:42 |
| trown | dprince: you had https://review.openstack.org/#/c/300594/1 yesterday :p | 18:42 |
| openstackgerrit | Ryan Hallisey proposed openstack/diskimage-builder: Workaround to build Fedora 23 images https://review.openstack.org/301866 | 18:42 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Test with latest master heat https://review.openstack.org/301554 | 18:42 |
| dprince | trown: oh, nice. I hadn't even noticed | 18:42 |
| slagle | dprince: yea, i think we should revert shardy's patch once we get current-tripleo promoted | 18:42 |
| dprince | slagle: ++ | 18:43 |
| trown | but there have been about 2 patches per day passing all three jobs which is pretty rough | 18:43 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Wait 20 minutes for tenant stack https://review.openstack.org/301832 | 18:43 |
| *** akuznetsov has joined #tripleo | 18:43 | |
| rhallisey | anyone know how to increase the size of the instack vm? | 18:49 |
| rhallisey | on creation | 18:50 |
| rhallisey | 30G is not enough in container land :) | 18:50 |
| *** lblanchard has quit IRC | 18:51 | |
| slagle | rhallisey: on creation, you'd have to hack instack-virt-setup | 18:52 |
| rhallisey | crap | 18:52 |
| slagle | there is a hardcoded "30" in there that you'd change | 18:52 |
| trown | rhallisey: you could try tripleo-quickstart :), it is configurable there | 18:52 |
| slagle | oh look at this guy with his fancy quickstart | 18:52 |
| trown | :) | 18:52 |
| trown | it is kind of like a giant container | 18:53 |
| slagle | after creation, i think qemu-img resize will work, and then xfs_growfs when the image is booted | 18:53 |
| rhallisey | slagle, ok I think that's the part I was missing | 18:53 |
| trown | ya, that would probably be simpler | 18:53 |
| rhallisey | trown, ya hardcord ftw | 18:53 |
| trown | s/probably// | 18:53 |
| rhallisey | :P | 18:54 |
| rhallisey | 8 node virt-virt containerized HA cluster here we come :) | 18:56 |
| jaosorior | bnemec: Hey dude, got time to check this one out? https://review.openstack.org/#/c/299475/ | 19:05 |
| jaosorior | and if someone wants to see some code removed from the loadbalancer manifest, here's some :D https://review.openstack.org/#/c/301493/ | 19:05 |
| *** akuznetsov has quit IRC | 19:11 | |
| bnemec | dprince: I can't say that I have. It isn't happening for me locally either. | 19:27 |
| dprince | bnemec: weird. I saw it on both Fedora 23 and Fedora 22 | 19:28 |
| dprince | bnemec: maybe it is a Fedora thing | 19:28 |
| dprince | bnemec: did you try cleaning your .tox directory maybe | 19:28 |
| dprince | bnemec: anyways, don't spend to much time on it. | 19:28 |
| *** pradk has quit IRC | 19:28 | |
| bnemec | dprince: My VM is still on 21. Maybe it's a dnf thing? | 19:28 |
| dprince | probably safe to bump that dependency anyways | 19:28 |
| *** pradk has joined #tripleo | 19:29 | |
| bnemec | I did tox -r | 19:29 |
| *** ibravo2 has joined #tripleo | 19:29 | |
| dprince | bnemec: yeah, Well I'll look into the code changes | 19:29 |
| dprince | bnemec: aside from the pep8 tests I needed this to fix py27 too | 19:30 |
| dprince | bnemec: I'll just carry it locally until I get the hacking issues fixed. May as well bump it I guess | 19:30 |
| bnemec | dprince: Yeah, I'm totally good with bumping our hacking version. | 19:30 |
| *** julim has quit IRC | 19:30 | |
| * bnemec should probably move his dev vm to a supported version anyway... | 19:31 | |
| *** mburned has left #tripleo | 19:32 | |
| *** mburned has joined #tripleo | 19:32 | |
| openstackgerrit | Ryan Hallisey proposed openstack/diskimage-builder: Workaround to build Fedora 23 images https://review.openstack.org/301866 | 19:33 |
| *** akuznetsov has joined #tripleo | 19:34 | |
| *** jprovazn has quit IRC | 19:34 | |
| *** Goneri has quit IRC | 19:35 | |
| *** akuznetsov has quit IRC | 19:47 | |
| *** paramite has quit IRC | 19:49 | |
| *** ibravo2 has quit IRC | 19:53 | |
| *** jaosorior has quit IRC | 19:54 | |
| *** Goneri has joined #tripleo | 19:55 | |
| *** egafford has joined #tripleo | 19:55 | |
| *** dprince has quit IRC | 19:57 | |
| *** eil397 has left #tripleo | 20:07 | |
| *** akrivoka has quit IRC | 20:10 | |
| *** apetrich has joined #tripleo | 20:17 | |
| *** ibravo has quit IRC | 20:17 | |
| *** jayg is now known as jayg|g0n3 | 20:25 | |
| *** panda has quit IRC | 20:26 | |
| *** panda has joined #tripleo | 20:27 | |
| *** david-lyle_ has joined #tripleo | 20:28 | |
| *** david-lyle has quit IRC | 20:29 | |
| *** david-lyle_ is now known as david-lyle | 20:29 | |
| weshay | trown, ok.. have a undercloud-post role now, so I'm setup to try and get net-iso working w/ the outside roles | 20:33 |
| trown | weshay: nice I saw that repo go up | 20:36 |
| weshay | ya.. the only diff w/ this one is that I'll start to use --skip-tags | 20:37 |
| trown | I think network isolation is a pretty fundamental requirement for dev environments, so I would love for that to get added to the tripleo-quickstart roles too | 20:37 |
| weshay | trown, agree | 20:38 |
| trown | since it will be a bit before it would consume the external roles | 20:39 |
| *** dtrainor has quit IRC | 20:44 | |
| *** Goneri has quit IRC | 20:48 | |
| *** rhallisey has quit IRC | 20:49 | |
| *** rhallisey has joined #tripleo | 20:51 | |
| *** rhallisey has quit IRC | 20:52 | |
| openstackgerrit | xin wu proposed openstack/os-net-config: Set check_exit_code to False for ifup https://review.openstack.org/301912 | 20:53 |
| *** ebalduf has joined #tripleo | 20:54 | |
| openstackgerrit | xin wu proposed openstack/os-net-config: Set check_exit_code to False for ifup https://review.openstack.org/301912 | 20:58 |
| *** trown is now known as trown|outtypewww | 20:59 | |
| openstackgerrit | xin wu proposed openstack/os-net-config: Set check_exit_code to False for ifup https://review.openstack.org/301912 | 21:00 |
| *** myoung is now known as myoung|afk | 21:02 | |
| *** florianf has quit IRC | 21:02 | |
| slagle | looking at a failed ha job | 21:02 |
| slagle | i assume it's not good that 2 of the galera nodes have partitioned | 21:02 |
| bnemec | Sounds :-( | 21:03 |
| slagle | i wonder if this just another symptom of stuff running too slow | 21:03 |
| bnemec | Moar hardwares plz | 21:03 |
| * bnemec should really get added to tripleo-ci-admins so he can log in to these systems | 21:04 | |
| bnemec | I wonder if we just need to drop the number of testenvs again until we get more/better hardware. | 21:04 |
| slagle | well this is all in the mysqld.log | 21:04 |
| bnemec | Yeah, but I'm wondering what the bottleneck on the hardware is. | 21:05 |
| *** dmacpher has joined #tripleo | 21:06 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 21:08 |
| slagle | bnemec: i see some cpu throttled messages on this host. i'll have to check if the timestamps match up | 21:11 |
| *** morazi has quit IRC | 21:13 | |
| *** hewbrocca is now known as hewbrocca-afk | 21:20 | |
| *** rhallisey has joined #tripleo | 21:23 | |
| *** tiswanso has quit IRC | 21:33 | |
| *** tiswanso has joined #tripleo | 21:34 | |
| *** tiswanso has quit IRC | 21:34 | |
| *** rbrady has quit IRC | 21:38 | |
| *** gfidente has quit IRC | 21:48 | |
| *** gfidente has joined #tripleo | 21:48 | |
| *** gfidente has quit IRC | 21:48 | |
| *** gfidente has joined #tripleo | 21:48 | |
| *** gfidente has quit IRC | 21:54 | |
| *** MaxPC has quit IRC | 21:59 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use split cirros image for ping test https://review.openstack.org/301951 | 22:15 |
| *** dshulyak has quit IRC | 22:23 | |
| *** leanderthal|afk has quit IRC | 23:13 | |
| *** Marga__ has joined #tripleo | 23:15 | |
| *** Marga__ has quit IRC | 23:15 | |
| *** Marga_ has quit IRC | 23:18 | |
| *** Marga_ has joined #tripleo | 23:25 | |
| *** leanderthal|afk has joined #tripleo | 23:25 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: add facts for fedora https://review.openstack.org/301174 | 23:34 |
| *** Goneri has joined #tripleo | 23:36 | |
| *** Marga_ has quit IRC | 23:37 | |
| *** akshai has quit IRC | 23:37 | |
| *** Marga_ has joined #tripleo | 23:44 | |
| *** Marga_ has quit IRC | 23:46 | |
| *** Marga_ has joined #tripleo | 23:46 | |
| *** Marga_ has quit IRC | 23:50 | |
| *** Marga_ has joined #tripleo | 23:51 | |
| *** Marga_ has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!