16:00:01 <thingee> #startmeeting Cinder
16:00:02 <openstack> Meeting started Wed Jun 24 16:00:01 2015 UTC and is due to finish in 60 minutes.  The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:07 <openstack> The meeting name has been set to 'cinder'
16:00:13 <thingee> hi all!
16:00:14 <dulek> o/
16:00:16 <vincent_hou_> hi
16:00:16 <eharney> hi
16:00:17 <geguileo> hi thingee!
16:00:18 <thangp> o/
16:00:18 <xek> o/
16:00:19 <scottda> hi
16:00:19 <geguileo> hi all
16:00:20 <cebruns_> Hi!
16:00:21 <earlephilhower> Howdy
16:00:25 <jseiler__> hi
16:00:28 <e0ne> hi
16:00:31 <dannywilson> hello
16:00:39 <patrickeast> hi
16:00:48 <liuxg> hi
16:00:49 <xyang1> Hi
16:00:49 <DuncanT> Hi
16:00:51 <erlon> hi
16:00:55 <thingee> so first off wanted to point out that liberty-1 has been tagged.
16:01:00 <thingee> has anyone seen the stats on this?
16:01:10 <jungleboyj> o/
16:01:11 <geguileo> I hear you mention them yesterday
16:01:14 <tbarron> hi
16:01:18 <geguileo> s/hear/heard
16:01:21 <dulek> These were quite high?
16:01:25 <thingee> everyone part of this team should feel really proud
16:01:27 <thingee> https://launchpad.net/cinder/+milestone/liberty-1
16:01:36 <thingee> 29 bps and 134 bug fixes
16:01:38 <thingee> 134!
16:01:50 <flip214> old bugs, or new ones?
16:01:59 <jungleboyj> Wow.
16:02:00 <ameade> o/
16:02:09 <flip214> ie. bugs from kilo fixed?
16:02:13 <vincent_hou_> Wow
16:02:31 <thingee> flip214: these were bugs that were targeted and then later there's a script that catches the ones that were in the l-1 time frame
16:02:34 <smcginnis> o/
16:02:50 <flip214> thanks for clarifying!
16:03:00 <thingee> so awesome job everyone
16:03:03 <thingee> :)
16:03:21 <thingee> alright lets get started with the agenda!
16:03:32 <thingee> https://wiki.openstack.org/wiki/CinderMeetings#Next_meeting
16:03:42 <thingee> #topic volume migration
16:03:46 <thingee> vincent_hou_, jungleboyj hi!
16:03:51 <vincent_hou_> HI
16:03:56 <asselin> o/
16:03:59 <thingee> backend rename:
16:04:01 <thingee> #link https://review.openstack.org/#/c/180873/
16:04:06 <vincent_hou_> Yes
16:04:07 <jungleboyj> Hey.
16:04:08 <thingee> spec:
16:04:10 <thingee> #link https://review.openstack.org/#/c/186327/
16:04:13 * jungleboyj looks at vincent_hou_
16:04:13 <vincent_hou_> I think it is ready
16:04:15 <jungleboyj> :-)
16:04:39 <vincent_hou_> I need reviews from cinder folks.
16:04:51 <vincent_hou_> about the back-end rename.
16:05:03 <thingee> jgriffith: around?
16:05:07 <jungleboyj> vincent_hou_: Ok, we can work ong etting reviews done.
16:05:45 <erlon> vincent_hou_: back-end rename?
16:06:03 <vincent_hou_> It is https://review.openstack.org/#/c/180873/
16:06:28 <vincent_hou_> AKA Implement the update_migrated_volume.
16:07:15 <jungleboyj> kjnelson: Will look as well.
16:07:17 <erlon> vincent_hou_: ok, ill test this and implent on our drivers too
16:07:36 <vincent_hou_> erlon: Yeah
16:08:41 <erlon> vincent_hou_: are you workign on the tempest tests too?
16:08:56 <vincent_hou_> And for the spec https://review.openstack.org/#/c/186327/, I have already started working on it and target it for L-2. Hope the spec can be merged.
16:09:26 <vincent_hou_> erlon: It is in the spec, and I am just about to start the tempest.
16:09:37 <thingee> vincent_hou_: how are you going to have gate tests of migrating from storwize to lvm, etc?
16:10:59 <vincent_hou_> thingee: I will do the case from one LVM to another LVM, and from Storwize to LVM. Jay and I are working the use cases now.
16:11:05 <erlon> vincent_hou_: ok, for the spec impl you need volume multi-attach to be merged in nova right?
16:11:27 <thingee> vincent_hou_: I get that. How are you going to do that though?
16:11:32 <vincent_hou_> thingee: I plan to get the devstack running with multiple backend
16:11:48 <thingee> vincent_hou_: infra does not have storwize machines
16:12:04 <thingee> so I'm not sure how you can gate on that case
16:12:20 <jungleboyj> thingee: I don't think we were planning to gate on that.
16:12:28 <vincent_hou_> thingee: that is part of the work I am looking on.
16:12:30 <jungleboyj> Planning to add it to the Storwize CI so it would get tested there.
16:12:31 <thingee> jungleboyj: should update the spec on that then ;)
16:12:43 <jungleboyj> thingee: :-)
16:12:49 <jungleboyj> thingee: Yes sir.
16:12:50 <DuncanT> Ceph to and from LVM we can gate on though
16:12:53 <thingee> it's listed under "tempest and gate tests"
16:12:59 <thingee> DuncanT: good idea!
16:13:10 <jbernard> that's on my todo list
16:13:18 <thingee> #idea have ceph to lvm migration in gate
16:13:36 <jungleboyj> thingee: ++
16:13:40 <erlon> thingee: +1
16:13:48 <flip214> vincent_hou_: perhaps we can get lvm <=> DRBD migration working, too.
16:13:58 <thingee> lets update the spec and get some plus ones on this if we're in agreement. would like to see this merged this week
16:14:02 <vincent_hou_> That is great.
16:14:02 <winston-d> only possible when jbernard's generic migration change lands
16:14:24 <thingee> vincent_hou_, jungleboyj thanks, anything else?
16:14:48 <smcginnis> winston-d: Do you have a link?
16:14:49 <vincent_hou_> That is all from me.
16:14:56 <erlon> winston-d: do you have the link?
16:15:08 <jungleboyj> thingee: All sounds great.  Thank you!
16:15:18 <thingee> winston-d: what do you mean?
16:15:52 <eharney> i'm also lost on the link between this spec and jbernard's generic migration work
16:16:14 <jbernard> https://review.openstack.org/#/c/187270/
16:16:34 <geguileo> eharney: To test on the gates migration between different backends
16:16:44 <thingee> jbernard, winston-d: can someone explain why this is a dependency?
16:16:50 <winston-d> thingee: ceph doesn't support migration yet
16:16:57 <winston-d> thingee: that's why
16:16:59 <hemna> mornin
16:17:34 <thingee> winston-d: ah ha so it doesn't. shows what I know
16:17:43 <jbernard> thingee: that patch adds file I/O migration support, which allows ceph to participate
16:18:50 <thingee> #info generic volume migration is needed to support volume migration with ceph in gate https://review.openstack.org/#/c/187270/
16:18:53 <winston-d> thingee: yes, jbernard has made good progress. rbd connector is landing. https://review.openstack.org/#/c/187270/ is basically functional
16:19:10 <thingee> ok thanks
16:19:37 <thingee> #topic reminder book rooms for midcycle meetup
16:19:39 <thingee> scottda: hi
16:19:42 <thingee> #link https://etherpad.openstack.org/p/cinder-liberty-midcycle-meetup
16:19:42 <scottda> hi
16:19:49 <scottda> Just a friendly reminder...
16:20:05 <scottda> Discount rooms are available, but only until Next Friday July 3rd.
16:20:20 <scottda> and 10 at $99 at one hotel, 10 at $139 at the other.
16:20:38 <scottda> So book it, since I hate to see people pay $200+
16:20:43 <scottda> that's all.
16:20:55 <thingee> #action thingee to email dev list about this reminder
16:21:00 <thingee> thanks scottda !
16:21:06 <jungleboyj> scottda: Thanks for getting those set up!
16:21:11 <ameade> awesome ty
16:21:14 <thingee> #topic Changes to how Nova calls Cinder volume detach
16:21:16 <thingee> scottda: hi!
16:21:21 <scottda> hi
16:21:21 <kmartin> if you're attending please add your name to the list https://etherpad.openstack.org/p/cinder-liberty-midcycle-meetup
16:21:22 <jungleboyj> scottda: How is the wireless?
16:21:29 <thingee> #link https://review.openstack.org/#/c/84048/44
16:21:34 <hemna> jungleboyj, it worked last time
16:21:47 <scottda> I was warned that someone was going to sneak a wifi jammer into the meetup...
16:22:01 <scottda> I am having HP security strip search this *somebody*
16:22:08 * jungleboyj whistles innocently.
16:22:10 <janonymous_> +1
16:22:12 <scottda> Anyway...
16:22:16 <scottda> Some notes under Future of 'nova volume-detach https://etherpad.openstack.org/p/CinderNovaAPI
16:22:16 <jungleboyj> oooooh.
16:22:33 <scottda> Nova devs are proposing changing the way volume detach works
16:22:33 <thingee> #link https://etherpad.openstack.org/p/CinderNovaAPI
16:23:01 <scottda> https://review.openstack.org/#/c/84048/44 is a Nova spec that discusses this.
16:23:24 <scottda> Basically, they questioned whether Nova could just 1) call libvirt.volume_detach....
16:23:37 <scottda> 2) delete entry from Nova BlockDeviceMapping
16:23:54 <scottda> 3) tell Cinder that Nova has detached and Cinder can do whatever it wants
16:24:06 <scottda> 4) Run away, without caring what happens on the Cinder side.
16:24:14 <hemna> scottda, leeantho and I found another major problem with Nova's usage of Cinder yesterday that we should talk about.  and it affects volume detaches, specifically for FC currently.
16:24:26 <hemna> so Nova needs to call cinder's terminate_connection though
16:24:38 <scottda> hemna: OK. Let's also get all this in the etherpad
16:24:41 <erlon> scottda: what does the current implentaion do? wait for cinder status?
16:24:42 <janonymous_> can i attend meetup ?
16:24:43 <hemna> because simply calling libvirt.volume_detach will have volumes show back up on the host
16:24:47 <hemna> and leave them dangling
16:24:58 <thingee> janonymous_: anyone can
16:25:03 <hemna> specifically for iSCSI volumes, that will defintely happen
16:25:23 <janonymous_> what's the venue ?
16:25:37 <dulek> janonymous_: All the info is here: https://etherpad.openstack.org/p/cinder-liberty-midcycle-meetup
16:25:50 <hemna> if the cinder driver doesn't get a chance to terminate_connection (un export the volume from the array), then iSCSI volumes can show back up on the compute host.
16:26:01 <hemna> especially if there are other iSCSI sessions to that same host
16:26:13 <scottda> I understand. We need to make sure we are communicating all these known issues to Nova
16:26:13 <hemna> the scsi subsystem will get a rescan and then that volume will show back up.
16:26:14 <janonymous_> i hv to pay to attend meeting ?
16:26:31 <hemna> the nova -> cinder interactions are so broken :(
16:26:45 <kmartin> janonymous_, it's free
16:26:46 <scottda> hemna: I agree. I tried to slow all of this down...
16:26:48 <hemna> live migration is a mess as well
16:27:09 <scottda> I think it is very rushed to try to get this fix in without a broader plan.
16:27:15 <hemna> scottda, yah
16:27:18 <winston-d> janonymous_: you can save meetup question in #openstack-cinder channel, let's not hijack scottda's agenda
16:27:22 <hemna> we need to really discuss this at the meetup I think
16:27:44 <hemna> I'm probably going to have to put a fix/change in os-brick to try and compensate for some of the nova badness at this point.
16:27:45 <scottda> agreed. I at least wanted to bring this up because of proposed changes to Nova
16:27:56 <janonymous_> m sry... jst wanted to know hw it works.. please continue
16:28:07 <scottda> WE can, of course, make note in the patch review
16:28:15 <winston-d> scottda, hemna: i'm going through all our attach/detach support tickets as well as Nova code
16:28:40 <winston-d> scottda: will add notes to etherpad and reviews
16:28:40 <hemna> scottda, winston-d specifically, this is broken in several ways:  https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L281-L294
16:28:56 <hemna> that code is called multiple times during live migration
16:29:07 <scottda> Well, we've the ongoing work and that is good. Perhaps interested persons (hemna , winstond-d) could weigh in on that Nova spec to make sure they don't go down this road..
16:29:12 <hemna> and it loses information in the BDM for FC attaches
16:29:17 <geguileo> I believe delayed detach is also broken
16:29:17 <scottda> https://review.openstack.org/#/c/84048/4
16:29:27 <hemna> scottda, ok I'll take a look at it
16:29:31 <geguileo> I hear it fails 100% of the time
16:29:32 <scottda> thanks.
16:29:57 <scottda> That's all. Just trying to make folks aware that this is going on in Nova.
16:30:22 <hemna> leeantho and I have been doing some live migration testing with os-brick patch and found some this stuff.   it's bad mmmkay
16:30:26 <scottda> hemna: winston-d we can carry on our work and discuss at the mid-cycle.
16:30:54 <hemna> the bdm's are losing data, and they don't even know it. :(
16:30:56 <scottda> shorter list is: What is not broken in cinder <-> nova API?
16:31:00 <hemna> heh
16:31:02 <winston-d> scottda: sure, thx. i will definitely join the discussion.
16:31:04 <ameade> +1
16:31:15 <geguileo> XD
16:31:28 <scottda> OK. I'm done.
16:31:32 <thingee> scottda: thanks :)
16:31:46 <thingee> #topic VersionedObjects + DateTime fields - timezone aware or not?
16:31:51 <thingee> dulek: hi
16:31:52 <hemna> I don't think most driver developers know that Nova can randomly call the driver's initialize_connection, with no intentions of attaching a volume, but simply to build the connection_info of that volume. :(!!
16:31:56 <dulek> hi!
16:31:58 <thingee> #link https://review.openstack.org/#/c/183222/7/cinder/test.py
16:32:12 <ameade> hemna: bit us once
16:32:36 <dulek> This is simple question - are our DB datetime fields timezone aware?
16:33:12 <dulek> geguileo proposed solution from the link to be able to compare datetime fields in tests.
16:33:14 <smcginnis> My understanding was UTC everywhere.
16:33:29 <thingee> ditto
16:33:31 <akerr> +1 for UTC
16:33:34 <tbarron> smcginnis: ++
16:33:37 <scottda> +1 UTC
16:33:43 <kmartin> UTC +1
16:33:44 <geguileo> The problem with tests is sqlite does not support timezones
16:34:12 <geguileo> So when you try to update a date from an object oslo versioned_objects will fail on a comparison
16:34:14 <smcginnis> geguileo: So as long as everything is stored in UTC, it doesn't matter if the DB supports it.
16:34:28 <dulek> But it matters for datetime objects.
16:34:42 * e0ne searching cross-project guideline for it
16:34:47 <dulek> datetime objects can be either timezone aware or not
16:34:49 <geguileo> smcginnis: Yeah, but I was refering to why that code was needed in the test
16:34:50 <thingee> e0ne: thanks :)
16:34:56 <DuncanT> dulek: Just touch the DB layer to decorate all dates coming out of the DB with UTC timezone
16:35:01 <e0ne> https://github.com/openstack/api-wg/blob/master/guidelines/time.rst
16:35:10 <e0ne> #link https://github.com/openstack/api-wg/blob/master/guidelines/time.rst
16:35:12 <dulek> this means - do datetime objects *know* that they are in UTC?
16:35:29 <ameade> dulek: they can, they can also be TZ naive
16:35:45 <thingee> e0ne: well that's api
16:35:50 <thingee> that's not what's stored in the db
16:35:51 <smcginnis> If I remember right they assume they are in the local machines TZ unless told otherwise.
16:36:16 <e0ne> thingee: yes:(. storing in DB was removed in review request for it:(
16:36:28 <eharney> can't we do what DuncanT says, and just stamp any that aren't tz-aware with UTC?
16:36:38 <e0ne> thingee: i hope, we use UTC everywhere
16:36:40 <dulek> Okay so which solution's better now.
16:37:03 <dulek> First - objects are tz aware and we're using geguileo's hack for tests?
16:37:03 <geguileo> dulek: Apparently patching DB access is the winner for tests
16:37:31 <ameade> oslo.utils.timeutils.normalize_time
16:37:32 <dulek> Second - we're changing datetime fields to be not timezone aware
16:37:35 <dulek> Like here: https://review.openstack.org/#/c/160417/31/cinder/objects/base.py
16:37:36 <thangp> oslo_versionedobjects assumes that it UTC by default - https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/fields.py#L328
16:37:42 <e0ne> https://review.openstack.org/#/c/159854/ - related changes in cinder
16:37:48 <ameade> """Normalize time in arbitrary timezone to UTC naive object."""
16:38:53 <thingee> ameade: +1
16:38:59 <dulek> thangp: Yes, the problem is "and self.tzinfo_aware"
16:39:34 <dulek> thingee, ameade: I don't think this will help us, we need to stick to oslo_versionedobjects conventions.
16:39:39 <thangp> dulek: tzinfo_aware=True in L310
16:40:07 <dulek> thangp: Yes, but you also can set it to False when defining a field.
16:40:28 <dulek> So do anyone have a preference for first and second solution I've mentioned?
16:40:46 <ameade> second one
16:40:56 <geguileo> dulek: Second one
16:41:26 <thangp> dulek: true, so we should just use the default tzinfo_aware=True based on this feedback
16:41:50 <thangp> dulek: or just take the default which already has it as True
16:42:15 <dulek> So ameade and geguileo are for tzinfo_aware=False, thangp thinks otherwise. Anyone else?
16:42:43 <thingee> why do we want tz aware?
16:43:02 <geguileo> thingee: No reason that I can think of :)
16:43:17 <smcginnis> thingee: +1
16:43:18 <dulek> thingee: Frankly I don't know, I've hoped someone will help me on that. ;)
16:43:19 <jungleboyj> geguileo: ++
16:43:21 <ameade> can't https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/fields.py#L328 just be calling normalize_time?
16:43:32 <thingee> thangp: do you have reservations on tz aware?
16:43:33 <ameade> i strongly dislike even the existence of timezones lol
16:43:39 <xek> dulek, I also think tzinfo_aware=False is the right solution, especially since the problems arose with the current implementation, which is not tzinfo aware
16:43:46 <eharney> it seems like the difference is tz_aware=True means you track that it's UTC up front, and tz_aware=False means that you assume it's UTC later when you use the information?
16:43:49 <thangp> if your services are distributed across different continents, then maybe
16:44:07 <dulek> ameade: Probably can't - it's in oslo, some projects may want not to use UTC.
16:44:17 <xek> dulek, the awareness of timezones can be added later, in a separate patch
16:44:18 <dulek> eharney: Correct.
16:44:20 <thingee> thangp: can you clarify what eharney asked?
16:44:29 <thingee> dulek, thangp oh nevermind thanks
16:44:50 <eharney> just as a general smell test, i'd worry more about problems coming from the latter, but i'm not going to claim i'm an expert on this
16:45:04 <thangp> the current cinder object base uses the default
16:45:23 <DuncanT> `15 minute warning (though this is the last agenda item)
16:45:26 <thingee> I guess we're in agreement with the default
16:45:34 <thangp> where tzinfo_aware=True, and so puts the timezone as UTC
16:45:37 <dulek> thangp: Yes, that's because snapshot tests apparently doesn't compare dates.
16:45:49 <thangp> dulek: yup, but volumes does
16:46:02 <thangp> dulek: and that was a pain to update all those
16:46:18 <geguileo> dulek: It's not comparison between objects, it's comparison in oslo library
16:46:39 <geguileo> dulek: When saving an object to sqlite
16:47:10 <geguileo> Ok, so tz aware objects or no?
16:47:14 <geguileo> s/no/not
16:47:17 <dulek> :D
16:47:31 <thangp> so I can go either way.  but if we are to use tzinfo_aware=False, then it should be done in the cinder base object code, where you specify tzinfo_aware=False in the fields
16:47:49 <dulek> thangp: That's right.
16:47:59 <dulek> thingee mentioned that we're in agreement to have default - that would mean tz aware objects. Does everyone have that feeling?
16:48:00 <geguileo> thangp: We agree on that
16:48:04 <thangp> https://github.com/openstack/cinder/blob/master/cinder/objects/base.py#L90
16:48:06 <geguileo> thangp: But first we must decide
16:48:15 <thingee> geguileo: I think the kwarg is confusing unless I'm still not understanding things, but if true means utc upfront, I think that's what we're agreeing on
16:48:55 <geguileo> thingee: I'm not sure that we are agreeing...
16:49:10 <geguileo> dulek and eharney Wanted to patch DB access for tests
16:49:33 <geguileo> And others wanted to change objects to be time zone unaware
16:50:11 <dulek> Well, actually I don't feel like having a preference now. ;)
16:50:22 <geguileo> dulek: XD
16:50:26 <dulek> Someone please flip a coin?
16:51:11 <ameade> not sure i can pick a side lol
16:51:16 <xek> dulek, I think the decision is yours :)
16:51:41 <thangp> it we assume UTC for everything, then tzinfo_aware should be True
16:51:45 <thangp> *if
16:52:01 <thingee> thangp: I've said that twice now, but I'm invisible right now :)
16:52:07 <smcginnis> Should it even matter?
16:52:16 <thingee> that's what i'm saying. why does this matter?
16:52:25 <thangp> thingee: :)
16:52:40 <dulek> thingee: I think we need cosistent approach with all VersionedObjects work.
16:52:42 <thangp> thingee: this is really just to have unit tests pass
16:52:50 <ameade> i thought it was because the tests or something are using sqlite which has naive datetimes
16:52:53 <dulek> Okay, let's stick to tz_aware=True and UT hack.
16:53:05 <geguileo> dulek: Ok, lets leave the objects as they are (tz aware) and I'll find the way to patch the DB access to datetime fields
16:53:06 <dulek> And we're done. :)
16:53:07 <eharney> i really think that leaving them tz aware is a better payoff than avoiding hacking tests for sqlite
16:53:08 <thingee> I'm your coin. I gave you tails both times for tz_aware. :D
16:53:20 <ameade> yeah UT shouldn't even be hitting the db right? so it's a non-issue?
16:53:22 <dulek> thingee: Thanks. :D
16:54:01 <thingee> #topic open discussion
16:54:01 <jungleboyj> eharney: ++
16:54:02 <dulek> #agreed: Datetime fields in objects should be timezone aware.
16:54:04 <tbarron> ameade: ++ :-)
16:54:06 <thingee> doh
16:54:09 <tbarron> mock
16:54:20 <tbarron> but we know lots of them do
16:54:39 * jungleboyj was hoping jgriffith would be here.  Wondering how the replication code is coming.
16:54:50 <ameade> i'll be on vacation next week so I will try to leave all the APIImpact reviews in a good state (since i'm the API WG liason)
16:55:48 <ameade> also, thank you to the folks who worked on the CI scoreboard, it's really helpful...patrickeast
16:56:00 <jungleboyj> ameade: ++
16:56:09 <smcginnis> haypo isn't here, but the question came up earlier about making the py34 gate test voting.
16:56:17 <smcginnis> My response was to wait for stability first.
16:56:30 <smcginnis> Just want to make sure there's agreement on that.
16:56:36 <patrickeast> ameade: jungleboyj: if you guys like it, make sure to check out https://review.openstack.org/#/c/194437/
16:56:36 <jungleboyj> smcginnis: ++
16:56:39 <smcginnis> I think I have yet to see it actually pass.
16:56:41 <patrickeast> its a spec to get it hosted by infra
16:56:53 <thingee> smcginnis: +1
16:57:02 <eharney> smcginnis: the idea is to have it passing on a known-good subset of tests
16:57:05 <ameade> patrickeast: right on
16:57:12 <eharney> smcginnis: so i assume that establishing that that is stable shouldn't take long
16:57:15 <cebruns> patrickeast: +1 - very cool.
16:57:39 <smcginnis> eharney: I would hope so too, but I definitely don't think it's ready to be voting yet.
16:58:01 <eharney> smcginnis: if we wait very long, the complaints will start about having to re-fix py34 from new breakages
16:58:16 <smcginnis> For the record, I do want to see it there and have compatibility.
16:58:33 <eharney> people just need to understand that that's the tradeoff being made
16:58:50 <smcginnis> But if everything has been "fixed" and the test still isn't passing, I think enabling voting would be a nightmare.
16:59:00 <liuxg> ameade: Where can I find the CI dcoreboard? Is there a link?
16:59:20 <eharney> smcginnis: we just merged the fix to make it work today?
16:59:38 <asselin> liuxg, https://git.openstack.org/cgit/stackforge/third-party-ci-tools/
16:59:50 <asselin> https://git.openstack.org/cgit/stackforge/third-party-ci-tools/tree/monitoring/scoreboard
16:59:52 <aarefiev> smcginnis: py34 job will run only subset of test, that already fixed
16:59:53 <ameade> liuxg: http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/
17:00:02 <thingee> #endmeeting