21:00:40 <jd__> #startmeeting ceilometer
21:00:41 <openstack> Meeting started Wed Aug 28 21:00:40 2013 UTC and is due to finish in 60 minutes.  The chair is jd__. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:44 <openstack> The meeting name has been set to 'ceilometer'
21:00:47 <thomasm> o/
21:00:48 <dragondm> o/
21:00:49 <dhellmann> o/
21:00:52 <terriyu> o/
21:00:52 <sandywalsh> o/
21:00:54 <herndon> o/
21:00:54 <llu-laptop> o/
21:01:00 <eglynn> o/
21:01:14 <gordc> o/
21:02:18 <jd__> #topic Review Havana-3 milestone
21:02:21 <sileht> o/
21:02:25 <jd__> #link https://launchpad.net/ceilometer/+milestone/havana-3
21:02:52 <asalkeld> o/
21:02:52 <eglynn> I've made progress on the alarm history API
21:03:02 <eglynn> (currently under review)
21:03:10 <jd__> I'm sad to announce we're probably the worst project
21:03:11 <eglynn> on the alarm service partitioner ...
21:03:25 <eglynn> jd__: harsh!
21:03:27 <jd__> so BOOO!
21:03:29 <jd__> :-)
21:03:34 * dhellmann hangs head in shame
21:03:46 * terriyu feels bad
21:03:47 <eglynn> I've been back to the drawing board a few times on the partitoner
21:03:47 <gordc> worst as in we have the most open still?
21:03:54 <jd__> more seriously, we're getting better since eglynn's back we really need more review at this point
21:04:06 <dhellmann> I've allocated time this sprint to do a lot of reviewing, though, and don't really expect to submit any code myself between now and h3
21:04:07 <eglynn> but I think I've workable and reasonably simply solution now for the paritioner
21:04:12 <jd__> gordc: yes
21:04:23 <eglynn> (with the coordination between evaluators done over fanout AMQP)
21:04:31 <eglynn> ... cleaning it up currently
21:04:32 <jd__> but Low priority are not really followed
21:04:38 <eglynn> and adding test coverage
21:04:45 <jd__> eglynn: fair enough
21:04:46 <eglynn> hope to have a patch series proposed by end of week
21:04:55 <gordc> we can close off count-api-requests bp right?
21:04:56 <sandywalsh> jd__, once I get this branch pushed up, I'll be doing a lot more review
21:05:06 <jd__> considering the amount of time needed for reviewing, the end of the week is likely the ultimate deadline
21:05:14 <jd__> sandywalsh: cool thanks
21:05:16 <gordc> i'll be doing pure reviews/bug fixes from now on.
21:05:18 <sileht> jd__, I think db-tests-with-scenarios is finished
21:05:18 <dragondm> I'm reviewing as well.
21:05:25 <jd__> gordc: I need to update our oslo copy to bring the middleware before
21:05:26 <eglynn> k
21:05:44 <jd__> sileht: if you think so, can you change the status?
21:05:44 <gordc> jd__: got it.
21:05:52 <terriyu> I just submitted a new patch for my group by blueprint today
21:05:59 <jd__> thanks dragondm
21:06:08 <sileht> jd__, sure
21:06:09 <terriyu> code review on this patch would be great https://review.openstack.org/#/c/44130/
21:06:24 <jd__> I think some effort should be done on the pagination API, the patches are now in the queue for a long time
21:06:32 <jd__> too long in my opinion
21:07:02 <gordc> did we settle the marker_pair stuff? i remember seeing a discussion on that.
21:07:07 <eglynn> do we have a definition conclusion on the marker field for paging thru meters?
21:07:11 <eglynn> snap ...
21:07:17 <gordc> eglynn: :)
21:07:43 <jd__> well I think dhellmann proposed a sane solution by using a combination of multiple fields
21:07:58 <eglynn> ah yes
21:08:00 <jd__> the problem is that a lot of the discussion has been confused between samples/meters
21:08:12 <dhellmann> yes, all of that is confusing and also irrelevant
21:08:18 <dhellmann> we don't need numerical ids
21:08:40 <dhellmann> the unique id for the value returned by the meters query is the name of the meter and the id of the resource on which that meter is collecting data
21:09:11 <dhellmann> so the marker should be the pair of those values, combined in some semi-opaque way so we can make it a single string
21:09:58 <jd__> I hope it's clear to Fengqian
21:09:58 <eglynn> but we won't rely on the caller to do the attribute concatention or anything like that?
21:10:02 <sileht> dhellmann, I agree the rest API user shouldn't be able to choose the marker
21:10:16 <gordc> dhellmann: i should track the comments from this patch? https://review.openstack.org/#/c/35454/
21:10:19 <dhellmann> eglynn: no, we give the caller a marker value and they give it back to us, they shouldn't have to know what it is
21:10:31 <eglynn> dhellmann: cool
21:10:58 <dhellmann> that's what jay was saying about not having them give us "keys" as well as values -- they just give us one value
21:11:12 <dhellmann> gordc: looking
21:11:39 <dhellmann> ok, sorting
21:11:44 <sandywalsh> right
21:11:51 <jaypipes> oh, hi guys.
21:11:53 <dhellmann> we have to sort on the marker value before any user-provided sort keys
21:11:53 <sandywalsh> it's a magic cookie
21:12:02 <dragondm> yup
21:12:04 <dhellmann> and we always sort on that value, no matter what
21:12:13 <jd__> sir yes sir! :)
21:12:16 <dhellmann> that way the marker always works, even if they don't specify a sort
21:12:21 <jaypipes> dhellmann: right, and then in addition, sort on any user-provided sort keys.
21:12:25 <dhellmann> right
21:13:05 <jd__> if anything's not clear in the review about that we need to make it clear what we expect to Fengqian otherwise I don't think it'll progress
21:13:11 <dhellmann> so we don't need to look in the database to make sure the marker is valid or anything like that any more
21:13:23 <dhellmann> ok
21:13:32 <eglynn> dumb question ... how is the marker attribute distinguished in the representation returned to the caller?
21:13:34 <dhellmann> I'll add some comments to that review
21:13:49 <jd__> and cigar number 3's coming fast
21:14:00 * jaypipes thinks it would behoove the project to first start with a cleanup of the meter table to a) rename message_id to sample_id, and b) make it the primary key... then you only ever would need to provide a single marker value...
21:14:02 <dhellmann> eglynn: it's a completely separate part of the return value, either in the "next" link or as a separate value
21:14:10 <dhellmann> so you get a list of return values, a marker, a list of links, etc.
21:14:17 <eglynn> dhellmann: a-ha, got it
21:14:25 <jd__> jaypipes: +inf
21:14:51 <jd__> jaypipes: I think it's too late but I've pushed a lot of effort in the rest of the code base toward this, so doing it so in the storage backends would be much welcome
21:14:53 * dhellmann thinks about what jaypipes said
21:15:28 <jaypipes> dhellmann: would make the sorting/marker code easier, since only single field primary keys...
21:15:40 <dhellmann> I think I see it. We're joining on the samples already, so if we return one of their ids as a marker then we don't need a composite marker? And that works because the sample id implies both a resource and a meter name
21:15:55 <jd__> that could work indeed
21:16:17 <sandywalsh> well, we should consider a proper sample table since the motivation of it is good
21:16:58 <dhellmann> "proper sample table"?
21:17:17 <thomasm> I am about to submit a bug fix that sorts on timestamp and ID as a tiebreaker.
21:17:23 <thomasm> in that function
21:17:38 <thomasm> Well, get_resources
21:17:43 <gordc> i like this message_id to sample_id idea... do we have time to get both that and pagination stuff in by next week?
21:18:05 <gordc> i would think one would have to take priority.
21:18:06 <sandywalsh> dhellmann, an actual Sample table ... a smaller table that back links to Meter, with fewer fields that represent just the change in measurement
21:18:14 <jd__> gordc: I don't think so
21:18:19 <jd__> we need to wrap up
21:18:21 * dhellmann wonders if we don't want to just say we'll need a ffe for pagination now, since it's likely
21:18:34 <dhellmann> sandywalsh: ah, ok
21:18:47 <gordc> jd__: yeah, that's what i was thinking. wishful thinking for havana.
21:18:55 <jd__> gordc: icehouse :)
21:19:43 <dhellmann> so do we say "no pagination in havana" or "pagination will not make the h3 freeze"?
21:19:59 <dhellmann> I wonder if we can get it done for havana at all, but I'm not writing it, so...
21:19:59 <dragondm> yah, & if you redo the sample table in icehouse, mebbe we should redo the timestamps in atleast the sql backend to increase precision to < 1s ?
21:20:10 <jd__> dhellmann: I think we're going to try until the end
21:20:17 <thomasm> dragondm +1
21:20:37 <jd__> since the feature have been proposed long before the freeze we can have a freeze exception I guess
21:20:43 <thomasm> It depends also on if we have a floor version for some of the backends
21:20:45 <dhellmann> jd__: cool
21:20:48 <sandywalsh> dragondm, +1
21:21:09 <jd__> #topic Ditch Alembic for Havana? (sandy)
21:21:12 <thomasm> Well, not exactly if we want to go with the Decimal approach
21:21:20 <dhellmann> if we're going for it, we should try to change the schema only as much as is needed for pagination, and not for unrelated things
21:21:24 <jd__> that's kind of a good topic as a following I guess
21:21:46 <jaypipes> dhellmann: exactly.
21:21:49 <sandywalsh> so, not rip out the alembic work, but the held up branches can use sqlalchemy-migration for H3
21:21:52 <dragondm> dhellmann: yes.
21:22:11 <jd__> I really think that it's too late for such a drastic change
21:22:18 <jd__> couldn't we just make things works?
21:22:23 <sandywalsh> (since there are big issues with alembic under sqlite)
21:22:28 <dhellmann> we don't need to take it out
21:22:36 <dhellmann> we can just allow more sqlalchemy-migrate scripts
21:22:45 <sandywalsh> right, don't remove anything, just add a few new migrations to sqlalchemy-migrate
21:22:49 <gordc> for reference: https://bugs.launchpad.net/ceilometer/+bug/1217156
21:22:50 <dhellmann> this is *exactly* why I didn't want the old scripts migrated and the old tool removed
21:22:50 <uvirtbot> Launchpad bug 1217156 in ceilometer "Alembic migrations not tested -- and they don't work for SQLite" [Undecided,New]
21:22:55 <jaypipes> I think a good compromise would be doing the sample_id PK thing and then getting the migration upgrade removed from the base test case...
21:23:37 <jaypipes> that way tests would use the sqlalchemy.MetaData.create_all() call and would not re-run migrations on every test case of the storage test cases
21:23:52 <jaypipes> and we could write specific tests for migrations in the style of Glance and Nova
21:24:03 <dhellmann> I think those are 2 separate things though, right? I agree we want both at some point, but I'm not sure we need the second now.
21:24:20 <jaypipes> heck, Alembic doesn't work with SQLite, anyway, so we can't test the alembic migrations in the unit tests..
21:24:23 <sandywalsh> right, that would be the goal, but unlikely given the time frame
21:24:47 <dhellmann> jaypipes: alembic and sqlalchemy-migrate have the same limitations with sqlite, the difference is we're monkeypatching sqlalchemy-migrate all over the place
21:25:06 <jd__> well, there's a MySQL usable in CI for unit testing I think, jaypipes
21:25:08 <jaypipes> dhellmann: yes, indeed :) I've written many o monkey patched sqlite migration :)
21:25:32 <jd__> I imagine you already know though :)
21:25:37 <jaypipes> jd__: well, that's great for testing migrations, yes. SQLite is fine for unit tests though...
21:25:39 <dhellmann> jaypipes: did you see the stuff boris's team put in oslo's db module to catch some changes and translate them automatically?
21:25:49 <jd__> jaypipes: agreed :)
21:26:11 <jaypipes> dhellmann: some of it... you talking about stuff like the unique constraint naming?
21:26:24 <dhellmann> jaypipes: also column renaming I think
21:26:56 <jaypipes> dhellmann: heh, but it doesn't work unfortunately :) or at least, it doesn't work when you try and run the very first alembic migration in Ceilometer against a SQLite db.
21:27:06 <gordc> sorry i need to drop, talk to you guys later.  jd__: my meeting topic will be posted to mailinglist.
21:27:11 <dhellmann> #link https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/migration.py#L163
21:27:29 <dhellmann> looks like it is just about unique constraints at this point
21:27:44 <dhellmann> jaypipes: right, this was just for sqlalchemy-migrate :-(
21:28:05 <jaypipes> dhellmann: right.
21:28:32 <dhellmann> anyway, did we agree to just let folks continue to write sqlalchemy-migrate scripts instead of moving to alembic for havana?
21:29:00 <jd__> hm, but don't we already have alembic stuff? isn't there an upgrade order issue likely?
21:29:39 <jaypipes> so, here's another idea... at the start of the next release cycle, we're supposed to condense all existing migrations into a single one, named for the release. How about we just put off any of these changes until the start of Icehouse, and then immediately do a single "init" Alembic migration for havana, and then delete SA-migrate entirely.
21:29:47 <sandywalsh> jd__, don't think so ... i've converted mine from alembic back to migrate and it seems happy
21:29:52 <dhellmann> jd__: did anyone successfully create an alembic script?
21:30:00 <jd__> dhellmann: I thought so?
21:30:09 <jaypipes> dhellmann: I've successfully created one, sure... running it? not so much ;)
21:30:12 <herndon> yes, there are a couple
21:30:25 <sandywalsh> dhellmann, only simple ones, the index/constraint related ones puke
21:30:26 <jd__> there's things in ./storage/sqlalchemy/alembic/versions
21:30:32 <jaypipes> herndon: I could not get either alembic migration to run.
21:30:36 <dhellmann> jaypipes: how would that work for people doing continuous deployment? they would end up with a new migration that was apparently not run but that tries to put their database into a state that it's already in
21:31:55 <jaypipes> dhellmann: we could add a single check to the migrate() routine that does an introspection of a single table that we know should have a particular schema... if the check returns True, then we simply instruct alembic that the schema is already up to date?
21:32:20 <dhellmann> jaypipes: I guess that would work.
21:32:23 * dhellmann hates rolling up migrations
21:33:04 <jaypipes> dhellmann: there is another alternative :) We could convert all of the sqlalchemy-migrate versions into alembic migrations.
21:33:13 <jaypipes> and then remove sa-migrate entirely.
21:33:17 <dhellmann> jaypipes: but if alembic doesn't work...
21:33:22 <sandywalsh> I'd rather that approach
21:33:37 <jaypipes> dhellmann: well if alembic doesn't work, then either way we're screwed :)
21:33:43 <sandywalsh> well, if alembic doesn't work, we've got larger issues
21:33:43 <dhellmann> and we have the same problem, because we have to *for each migration* figure out if it needs to be run or not
21:33:48 <sandywalsh> uh, what jay said :)
21:33:50 <jaypipes> dhellmann: since alembic is already in ceilometer :)
21:33:57 <dhellmann> but if it's not actually doing anything
21:34:02 <dhellmann> it doesn't matter, right?
21:34:08 <jaypipes> dhellmann: that's partly why I wrote that email recommending to remove one  or the other..
21:34:14 <sileht> dhellmann, we have 2 migrations scripts in alembic
21:34:20 <dhellmann> ah, well, ok
21:34:22 <jaypipes> sileht: they don't work.
21:34:35 <jaypipes> sileht: https://bugs.launchpad.net/ceilometer/+bug/1217156
21:34:36 <uvirtbot> Launchpad bug 1217156 in ceilometer "Alembic migrations not tested -- and they don't work for SQLite" [Undecided,New]
21:34:55 <jaypipes> sileht: or at least, they don't work in the current ceilometer test setup...
21:35:12 <eglynn> is the problem limited to sqlite?
21:35:21 <dhellmann> we had a long discussion about this on the mailing list, and several projects agreed to the same approach, so if we're going to change directions entirely we should bring it back there I guess
21:35:30 <dhellmann> eglynn: yes, sqlite doesn't support some forms of ALTER TABLE
21:35:49 <eglynn> dhellmann: gotcha, thanks
21:35:55 <jaypipes> eglynn: don't think so.. it's a Python error in that bug report.
21:36:32 <dhellmann> how is it that the tests are running the migrations and they are in master but are now failing?
21:36:37 <dhellmann> what happened to them in the gate?
21:36:43 <jaypipes> eglynn: "AttributeError: 'Column' object has no attribute 'create'" for the fix_meter_resource_m migration.
21:37:13 <jaypipes> dhellmann: I mention in the bug report I just do not think these migrations are running in the tests...
21:37:28 <dhellmann> ah
21:37:38 <sandywalsh> it's the alter table that was giving me the grief. Complaining that the new column already existed. I suspect there was a create_all() going on under the hood.
21:37:55 <herndon> jaypipes: what's different about them? tests from your event types patch are running and failing. does not compute...
21:38:40 <jaypipes> herndon: I'm not sure :( Perhaps there is something about my submitted alembic migration that doesn't jive with SQLite...
21:40:19 <herndon> sandywalsh: +1. jaypipes: your alembic migration is supposed to follow the existing migrations right? If there is a python error in those tests, I would never expect it to get to the point where it is trying to create event_types in alembic… am I missing something?
21:40:51 <jaypipes> herndon: that is my understanding as well... but I can't get to a testable place :(
21:41:04 <sandywalsh> so, the way I see it is, there are other people actively working on the problem. We can allow new sqlalchemy-migrations without issue and, once they resolve the problem, either convert fully to alembic or run both as they are now. Whatever they recommend.
21:41:15 <jaypipes> herndon: when I tried to "step through" the alembic migrations after running all the sa-migrate migrations successfully, i ran into that bug above :(
21:41:23 <sandywalsh> but we shouldn't take on big make-work projects like this so close to H3
21:41:37 <jaypipes> sandywalsh: my original patch had a working sa-migrate migration :)
21:41:49 <sandywalsh> jaypipes, me too
21:42:29 <sandywalsh> if anything, we should likely convert the existing alembic migrations to migrate just in case
21:42:44 <jaypipes> look, I don't mind going the alembic route at all... I tried to create a working alembic migration, and from what I can tell, it should have worked, but I can't get an environment running that enables me to test it in a step-by-step fashion :(
21:43:16 <sandywalsh> +1 ... I'm a fan of alembic, looks quite nice. It's just a rough patch for us right now
21:43:32 <jaypipes> agreed...
21:43:43 <herndon> +1
21:43:59 <sandywalsh> (heh, the three people with alembic patch issues :)
21:44:02 <jaypipes> lol
21:44:06 <jd__> fair enough
21:44:40 <jaypipes> ok, so dhellmann and jd__, should we stick to sa-migrate patches for right now, then deal with alembic (translation of sa-migration stuff and removal) in first order of business in Icehouse?
21:44:41 <dragondm> I think the problem is more that sqlite is a PITA, and we haven added the requisite hacks to alembic yet to deal w/ it.
21:45:09 <dhellmann> jaypipes: I think that's probably the best approach
21:45:10 <sandywalsh> fair point
21:45:16 <jaypipes> dragondm: and Mike Bayer has said he is not interested in hacking alembic to support failures in sqlite :)
21:45:17 <jd__> jaypipes: yes that'll be good enough
21:46:01 <jaypipes> jd__: ok. well I will volunteer to make the Icehouse sa-migrate stuff a top priority for me. I can do a lot of that work.
21:46:12 <herndon> still unresolved: do we need to move the existing alembic migrations to sa-migrate until alembic is more stable?
21:46:17 <jd__> one point we may want to stop playing with sqlite and starts mysql at test time like we do with mongo for example, if that's something possible (thinking out loud)
21:46:21 <dragondm> jaypipes: yah, and I can agree with that. (by add hacks read: workarounds so we don't need to do migrations on sqlite during tests )
21:46:31 <jaypipes> yeah
21:46:34 <jd__> jaypipes: sign here please -->
21:46:47 <jaypipes> jd__: possibly, for the sqlalchemy storage tests at least, yes.
21:46:56 * jaypipes signs paperwork.
21:47:07 <jd__> great :)
21:47:10 <jaypipes> back to what herndon was saying...
21:47:26 <jaypipes> you want me to submit a preliminary patch that moves the two alembic migrations to sa-migrate ones?
21:47:31 <jd__> herndon: I don't think we *need* to
21:47:43 <jd__> if there's enough time it might be even better to do though
21:47:53 <dragondm> jd__: +1 sqlite is fine for unittests where  we can just dump a schema in a blank db. for testing data migrations, etc, starting mysql should be acceptable .
21:47:54 <jaypipes> jd__: yeah.. I think it's worth it..
21:47:54 <dhellmann> yeah, I would be worried about anyone doing CD who has already applied those changes
21:47:59 <jd__> do you, sir jaypipes signatory of the sqlalchemy, could do that?
21:48:00 <jaypipes> dragondm: ++
21:48:05 <jaypipes> jd__: sure thing.
21:48:08 <jaypipes> I can work on that tonight.
21:48:24 <jd__> LADIES AND GENTLEMEN, HE CAN DO IT!
21:48:25 <sandywalsh> dragondm, +1
21:48:29 <jaypipes> lol
21:48:32 <jd__> jaypipes: thanks :)
21:48:47 <sandywalsh> :)
21:48:52 <jaypipes> np
21:48:55 <jd__> that'll make us ship something I'll be less ashame of in this regard
21:49:02 <sandywalsh> jaypipes gets an "atta boy"
21:49:07 <jaypipes> heh
21:49:45 <jd__> #topic Release python-ceilometerclient?
21:49:50 <jd__> I don't think we need it
21:49:54 <eglynn> me neither
21:50:09 <jd__> #topic Open discussion
21:50:13 <jaypipes> lol
21:50:24 <jaypipes> quickest topic EVAR.
21:50:47 <nealph> question on "dispatchers" and "publishers": we seem to have overlapping functionality. IIRC, we decided to keep both  so folks can push data out directly from the agent (i.e. parallel to rpc publish) and/or at the end of the pipeline.
21:51:05 <nealph> am I remembering correctly?
21:52:05 <litong> @nealph, you can not use publisher to replace a dispatcher. they are not the same as far as I can tell.
21:52:10 <sandywalsh> nealph, dispatchers are on raw notifications and publishers are with samples, iirc
21:52:30 <dhellmann> sandywalsh: right
21:52:56 <nealph> hrmmm, okay, I'll look at it a little deeper.
21:52:58 <litong> publisher deals with meter (object), dispatcher takes the data as is.
21:52:59 <sandywalsh> and dragondm's trigger pipeline will be with events
21:53:11 <dragondm> yup.
21:54:09 <dragondm> (and honestly I'd like to convert raw notification proccessing to use events instead)
21:54:55 <litong> @dragondm, not really, raw data can contain more stuff then you can possibly predict.
21:55:15 <litong> @dragondm, you do not really want to do that.
21:55:23 <dragondm> yes, but you know what you need.
21:55:39 <sandywalsh> litong, events are the stripped down version of the raw notification with the stuff you know you want
21:56:05 <sandywalsh> litong, but we will still store the full raw notification
21:56:11 <sandywalsh> (just in case, as you say)
21:56:13 <litong> that is exactly the point, you want the dispatchers to deal with the data logic.
21:56:24 <sandywalsh> ah, I see your point
21:56:28 <sandywalsh> yes, true
21:56:40 <dragondm> Yup. The reason, for one, is so we can process the data with the same tools whether it's just arrived, or has been persisted awaiting future needed data.
21:57:13 <dragondm> data logic?
21:57:38 <litong> data logic, the structure, whatever it maybe. how to interpret what is in raw data etc.
21:58:17 <sandywalsh> (like the log dispatcher, it has no idea what the notification has, but it wants to log the whole thing)
21:58:21 <litong> who knows what might be in it. but a dispatcher can be very specific. all raw data goes through a dispatcher if it is enabled.
21:58:32 <dragondm> litong: ah. yes. Have you looked at https://review.openstack.org/#/c/42713/
21:58:34 <litong> @sandywalsh, +1
21:59:07 <litong> @dragondm, no, I have not.
21:59:34 <jd__> sorry, I have to end the meeting guys
21:59:42 <litong> good day folks.
21:59:42 <sandywalsh> gg
21:59:44 <jd__> feel free to continue on #openstack-metering if needed :)
21:59:49 <jd__> have a nice day and happy hacking
21:59:51 <nealph> k. thanks!
21:59:52 <eglynn> 'night folks
21:59:56 <dhellmann> good night!
22:00:01 <jd__> #endmeeting