21:01:30 <jd__> #startmeeting ceilometer
21:01:31 <openstack> Meeting started Wed Jan 29 21:01:30 2014 UTC and is due to finish in 60 minutes.  The chair is jd__. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:34 <openstack> The meeting name has been set to 'ceilometer'
21:01:42 <dhellmann> o/
21:01:42 <dragondm> o/
21:01:46 <ildikov> o/
21:01:50 <lsmola> hello
21:01:53 <eglynn-afk> o/
21:01:53 <llu-laptop> o/
21:01:53 <gibi> o/
21:01:55 <gordc> o/
21:02:03 <tongli> o/
21:02:11 <nealph> o/
21:03:34 <jd__> #link https://wiki.openstack.org/wiki/Meetings/Ceilometer
21:03:40 <terriyu> o/
21:03:41 <jd__> #topic Milestone status icehouse-3
21:03:54 <jd__> #link https://launchpad.net/ceilometer/+milestone/icehouse-3
21:04:05 <jd__> I've removed a few blueprints for i3 already
21:04:13 <eglynn-afk> the urgency of the oslo-messaging switchover has increased recently /me thinks
21:04:14 <jd__> there's no chance we do all of that in that timeframe
21:04:20 <eglynn> #link https://blueprints.launchpad.net/ceilometer/+spec/switch-to-oslo.messaging
21:04:32 <eglynn> ... as it becomes clear that oslo-rpc is not long for this world
21:04:37 <jd__> so please drop your blueprint from the list of you know you won't do it
21:04:54 <jd__> eglynn: sileht is working on that with dhellmann and markmc already
21:04:59 <eglynn> i.e. seems oslo-rpc is being effectively locked down and IIUC fixes are only going into olso-messaging
21:05:01 <jd__> I don't think we can do more
21:05:04 <eglynn> cool
21:05:26 <DanD> o/
21:05:30 <jd__> but I do agree that it's an important stone
21:05:31 <dhellmann> eglynn: yeah, we're trying to treat the rpc code in the incubator as a "stable branch" of oslo.messaging
21:06:02 <jd__> I don't have much on that topic yet, anyone?
21:06:03 <eglynn> dhellmann: yeah so anything olso-rpc based is probably going to become increasingly problematic from a supportability PoV
21:06:13 <eglynn> (... said he with his distro hat on)
21:06:59 <dhellmann> eglynn: we may have to loosen our restrictions, but I need to keep the 2 from growing too far out of sync
21:07:15 <dhellmann> so non-critical fixes may be allowed into the incubator after they land in oslo.messaging as well
21:07:32 <eglynn> dhellmann: understood ... I think the preference all round is for all project to get off olso-rpc ASAP
21:07:35 <dragondm> btw: https://blueprints.launchpad.net/ceilometer/+spec/notification-pipelines should be ready for review in the next few weeks.
21:07:52 <dhellmann> eglynn: let me know if this causes issues internally at red hat
21:07:53 <eglynn> dragondm: cool, I'll be interested in reviewing
21:08:06 <eglynn> dhellmann: thanks, will do
21:08:08 <dragondm> eglynn: noted.
21:09:10 <jd__> #topic Tempest integration
21:09:23 <jd__> nadya_: do you have anything new?
21:09:55 <jd__> or anyone? :)
21:09:59 <_nadya__> I'm here for my part, hi all. Actually no, I don't have much about it. We are continue to work
21:10:12 <jd__> ok cool :)
21:11:04 <_nadya__> we may move on if nobody has any questions
21:11:28 <jd__> #topic Release python-ceilometerclient?
21:11:32 <eglynn> 1.0.9 is fresh off the presses ... https://pypi.python.org/pypi/python-ceilometerclient/1.0.9
21:11:40 <jd__> shiny
21:11:51 <jd__> thanks eglynn
21:12:03 <jd__> #topic Complex query expression API design (ildikov)
21:12:12 <jd__> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/025352.html
21:12:14 <tongli> @jd__, @eglynn, will there be another client release?
21:12:27 <tongli> I mean in icehouse.
21:12:30 <eglynn> tongli: sure, it's a very lightweight process
21:12:45 <jd__> I replied to that in the review, eglynn did so I saw
21:12:47 <ildikov> this topic is about the following bp: #link: https://blueprints.launchpad.net/ceilometer/+spec/complex-filter-expressions-in-api-queries
21:12:49 <tongli> @eglynn, since I am working on a BP which needs changes in the client.
21:12:56 <jd__> is there anything else we need to discuss right now ildikov?
21:13:15 <eglynn> tongli: cool, just ping me when you're ready to go with it
21:13:15 <ildikov> jd__: thanks, we replied to your comment
21:13:15 <gordc> jd__: i took a screenshot for records as requested. can confirmed you replied.
21:13:28 <tongli> @eglynn, thanks
21:13:29 <eglynn> gordc: lol
21:13:30 <ildikov> my question here would be to keep that discussion in gerrit?
21:13:51 <dhellmann> the wiki page linked from the ML thread doesn't seem to exist? https://wiki.openstack.org/wiki/Ceilometer/ComplexFilterExpressionsInAPIQueries.
21:14:02 <ildikov> jd__: it would be good to get it landed as it is up for review for a while now
21:14:10 <dhellmann> oh, nevermind, that's the mail archive adding a "." to the link
21:14:16 <dhellmann> https://wiki.openstack.org/wiki/Ceilometer/ComplexFilterExpressionsInAPIQueries
21:14:45 <eglynn> so the outstanding issue is whether the new proposed list-based syntax is problematic for jsonschema-based verification?
21:14:54 <gibi> jd__, dhellmann: for me
21:15:02 <ildikov> dhellmann:yes, it's working without the dot
21:15:06 <gibi> only the question about the filter syntax remains. I put my view in gerrit as an answer for jd__'s comment
21:15:24 <gordc> are we the first project to do this level of querying?
21:15:24 <ildikov> eglynn: yes, we could not use jsonschema with that design
21:15:52 <gordc> apologies haven't looked at it...been focused a few other items recently.
21:15:56 <ildikov> gordc:in OpenStack or in general?
21:16:09 <eglynn> TBH the dict form, when pretty-printed, does not make my eyes bleed
21:16:18 <gordc> in openstack i guess... unless you know of any other projects outside which we can reference.
21:16:28 <dhellmann> why do we need jsonschema?
21:16:53 <gibi> dhellmann: jsonschema makes the syntax validation easier, no need to write loops and ifs
21:16:59 <gordc> eglynn: agreed. one of the reasons i haven't really tracked it. it seems really... involved.
21:17:02 <jd__> ok ildikov I didn't read it yet
21:17:26 <ildikov> dhellmann: we use it for validating both the query syntax and the field names
21:17:39 <dhellmann> gibi: my only concern with that is that we are, as openstack, trying to standardize on the tools we use for web apis
21:17:47 <dhellmann> and ceilometer set the standard with pecan and wsme
21:17:59 <dhellmann> so now if we are going to change, we need a better reason than "we don't like to write for loops"
21:18:34 <gibi> dhellmann: unfortunately wsme does not support recursive stuff and jsonschema does support it
21:19:10 <dhellmann> I restate my position that the recursive expressions encoded as json are ugly.
21:19:18 <sileht> o/
21:19:24 <dhellmann> but, as I said in the review, I don't have time to write an expression parser to replace it
21:19:25 * jd__ stands with dhellmann on the unified tool side
21:19:25 <eglynn> gibi: in this context, 'recursive' == 'nesting to an arbitrary number of levels' ?
21:19:55 <ildikov> dhellmann: as gibi mentioned we had some issues with the recursive structures in wsme
21:20:03 <gibi> eglynn: yes
21:20:18 <dragondm> I know I had to parse such for the notifications stuff. that's why I used jsonpath.
21:20:27 <dhellmann> ildikov: if you write an expression parser so the caller can pass a single string expression, you don't need a recursive data structure
21:21:02 <dragondm> Er recursive, or abitrary nesting?
21:21:07 <gibi> dhellmann: string expression will contain braces for the same reason
21:21:08 <eglynn> dhellmann: is there any prior art that would address that?
21:21:13 <dhellmann> I'm talking about the difference between passing ["and", ["=", "a", "b"], [">", "c", "d"]] and: (a = b) and (c > d)
21:21:34 <eglynn> (seems a big ask to write an expression parser, given the effort invested so far)
21:21:35 <dhellmann> the documentation for PLY and pyparsing are both full of examples
21:21:52 <dhellmann> http://pyparsing.wikispaces.com/Examples
21:21:54 <Alexei_987> we could use python itself to parse expressions
21:22:02 <Alexei_987> doing some kind of eval
21:22:03 <dhellmann> http://www.dabeaz.com/ply/
21:22:08 <dhellmann> Alexei_987: eval isn't safe
21:22:11 <jd__> dhellmann: is rply relevant?
21:22:12 <Alexei_987> true
21:22:19 <gibi> dhellmann: I checked pyparsing the results are in the review as comment
21:22:21 <dragondm> Alexei_987: True, but that could be a security risk.
21:22:21 <dhellmann> jd__: possibly, I haven't used that one
21:22:28 <jd__> https://github.com/alex/rply
21:22:31 <jd__> we use that in Hy
21:22:52 <dhellmann> jd__: I'm not sure if we need to be that restrictive
21:23:07 <jd__> I can also suggest using Hy if we wanted to Lisp our API a bit
21:23:18 * jd__ *cough cough*
21:23:30 <Alexei_987> dragondm: we can do it a bit safer by passing an empty env to it and appliying some regex before
21:23:33 * dhellmann is hunting for the link to the review
21:23:38 <Alexei_987> but it's more like a crazy idea
21:23:49 <jd__> dhellmann: https://review.openstack.org/#/q/status:open+project:openstack/ceilometer+branch:master+topic:bp/complex-filter-expressions-in-api-queries,n,z ?
21:23:50 <gibi> dhellmann: https://review.openstack.org/#/c/62157/
21:23:51 <dragondm> Yah.
21:24:26 <dragondm> Alexei_987: yah, true, but that's been tried before, and there lies the PHP path of madness....
21:24:39 <Alexei_987> http://docs.python.org/2/library/ast.html?highlight=literal#ast.literal_eval
21:25:14 <eglynn> so I'm struggling to understand why ["and", ["=", "a", "b"], [">", "c", "d"]] is so bad given that we already have some hard-to-read WSME queries
21:25:25 <dhellmann> gibi: your example in http://paste.openstack.org/show/61513/ looks like it is going in the right direction
21:25:26 <eglynn> like q.field=timestamp&q.field=timestamp&q.op=gt&q.op=le&q.value=START&q.start=END
21:25:41 <dragondm> Yup, that^
21:25:51 <eglynn> s/q.start=END/q.value=END/
21:25:53 <dragondm> We already do that.
21:26:18 <jd__> the point that the current syntax is near Mongo's one is a good argument
21:26:28 <jd__> to me, at least.
21:26:42 <jd__> (even if I dislike Mongo – see how open minded I can be)
21:27:20 <Alexei_987> :) maybe new query API should be discussed in details on the next summit and become a part of V3 API?
21:27:33 <ildikov> jd__: yes, the only thing is that Mongo does not have "=" so we changed that a bit, but it's really close
21:27:34 <eglynn> well I kinda view mongo as our canonical storage driver, so that's a not unsubstantial point
21:27:34 <dhellmann> ok
21:27:46 <dragondm> Whichever we decide, I'm interested in how it works out. I think I will be doing something similar for the event triggers blueprint.
21:27:54 <dhellmann> I'm not sure why exposing implementation details is a good thing?
21:28:16 <jd__> dhellmann: it's not, but it's just reassuring somebody designed this the same way
21:28:35 <jd__> since, well I think Mongo has more users than Ceilometer :)
21:28:36 <dhellmann> ok
21:28:57 <dragondm> I don't think it's implementation details. It's a separate syntax, translated into the db query syntax. It's just usefull if the translation is simple for our most common db driver.
21:29:06 <gibi> jd__, dhellmann Do I feel right that we have an agreement here? :)
21:29:11 <eglynn> dragondm: =1
21:29:13 <eglynn> +1
21:29:32 <ildikov> dragondm: +1
21:29:43 <jd__> agreement probably not, but a "good enough" from me, likely
21:29:45 <jd__> :)
21:29:46 <Alexei_987> I'm going to put a -1 on it cause this syntax is far from user friendly
21:30:02 <eglynn> good enough for me too TBH
21:30:08 <Alexei_987> it's going to be quite hard to write it by hand
21:30:10 <dragondm> Alexei_987: for which? the mongo-ish one or the lists?
21:30:16 <Alexei_987> the lists
21:30:39 <Alexei_987> mostly the way how it's passed in request
21:30:59 <tongli> I think that the proposed solution is to use POST
21:31:12 <tongli> which is not very common.
21:31:25 <gibi> Alexei_987: we are trying to agree on the mongoish syntax here. Passing the json in a string is not the best thing but wsme limits us there
21:31:33 <ildikov> Alexei_987:I'm not sure that the rest api is for human to machine interface or not in most of the cases
21:32:06 <Alexei_987> ildikov: I agree but I still think that good API should be human readable
21:32:12 <ildikov> Alexei_987: and maybe if the Mongo syntax is a good enough here, than we could keep it as a first step
21:32:47 <Alexei_987> I'm ok with mongo syntax itself I don't like the way how it's passed to api
21:33:03 <dragondm> +1 for the Mongo-like syntax.  As for passing as a string.... Bleh...
21:33:08 <Alexei_987> cause json sting is not good enough for GET requests as mentioned by tongli:
21:34:05 <tongli> so something like this?
21:34:07 <Alexei_987> I would like to see a custom query url something like ?a>b&c<10
21:34:08 <tongli> find( { type: 'food', price: { $lt: 9.95 } } )
21:34:58 <Alexei_987> "type='food'&price<9.95" ?
21:35:33 <gibi> Alexei_987: that syntaxt gets complicated with "and" "or" and braces for precendence
21:35:45 <ildikov> Alexei_987: it also can be a mess if you have also or and not as a supported operator, or in and you have to write lists and brackets
21:35:52 <Alexei_987> I agree
21:36:02 <Alexei_987> but proposed syntax will have to be encoded
21:36:08 <Alexei_987> to pass it via GET
21:36:43 <Alexei_987> after encoding it's going to look like a complete mess
21:36:51 <eglynn> IIUC this is POST-only
21:37:03 <Alexei_987> but query API is GET by design
21:37:07 <dragondm> Btw: the way tongli phrased it is even cleaner.
21:37:31 <jd__> passing a body in GET is possible in theory if you want
21:37:32 <tongli> @dragondm, I thought that is how mongodb specifies conditions.
21:37:34 <dragondm> (i.e. flipping the dictionary around a bit.
21:37:34 <jd__> problem solved.
21:37:39 <jd__> can we move on?
21:37:40 <ildikov> Alexei_987: the Mongo-like syntax is still seems to be easier to process and transform to the db queries
21:37:54 <Alexei_987> ildikov: agree
21:38:02 <ildikov> tongli:yes, it's the exact Mongo syntax
21:38:04 <Alexei_987> jd__: +1
21:38:05 <lsmola> +1 for json
21:38:13 <dragondm> tongli: Yes it is. I was comparing that to the slightly more complex syntax in the review.
21:38:31 <gibi> in that sense mongo is inconsistent as "and" "or" is prefix op but $lt is infix
21:38:44 <jd__> I think we're good on this one guys, I'll move on now
21:38:47 <gibi> so we made everythin infix
21:38:48 <lsmola> ildikov, btw. it is also similar to elastic search queries right?
21:38:56 <ildikov> and also does not have "="
21:39:02 <jd__> #topic status of Alembic migrations (gordc)
21:39:12 <ildikov> lsmola: yes, elastic search use json, IIRC
21:39:19 <tongli> @ildikov, can I propose for simpler ones, still support GET
21:39:20 <jd__> feel free to continue in #openstack-ceilometer
21:39:26 <jd__> gordc: floor is yours
21:39:36 <gordc> ok. so this is just a general question. but currently we have 4 alembic scripts which were written sometime around the 010 migrate script
21:39:36 <ildikov> jd__:sure, sorry
21:39:59 <gordc> everytime we check in a new migrate script we need to check the alembic scripts to make sure they're ok to run after the new script
21:40:14 <gordc> does anyone know what the status is for alembic? is it ready?
21:40:20 <Alexei_987> I know
21:40:30 <gordc> Alexei_987: go for it :)
21:40:37 <Alexei_987> so the current status is the following:
21:40:43 <eglynn> wasn't there a proposal a while back to back-port the alembic migrations to sqlalchemy-migrate?
21:40:56 <gordc> i'd like to either drop alembic or switch very soon... it's bothering me.
21:41:03 <Alexei_987> 1) alembic itself is 100% ready for production and we can switch to it at any moment
21:41:17 <viktors_> Alexei_987: you talking about this? https://github.com/openstack/oslo-incubator/tree/master/openstack/common/db/sqlalchemy/migration_cli
21:41:23 <gordc> eglynn: yeah, in havana... but that died... i'm bringin it back.
21:41:31 <Alexei_987> no.. just ceilometer implementation
21:41:39 <Alexei_987> 2) alembic doesn't have a full support for sqlite alter operations
21:42:00 <Alexei_987> cause of 2* it's not working correctly in tests or requires a lots of hack to make migrations work
21:42:06 <dragondm> sqlite == boatanchor. :P
21:42:06 <jd__> I think we should switch but nobody knows what should be done
21:42:10 <jd__> or wants to do it :)
21:42:27 <Alexei_987> 3) we are working on a plan to avoid using migrations in tests at all
21:42:46 <Alexei_987> in tests we create an empty DB and we can create it directly from models metadata
21:42:49 <gordc> Alexei_987: is it any worse than sqlalchemy-migrate? i've tried migrate and some of our downgrade scripts don't work on sqlite... and others just have skip clauses
21:43:10 <Alexei_987> downgrade is another topic that should be discussed
21:43:26 <Alexei_987> I think that we should officially stop supporting downgrade for migrations
21:43:36 <Alexei_987> there is no sense in writing them
21:43:57 <Alexei_987> for some migrations there is no way to revert DB to previous state
21:44:08 <Alexei_987> and some use dirty hack like dump_ tables
21:44:26 <Alexei_987> I don't think that someone would ever use downgrade in production
21:44:29 <gordc> Alexei_987: yes... not a fan of those dump tables.
21:44:45 <Alexei_987> back to the main topic:
21:44:49 <gordc> regardless, Alexei_987you seem to be tracking alembic.
21:44:55 <Alexei_987> true
21:44:56 <gordc> you think it's good enough to migrate?
21:45:00 <Alexei_987> yes
21:45:16 <eglynn> what does migrate mean in this context?
21:45:27 <Alexei_987> create or update db structure
21:45:29 <gordc> compared to sqlalchemy-migrate
21:45:32 <eglynn> all *future* migrations expressed in alembic?
21:46:12 <Alexei_987> 4) to implement 3* we need to be sure that models metadata creates the same structure as migrations
21:46:21 <dragondm> Alexei_987: If a rollout goes bad, and there's no downgrade, that would be a serious issue for any ops/ deploy folks.
21:46:21 <Alexei_987> which is not true for now
21:46:47 <Alexei_987> dragondm: if a devops doesn't have a daily backup there is a serious issue already
21:47:05 <gordc> yes. all future migrations in alembic... not sure what we do with sqlite.
21:47:06 <Alexei_987> and our update procedure clearly states of the need of db backup in any case
21:47:07 <eglynn> we tried issueing the alembic-only-from-now-on edict before and we had to resile on it
21:47:19 <Alexei_987> eglynn: yes cause of sqlite issues
21:47:21 <dragondm> Alexei_987: if the answer is 'restore from backup'  most devops will reject it.
21:47:30 <dragondm> That's loose data.
21:47:34 <dragondm> er that'd
21:47:38 <gordc> eglynn: should we continue with migrate until we can get mysql/postgresql testing implemented?
21:47:50 <gordc> sqlal-migrate that is
21:47:58 <eglynn> gordc: that seems wise to me
21:48:01 <Alexei_987> dragondm: if the answer is "restore from downgrade migration that was not properly tested" it's not good either
21:48:20 <Alexei_987> gordc: you are right real backend will help us much
21:48:21 <dragondm> Alexei_987: yah, it should be tested too :>
21:48:30 <gordc> eglynn: seems like a safer approach... so regarding current alembic scripts
21:48:52 <jd__> eglynn: I thought so too
21:49:00 <Alexei_987> dragondm: as I said some migrations simply cannot be reverted
21:49:07 <Alexei_987> like change 1-* to *-*
21:49:17 <gordc> do we dump them for sql-migrate scripts or do we continue to risk it.... i have a patch to undo them...but if we make no more changes to db, there's no point to undo them.
21:49:40 <gordc> them being alembic scripts
21:49:43 <dragondm> Alexei_987: yah, that's the reason for copying  some data to a temp table. Ugly. But I've seen it needed w/ nova.
21:50:27 <Alexei_987> dragondm: but it also means loosing data, no?
21:50:34 * dragondm would much prefer alembic if it's ready.
21:50:55 <Alexei_987> cause you cannot fit new data in older structure
21:50:59 <jd__> could someone write up a plan on what we need to do in a blueprint or the like?
21:51:05 <Alexei_987> I can do it :)
21:51:06 <dragondm> Alexei_987: Less than the X hrs between rollout an the last backup.
21:51:14 <gordc> dragondm: same. but it doens't handle sqlite and mysql/postgresql test implementations aren't ready.
21:51:15 <jd__> because I think that'd help a lot
21:51:39 <jd__> it's likely we want to have our integration test running first indeed if we drop sqlite
21:51:53 <Alexei_987> jd__: I (and my team) can also take part in implementing tests for real backends
21:52:00 <jd__> Alexei_987: great
21:52:09 <jd__> go ahead then :)
21:52:26 <Alexei_987> agreed on this
21:52:27 <gordc> how's this for a plan. i keep the patch to undo alembic scripts as WIP. we apply it if we ever run into a migration script that affects the existing alembic scripts.
21:52:30 <dragondm> gordc: Yah, alembic never will handle sqlite, migrate only handles is at all due to hacks.  My vote would be to have a schema dump loaded into sqlite at the beginning of the test.
21:52:55 <gordc> and then switch to alembic when mysql/postgresql testing implementation is ready?
21:53:13 <viktors_> dragondm: yes, we are going to do it
21:53:16 <Alexei_987> dropping sqlite support will also bring additional benefits
21:53:16 <dragondm> gordc: For the while it could be checked in, and verified by the tests, like the default /etc  conffiles.
21:53:26 <Alexei_987> we won't have to compact migrations anymore
21:53:30 <viktors_> gordc: in process now
21:53:51 <dragondm> viktors_: Yay. +1e9
21:54:23 <dragondm> Will also speed up tests abit.
21:54:25 <Alexei_987> cause for empty DB we'll create it directly from metadata. And for existing DB we only have to support migrations for 2 cycles
21:54:46 <Alexei_987> so we can simply drop migrations that are more than 2 cycles old
21:55:20 <Alexei_987> jd__: so let's make a summary for this?
21:55:30 * gordc trying to read through what's been typed to see if there's a takeaway
21:55:32 <jd__> Alexei_987: yes, I make you in charge of that ;)
21:55:42 <Alexei_987> 1) I'll create a plan for alembic work
21:55:43 <jd__> having a blueprint would be good
21:55:54 <Alexei_987> 2) we take part in speeding up impl. of backends tests
21:56:06 <Alexei_987> 3) switch to alembic when it's ready
21:56:16 * jd__ nods
21:56:19 <gordc> 2) is key
21:56:20 <Alexei_987> 1* as a blueprint
21:56:38 <Alexei_987> for 2* I already have 2 patches in merge stage
21:56:51 <Alexei_987> and I'm waiting for one of them to merge
21:56:59 <Alexei_987> for 3 days already :(
21:57:03 <jd__> which one?
21:57:14 <Alexei_987> https://review.openstack.org/#/c/67850/
21:57:20 <Alexei_987> this oslo session fixes
21:57:34 <Alexei_987> stupid gate bugs make it restart for the 3rd time
21:57:38 <jd__> erf
21:58:13 <Alexei_987> ok so to the next topic?
21:58:18 <jd__> no next
21:58:21 <jd__> #topic open discussion
21:58:27 <jd__> you got one minute
21:58:40 <gordc> i think my worries about alembic have been tamed... a little bit
21:58:46 <jd__> :)
21:59:42 <jd__> considering the silence, I declare the end of this meeting
21:59:45 <jd__> #endmeeting