| *** VW has quit IRC | 00:49 | |
| *** VW has joined #craton | 00:49 | |
| jimbaker | anonymike, works for me | 00:54 |
|---|---|---|
| *** VW has quit IRC | 01:40 | |
| *** VW has joined #craton | 01:40 | |
| openstackgerrit | Jim Baker proposed openstack/craton master: Updates Alembic migration to better match SQLAlchemy models https://review.openstack.org/441644 | 04:16 |
| anonymike | running into some errors with dbsync :/ I'm gonna mess around for another hour or so | 05:08 |
| *** openstackgerrit has quit IRC | 09:03 | |
| sulo | yawn | 12:20 |
| sigmavirus | belated mornings | 12:41 |
| thomasem | Hi! | 12:47 |
| tojuvone | good afternoon | 12:59 |
| sigmavirus | thomasem: reading scrollback, are we really going to constrain variables like that? | 13:30 |
| sigmavirus | I can imagine folks writing some integration with craton from clojure (or another lisp) and wanting to use hyphenated var names | 13:30 |
| sigmavirus | (in which hyphenated var/function names are perfectly valid) | 13:31 |
| sigmavirus | I guess, I'm wondering if we want to restrict the functionality of the service purely because the first client written for it was written in a specific language | 13:31 |
| thomasem | sigmavirus: it's part of the API WG guidelines, though? | 13:36 |
| thomasem | It's an openstack API where fields are snake_case | 13:36 |
| sigmavirus | link? | 13:36 |
| sigmavirus | but variables aren't fields | 13:36 |
| sigmavirus | variables are user-defined data | 13:36 |
| sigmavirus | &/ user-defined metadata | 13:36 |
| sigmavirus | If you look at other openstack services that allow for user-defined metadata there's no such restrictions | 13:36 |
| sigmavirus | In fact, there's also Glance's Metadefinitions service that has things lke OS::Nova::InstanceType::.... | 13:37 |
| thomasem | Okay, then why was jimbaker saying that variable keys are supposed to be constrained to Python identifiers? | 13:37 |
| thomasem | If that's not the case, I'm cool w/ supporting hyphenated and going through that exercise, but I was under the impression that that was already decided upon as the direction. | 13:38 |
| thomasem | And adding complexity for what "might be" seemed a bit unwise right now. We could lift those restrictions later on if we find that we want to add that, though. I'm personally fine either way, it's a matter of what use-cases we have. | 13:39 |
| sigmavirus | So, right now, ansible/salt only allow variables with _s | 13:41 |
| *** VW has quit IRC | 13:41 | |
| sigmavirus | I'm checking chef and puppet given that ruby symbols can have hyphens and could ostensibly be used for variables | 13:41 |
| sigmavirus | I wasn't aware that this was already decided direction | 13:41 |
| *** VW has joined #craton | 13:42 | |
| thomasem | That's the impression I got. But, of course, I don't have the history that y'all do on the project. | 13:42 |
| sigmavirus | I guess my question to you/jimbaker is: What value to the user(s) is restricting variable keys in that manner providing? What benefit do we see from restricting values that way? Why would we restrict users from specifying CamelCase or even StuDlYCase variables? | 13:42 |
| sigmavirus | I also miss conversations about direction that happen in IRC given how not durable it is | 13:43 |
| sigmavirus | (And I forget discussions from the past) | 13:43 |
| thomasem | Lol, don't we all. | 13:43 |
| thomasem | It's all good. I'm glad you're bringing it up. | 13:43 |
| sigmavirus | Would that this was communicated over the mailing list, that'd be great =P Or via a specification | 13:43 |
| thomasem | Yeah, this should have been in a specification to begin with. | 13:44 |
| sigmavirus | honestly, having a spec to describe the direction taken at the end of this would be valuable to end-users | 13:44 |
| thomasem | Pressure to deliver and such | 13:44 |
| sigmavirus | Yeah, doesn't seem there's a spec around variables: https://craton.readthedocs.io/en/latest/specs/index.html Although I could have sworn one was sort of written before | 13:44 |
| thomasem | Yeah, git-harry was working on one, iirc. | 13:44 |
| thomasem | And then variables just... happened | 13:45 |
| sigmavirus | Was that spec just abandoned? | 13:45 |
| thomasem | Yep | 13:46 |
| sigmavirus | If so that's a crying shame | 13:46 |
| thomasem | That's what happens when people want things out the door _now_. | 13:46 |
| thomasem | unfortunately | 13:46 |
| thomasem | So, we need to pick it back up and finish it, imo. | 13:46 |
| thomasem | And generate from that any changes to get things consistent and how we want them to be. | 13:47 |
| *** openstackgerrit has joined #craton | 13:47 | |
| openstackgerrit | Ian Cordasco proposed openstack/craton master: Update status of existing specifications https://review.openstack.org/449157 | 13:47 |
| sigmavirus | Right | 13:47 |
| sigmavirus | Any constraints on variable names and values should be enumerated in that spec | 13:47 |
| sigmavirus | To be clear, I really want reasoning as to *why* we want to restrict them in such a way | 13:48 |
| thomasem | Agreed. | 13:48 |
| sigmavirus | What value does Craton derive from that? What value do the users derive from that? etc. | 13:48 |
| thomasem | Well, atm, it'd simplify the implementation while covering the most immediate use-case. | 13:49 |
| thomasem | But, that feels like a cop-out. | 13:49 |
| sigmavirus | Well the fact that itertools chain existed and simplified variables resolution was the reasoning for using a python-specific resolution ordering so | 13:50 |
| sigmavirus | ¯\_(ツ)_/¯ | 13:50 |
| anonymike | Chrome does NOT like owa :/ | 13:54 |
| thomasem | sigmavirus: yeah... Lemme refactor this a bit and it might be a non-issue. | 13:55 |
| thomasem | This is helpful: https://dev.mysql.com/doc/refman/5.7/en/json-path-syntax.html | 13:55 |
| thomasem | Except for we do need to agree on something here. | 13:55 |
| thomasem | anonymike: :( | 13:57 |
| fsaad | hi guys | 14:05 |
| sigmavirus | So Im' looking at the spec language for valid ECMA identifiers | 14:07 |
| sigmavirus | I'm 90% sure that - isn't allowed, but jsonpath also supports "strings" | 14:07 |
| sigmavirus | Which says that -'s are fine | 14:07 |
| thomasem | Yes, you have to wrap the key in quotes. | 14:07 |
| thomasem | So, things like $."hyphenated-key" | 14:08 |
| thomasem | Or $."hyphenated-key"[*].key | 14:08 |
| thomasem | and so on | 14:08 |
| sigmavirus | Hyphens are in the "Punctuation, Dash" unicode category fwiw | 14:09 |
| sigmavirus | (Makes sense, if you know all the categoreis) | 14:09 |
| sigmavirus | So while hyphens complicate the experience for users, I don't think that's a valid reason for restricting it inherently | 14:09 |
| sigmavirus | Rather, it is valid from a "don't give users footguns" but also seems like not very convincing reasoning to me | 14:10 |
| sigmavirus | Especially if we appropriately provide documentation on how to use the variable searching functionality | 14:10 |
| thomasem | sigmavirus: Are you suggesting that we would expect the user to quote it in the query? | 14:11 |
| thomasem | I was thinking we'd just handle that on our side so the user doesn't have to care... but, that may be one of those cases where you harm them by trying to be helpful in the long-run. | 14:12 |
| sigmavirus | Yeah, in the face of ambiguity like that, I tend to lean towards "educate the user about their life choices" | 14:12 |
| thomasem | Yeah. I was considering that we could just error when they give us something that the DB doesn't understand. | 14:13 |
| sigmavirus | like, as far as I can tell, we're already exposing them to jsonpath query language, right? | 14:13 |
| thomasem | Basically. | 14:13 |
| sigmavirus | I mean, we're going to document these queries, we may as well provide several examples covering use-cases | 14:14 |
| sigmavirus | Besides, I'm not sure we really want to tell users that the variable naming scheme their config management project uses is not storeable in craton | 14:14 |
| sigmavirus | if someone stores sshKeyId and someone else stores ssk_key_id, do we really need to care? | 14:14 |
| thomasem | We don't. | 14:15 |
| thomasem | It only becomes a problem when you start clashing with special characters for the language, naturally. | 14:15 |
| thomasem | So, really, we need to just handle the DB error and tell them they did it wrong. | 14:16 |
| sigmavirus | sure, but the way we're doing this in python right now doesn't clash with anything | 14:16 |
| sigmavirus | Yep | 14:16 |
| sigmavirus | We could even detect -s and suggest they try adding quotes | 14:16 |
| sigmavirus | (if we want to be fancy) | 14:16 |
| thomasem | Well I'm a fancy guy, so. | 14:16 |
| sigmavirus | I've not seen you wax your mustache, so I've no evidence of that =P | 14:17 |
| thomasem | LOL | 14:17 |
| sigmavirus | where are your tophat and monacle? eh? | 14:18 |
| anonymike | ha | 14:18 |
| thomasem | I had to stow them away because I became another person when I wore them. | 14:18 |
| thomasem | I criticized peoples' choice of beverage and blew cigar smoke in the direction of people I disagreed with. | 14:19 |
| sigmavirus | ah, as long as you didn't rove around london at night with a different name maiming people | 14:20 |
| thomasem | ... Who are you? | 14:20 |
| thomasem | And what do you know? | 14:20 |
| thomasem | :P | 14:22 |
| sigmavirus | I know NOTHING! https://i.imgur.com/nQdCiwp.jpg | 14:22 |
| thomasem | Lol, who's that? | 14:27 |
| anonymike | relocating, bbiab | 14:48 |
| sigmavirus | thomasem: you're making me feel old | 14:59 |
| sigmavirus | I'm not even 30 yet | 14:59 |
| sigmavirus | thomasem: https://www.youtube.com/watch?v=UgcxGFmYyPs | 15:00 |
| sulo | sigmavirus: thomasem: fsaad: i might miss the meeting today .. i need to step our for a bit .. might not be back in time. | 15:03 |
| sigmavirus | okay sulo | 15:03 |
| fsaad | ok thanks for heads up sulo | 15:10 |
| thomasem | sigmavirus: LOL I've never seen this show. | 15:11 |
| thomasem | sulo: No worries! | 15:12 |
| sigmavirus | thomasem: I (obviously) only ever saw it in re-runs | 15:13 |
| sigmavirus | Was on when my dad was a kid | 15:13 |
| sigmavirus | Sargent Schultz is one of his favorite characters on the show | 15:13 |
| sigmavirus | It's about British and American POWs in a Nazi run POW camp | 15:14 |
| anonymike | lol I've never heard of this show | 15:14 |
| sigmavirus | The humor is corny but it is still funny | 15:15 |
| anonymike | sometimes I like to have that kind of stuff playing in the background while I work on other projects | 15:16 |
| sigmavirus | anonymike: ah, but some of the jokes are fairly subtle | 15:16 |
| sigmavirus | So, you'll miss part of it | 15:16 |
| anonymike | i like the a e s t h e t i c | 15:17 |
| sigmavirus | ¯\_(ツ)_/¯ | 15:17 |
| sigmavirus | I'm not a cop, I'm just a sign | 15:17 |
| anonymike | lol | 15:17 |
| jimbaker | sigmavirus, thomasem, we are restricting variables names beyond possibly python identifiers? | 15:32 |
| jimbaker | news to me | 15:32 |
| sigmavirus | jimbaker: no, I'm wondering why we want to restrict them to valid python identifiers | 15:33 |
| thomasem | jimbaker: We aren't restricting anything yet. I was suggesting that we define a restriction. We discussed (I think last week) and it sounded like the expectation was that variable key names are supposed to be valid python identifiers? | 15:33 |
| thomasem | but, at this point, I agree that there's little need to restrict them to valid python identifiers. | 15:33 |
| sigmavirus | jsonPath has no such restriction | 15:34 |
| thomasem | http 'http://localhost:8080/v1/hosts?details=all&resolved-values=false&vars=foo.foo."nested-int":1' $CRATON_HTTP_HEADERS works just fine, btw. | 15:34 |
| sigmavirus | ^bingo | 15:34 |
| jimbaker | it has come up that it might be a good idea | 15:34 |
| thomasem | Of course, without the quotes it's returning 500, which I'm handling right now. | 15:34 |
| sigmavirus | jimbaker: what's the argument behind it being a good idea though? that's what I want to understand | 15:35 |
| sigmavirus | I've seen no reasoning behind that assertion personally | 15:35 |
| openstackgerrit | Merged openstack/craton master: Update status of existing specifications https://review.openstack.org/449157 | 15:35 |
| jimbaker | sigmavirus, i think it was with respect to 1. we want to reserve some nomenclature for namespaces - but even then it's just using the fact that there is no restriction | 15:36 |
| jimbaker | 2. we may want to have some sort of reserved keys | 15:36 |
| jimbaker | 3. for client usage, there was a consideration that attribute form could be nice. but we didn't implement that | 15:37 |
| sigmavirus | So I think attribute form would be a nightmare on the client side | 15:37 |
| thomasem | "attribute form" or "that attribute form"? | 15:37 |
| jimbaker | sigmavirus, in other words, there has not been a strong arg for restriction, or at least not strong enough | 15:37 |
| jimbaker | thomasem, being able to say obj.var1, obj.var2 | 15:37 |
| jimbaker | didn't implement, not going to implement | 15:38 |
| sigmavirus | Right now our crud.Resource object can handle anything that's a valid Python identifier, yes, but I would only abuse that for things that are knonw | 15:38 |
| sigmavirus | As in, users generally know what attributes are on a host/cloud/region/etc. | 15:38 |
| jimbaker | except in an intermediate form when we were first trying out client vars | 15:38 |
| sigmavirus | There's likely not a good way to know that for variables | 15:38 |
| jimbaker | sigmavirus, correct | 15:38 |
| sigmavirus | I think namespaces are something that we don't have a concrete use-case for (unless I'm missing the UC) | 15:38 |
| thomasem | Hence the reason we had to go the route we did, no? | 15:38 |
| thomasem | Wrt the MutableMapping | 15:38 |
| jimbaker | namespaces do have concrete use cases | 15:39 |
| sigmavirus | And reserved keys could absolutely be implemented regardless of whether variables are valid python identifiers | 15:39 |
| jimbaker | https://blueprints.launchpad.net/craton/+spec/craton-namespaces | 15:39 |
| sigmavirus | jimbaker: I'm not saying there's no use case, I'm saying in the list of dusty's use-cases, I've not seen it | 15:39 |
| sigmavirus | jimbaker: variables can presently be nested | 15:39 |
| jimbaker | sigmavirus, right, dusty's stuff is not likely to go into that detail | 15:40 |
| sigmavirus | does that not solve the namespace issue? | 15:40 |
| jimbaker | sigmavirus, no - namespaces are for us to implement custom functionality like secrets or maintenance | 15:40 |
| sigmavirus | as in things that we don't return to the user? | 15:40 |
| jimbaker | it's not about nesting | 15:40 |
| jimbaker | sigmavirus, they would be returned to the user | 15:41 |
| sigmavirus | so we'd migrate vars to be `user/foo` | 15:41 |
| sigmavirus | or have an implicit `user/` namespace? | 15:41 |
| jimbaker | sigmavirus, that's the implicit idea | 15:41 |
| jimbaker | but nothing has been worked out | 15:41 |
| jimbaker | so malleable clay | 15:41 |
| sigmavirus | Yeah, I think namespacing needs a good specification. I'm still not seeing how namespacing requires restriction to valid python identifiers | 15:41 |
| jimbaker | per blueprint, it doesn't make that claim either | 15:42 |
| jimbaker | at most it says reserve / | 15:42 |
| jimbaker | and give it special meaning | 15:42 |
| jimbaker | sigmavirus, btw - is tox against python-cratonclient supposed to run on OSX? | 15:44 |
| jimbaker | i would assume that should be possible, even if tox -e functional doesn't run on OSX | 15:45 |
| thomasem | FYI: http://eavesdrop.openstack.org/irclogs/%23craton/%23craton.2017-03-09.log.html#t2017-03-09T19:34:07 | 15:45 |
| jimbaker | for craton itself | 15:45 |
| sigmavirus | jimbaker: it runs on OSX for me | 15:45 |
| thomasem | This is where we were discussing the JSON path stuff, initially | 15:45 |
| thomasem | for context | 15:45 |
| jimbaker | sigmavirus, ok, just trying to get it to run. not certain what i'm missing | 15:46 |
| sigmavirus | jimbaker: what errors are you seeing? | 15:46 |
| sigmavirus | jimbaker: hop into Craton-Recordable quickly and screen-share? | 15:46 |
| jimbaker | sigmavirus, i will give you a paste | 15:46 |
| jimbaker | sure, we can do that too | 15:46 |
| sigmavirus | I'm in there, but whatever you'd prefer works for me | 15:47 |
| jimbaker | thomasem, so at this point, i think we can avoid the identitfiers must be python, since we haven't found a good reason for them to be | 15:47 |
| jimbaker | sigmavirus, one moment | 15:47 |
| jimbaker | clearly i don't hop as fast as you ... | 15:48 |
| thomasem | jimbaker: that sounds good to me. | 15:48 |
| sigmavirus | I attribute my hop speed to having to hop as fast as twin 4yr olds and a 2yr old | 15:49 |
| jimbaker | at best, restricting to identifiers should not use / would probably be fine for users, just so we can use in the future as desired - and i think it only impacts them if they use a registered namespace | 15:49 |
| jimbaker | which actually gets convention first, then more functionality later | 15:50 |
| sigmavirus | jimbaker: fair, but "valid python identifiers" is far more restrictive than not using / | 15:50 |
| jimbaker | ie you can use ssh/ now, but it's not a real secret store. in the future, it becomes one. bonus for your upgrade | 15:50 |
| sigmavirus | thomasem: have you run the cratonclient tests on osx? | 15:51 |
| jimbaker | sigmavirus, so i think we have already concluded we cannot come up with a good reason to "say valid python identifiers" | 15:51 |
| sigmavirus | yeah, so the arbitrary restrictions we imply should be enumerated better than what I expect was the intent when that phrase was first bandied about | 15:52 |
| jimbaker | other than a natural, maybe we want to reserve something. this gets into the sort of space of course: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#Reserved_keywords_as_of_ECMAScript_2015 | 15:53 |
| sigmavirus | jimbaker: right, Craton can have a list of reserved symbols/keywords/etc. | 15:53 |
| sigmavirus | As long as they're well documented and published somewhere | 15:53 |
| jimbaker | sigmavirus, sure. an obvious one: "craton/" | 15:54 |
| sigmavirus | Ostensibly a 400 resulting in finding one of those would also inform the user of their honest mistake to make it easier to debug | 15:54 |
| jimbaker | so anything with that prefix is reserved | 15:54 |
| sigmavirus | jimbaker: so what's the error you're seeing with tox in python-cratonclient? | 15:55 |
| jimbaker | sigmavirus, re-running | 15:55 |
| jimbaker | i can share my screen now | 15:55 |
| jimbaker | let me hop in that vidyo | 15:55 |
| thomasem | sigmavirus: I have not. | 15:59 |
| thomasem | #startmeeting craton | 16:00 |
| thomasem | #chair sigmavirus sulo jimbaker thomasem | 16:00 |
| thomasem | #link https://etherpad.openstack.org/p/craton-meetings | 16:00 |
| openstack | Meeting started Thu Mar 23 16:00:09 2017 UTC and is due to finish in 60 minutes. The chair is thomasem. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| thomasem | #topic Roll Call | 16:00 |
| openstack | The meeting name has been set to 'craton' | 16:00 |
| openstack | Current chairs: jimbaker sigmavirus sulo thomasem | 16:00 |
| thomasem | o/ | 16:00 |
| anonymike | o/ | 16:00 |
| thomasem | jimbaker: sigmavirus:? | 16:02 |
| sigmavirus | o/ | 16:02 |
| thomasem | git-harry: ? | 16:02 |
| jimbaker | o/ | 16:02 |
| fsaad | o/ | 16:02 |
| git-harry | hi | 16:03 |
| thomasem | tojuvone: ? | 16:03 |
| tojuvone | hi | 16:03 |
| thomasem | #topic Stand Up | 16:03 |
| thomasem | #info each team member briefly describes what they are working on this week, and describes blockers (if there are any) | 16:03 |
| thomasem | #topic Stand Up :: thomasem | 16:03 |
| thomasem | Working on JSON Path some more. Really far along, I'm keeping it up-to-date here: https://review.openstack.org/#/c/443941/. I see git-harry reviews and appreciate his comments (and agree, I was on the fence anyway about the JSON approximation) | 16:04 |
| thomasem | Adding handling for invalid JSON path expressions so we're not returning 500s over it and finishing up testing. Looking into the bug that git-harry found in that patch, too. | 16:05 |
| thomasem | done | 16:06 |
| thomasem | #topic Stand Up :: anonymike | 16:06 |
| anonymike | Working on setting up my environment and getting familiar with the code. I've submitted a few updates to the installation docs and I'm currently working on touching up the "wrapper functions" script to quick launch craton and easily hit the api. | 16:06 |
| anonymike | done | 16:06 |
| thomasem | #topic Stand Up :: sigmavirus | 16:06 |
| sigmavirus | Wrapping up cratonclient docs/betamax work for tomorrow (which is my last day) | 16:07 |
| sigmavirus | Trying to do reviews for others | 16:07 |
| sigmavirus | </fin> | 16:07 |
| fsaad | :'( | 16:07 |
| jimbaker | we will miss sigmavirus | 16:07 |
| sigmavirus | nah, I'm a pain in the toe | 16:07 |
| thomasem | indeed | 16:08 |
| anonymike | :( | 16:08 |
| sigmavirus | thanks for taking my side thomasem | 16:08 |
| sigmavirus | =P | 16:08 |
| jimbaker | minor pain, still can run with that pain in toe | 16:08 |
| thomasem | sigmavirus: see, it could be an answer to both. | 16:08 |
| thomasem | :P | 16:08 |
| thomasem | #topic Stand Up :: jimbaker | 16:08 |
| antonym | :/ | 16:08 |
| sigmavirus | clever | 16:08 |
| jimbaker | reviews, trying to understand betamax | 16:08 |
| jimbaker | (and why it's failing for me - sigmavirus and i were just going through a debug on that) | 16:09 |
| jimbaker | need to finish up alembic | 16:09 |
| jimbaker | done | 16:09 |
| thomasem | #topic Stand Up :: fsaad | 16:09 |
| fsaad | 'grats on yesterday's demo I think it went well regardless of what it seemed like :) | 16:10 |
| antonym | +1 | 16:10 |
| thomasem | :) Thank you! | 16:10 |
| fsaad | I'll be poking support for that subset of variables that would be useful to have in the host show | 16:10 |
| git-harry | fsaad: I didn't attend that, are there any action items from it? | 16:11 |
| fsaad | and also procuring hardware for the craton server today | 16:11 |
| fsaad | done | 16:11 |
| fsaad | git-harry: I think so, we can chat after standup! | 16:11 |
| thomasem | #topic Stand Up :: git-harry | 16:11 |
| git-harry | Mostly review related stuff. | 16:12 |
| git-harry | done | 16:12 |
| thomasem | #topic Stand Up :: tojuvone | 16:12 |
| tojuvone | OPNFV summit CFP and Nova side of the planned host maintenance. | 16:12 |
| tojuvone | done | 16:12 |
| thomasem | #topic Stand Up :: antonym | 16:12 |
| antonym | did some doc work and working on getting my provisioning env with craton set up in physical server lab (has graduated from running on fusion), also thought about putting together some docs with ansible use cases like using the uri module with craton for people to have some examples to start from | 16:13 |
| antonym | done | 16:13 |
| thomasem | +1! | 16:14 |
| thomasem | #topic git-harrry's question regarding action items from demo yesterday | 16:14 |
| thomasem | Wow, Harry, you got an extra "r" on the house. | 16:14 |
| jimbaker | blame autorepeat setting... | 16:15 |
| sigmavirus | I bet harry thought to himself, "arrrrrr matey" | 16:15 |
| thomasem | I have mine set pretty high. | 16:15 |
| thomasem | or low... as the case may be. | 16:15 |
| jimbaker | just giving you cover that's all | 16:15 |
| thomasem | Anyway, the question was "11:11 git-harry: fsaad: I didn't attend that, are there any action items from it?" | 16:15 |
| thomasem | Thanks jimbaker! :P | 16:15 |
| jimbaker | so two items | 16:16 |
| jimbaker | 1. get the exact data loaded that support requested | 16:16 |
| jimbaker | 2. be able to show a subset of variables | 16:16 |
| jimbaker | it also prompted some side conversations i had with thomasem re possibility of adding elasticsearch support to craton earlier than later | 16:17 |
| git-harry | I thought we'd already sourced the right data, what was wrong with it this time? | 16:18 |
| fsaad | I'd say we focus on #2 | 16:18 |
| fsaad | can work with Tim on #1 | 16:18 |
| thomasem | git-harry: it was missing, like, package information, iirc. | 16:18 |
| jimbaker | right, the craton's team focus is on #2 | 16:18 |
| fsaad | but if one of you feel like working on data loading while he's out, his script is publicly accessible too | 16:18 |
| fsaad | yeah | 16:18 |
| jimbaker | there were are also some other questions raised by tim - such as how to work with large result sets in the client | 16:19 |
| sigmavirus | I'm glad we're at the point where we recognize this isn't craton's problem per-say | 16:19 |
| jimbaker | basically he was using --limit, but not --marker | 16:19 |
| thomasem | sigmavirus: which part? | 16:19 |
| thomasem | jimbaker: I think he was expecting it to just paginate automatically to the specified limit. | 16:20 |
| jimbaker | thomasem, exactly | 16:20 |
| sigmavirus | thomasem: the part of loading in package info | 16:20 |
| jimbaker | which the client could readily do if we wanted to have that support | 16:20 |
| sigmavirus | cratonclient's shell can autopaginate if we want | 16:20 |
| jimbaker | given underlying generator | 16:20 |
| sigmavirus | that's like a 1 line change | 16:20 |
| jimbaker | yeah, sounds about right | 16:21 |
| sigmavirus | or maybe 5 (one for each -list command, idk) | 16:21 |
| antonym | #link https://github.com/pwnall1337/ops-generic/tree/master/osops_common | 16:21 |
| thomasem | He also ran into some bug with --limit when it was kind of low? I don't know what the boundaries there are, but he was getting 400s all of a sudden. | 16:21 |
| thomasem | I don't think he filed a bug about it yet, though? | 16:21 |
| jimbaker | no bug | 16:21 |
| thomasem | Like --limit 2 was giving 400s? | 16:22 |
| jimbaker | right, that's a real bug | 16:22 |
| jimbaker | given --limit passes straight through | 16:22 |
| git-harry | No, I think the schema enforces a minimum | 16:22 |
| thomasem | Yeah, it was weird. | 16:22 |
| jimbaker | ahh, good point git-harry. still might be worth having limit have a min of say 1 | 16:23 |
| git-harry | Yeah, minimum is 10 | 16:23 |
| thomasem | Then we need better messaging around that, or to remove the minimum... why have a minimum? | 16:23 |
| thomasem | Ahhhh | 16:23 |
| jimbaker | that's a useful limit | 16:23 |
| git-harry | The client doesn't expose error messages | 16:23 |
| thomasem | min of 1 is, yes. | 16:23 |
| jimbaker | i just want ONE item | 16:23 |
| thomasem | min of 10 I'm not sure I understand. | 16:23 |
| jimbaker | so two bugs here, plus a feature change | 16:23 |
| jimbaker | or something like that | 16:23 |
| thomasem | lol | 16:24 |
| jimbaker | 1. client should report errors precisely from api server; 2. limit of 1 is fine; 3. limit of anything is also fine, and should use underlying autopaginate | 16:24 |
| jimbaker | so that will fix up some usability issues | 16:26 |
| thomasem | For the issue with <resource>-show: https://bugs.launchpad.net/python-cratonclient/+bug/1675410 and I'll be creating another (or adding to this one) regarding affording for specifying variables to extract, and possibly RegEx support later on. | 16:26 |
| openstack | Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New] | 16:26 |
| jimbaker | thomasem, +1, this is the big ask from support | 16:27 |
| jimbaker | and easy enough to support | 16:27 |
| thomasem | Alright. Any other questions on that topic? | 16:28 |
| jimbaker | i should mention that one more thing that came up related to variables is that loading from ansible is not always going to produce highly useful things to search against | 16:28 |
| thomasem | or comments/concerns | 16:28 |
| jimbaker | i forget the exact one, but this was with respect to a list of processor strings. or was it even a list - could have been one big string | 16:28 |
| thomasem | And, not sure about what the environment looked like, but the search was pretty slow over a really small amount of data with the nested var search. | 16:28 |
| git-harry | jimbaker: that sounds like a user problem | 16:29 |
| thomasem | Would be good to figure out JSON column indexing and how that might help us. Again, this is where I think Elasticsearch would outshine our nested var searching. | 16:29 |
| jimbaker | thomasem, yeah, because we have no indexing on that json stuff. it's even worse because the alembic change i have under review did add a useful index for k/v search | 16:29 |
| jimbaker | to at least make that part work better | 16:29 |
| jimbaker | so must get alembic change in, even subsetting with respect to keys would be far better | 16:30 |
| thomasem | I still think at some point we're going to have to set some defined boundaries on what this nested stuff can do and say that for everything else, we need to use ES. | 16:30 |
| thomasem | Gotcha | 16:30 |
| jimbaker | thomasem, agreed about elasticsearch | 16:30 |
| jimbaker | i think we could readily do the following for craton | 16:30 |
| jimbaker | elasticsearch + kibana | 16:31 |
| jimbaker | as a dashboard | 16:31 |
| jimbaker | just requires a small amount of work to aggregate docs into elasticsearch | 16:31 |
| jimbaker | there are some issues here, such as how do we integrate our security model with elasticsearch | 16:32 |
| jimbaker | but it would demo well before we get to that point | 16:32 |
| anonymike | +1 | 16:32 |
| jimbaker | but it would be a fairly simple matter of programming to iterate over resources; dump into elasticsearch with the python interface it supports (use bulk mode) | 16:33 |
| jimbaker | i bet anonymike could get this going in an afternoon... | 16:33 |
| jimbaker | :) | 16:33 |
| anonymike | ;) | 16:33 |
| thomasem | Yeah, we can address the "how" more when we get someone dedicated to getting that running and documenting it. | 16:33 |
| jimbaker | i suggest we just do a little skunkworks to try out something | 16:34 |
| thomasem | Okay | 16:34 |
| jimbaker | then we can go back & figure out proper integration | 16:34 |
| jimbaker | given we know that elasticsearch has everything we need for security integration, plus keeping stuff live via its replication model | 16:34 |
| thomasem | jimbaker: I believe git-harry mentioned that the variable bloat from loading useless things from Ansible was a user concern. | 16:35 |
| jimbaker | thomasem, but that's more of a question of filtering i think | 16:35 |
| jimbaker | nothing i have seen is even close to something that would be expensive to manage in a database, assuming proper indexing | 16:35 |
| jimbaker | but it's super painful right now from the CLI | 16:36 |
| thomasem | The comment was specifically about loading variables from Ansible, though. "related to variables is that loading from ansible is not always going to produce highly useful things to search against" | 16:36 |
| jimbaker | it only takes a few kilobytes to be too much on the CLI. but megabytes... mysql has no problem | 16:36 |
| jimbaker | thomasem, agreed. lots of chaff | 16:37 |
| thomasem | Which I believe is where git-harry's comment came from. Basically: Cool, don't load variables you don't care about. | 16:37 |
| sigmavirus | got distracted, fwiw, the pagination minimum amount was agreed upon in the specification | 16:37 |
| jimbaker | right. and at least know what you want to filter on/search for/and be reasonable about you organize | 16:37 |
| thomasem | sigmavirus: interesting. I'll have to go look and see the rationale. | 16:37 |
| sigmavirus | so y'all need to update the spec in addition to updating the code =) | 16:37 |
| jimbaker | thomasem, and it could be more of the CLI should just the smart thing | 16:38 |
| sigmavirus | jimbaker: showing better messages from the CLI is absolutely a good goal too =) | 16:38 |
| jimbaker | you want --limit=1 - fine, i will just give you the first item | 16:38 |
| sigmavirus | But yeah, I think ES is also necessary for everything we're talking about | 16:38 |
| sigmavirus | also, I wanted to do the limitations on 'limit' in a pre-processing function instead of the schema but was directed to use the schema | 16:39 |
| sigmavirus | doing it in code would have allowed us to just return 10 instead of returning a 400 | 16:39 |
| jimbaker | yep, so anonymike, sound like a fun task for you? seems like right up your alley | 16:39 |
| jimbaker | in terms of elasticsearch | 16:39 |
| anonymike | jimbaker: elasticsearch? | 16:40 |
| anonymike | yezzir | 16:40 |
| thomasem | Excellent! | 16:40 |
| jimbaker | awesome | 16:40 |
| sigmavirus | anonymike++ | 16:40 |
| jimbaker | new guy, gets to work on fun stuff. what can be better? :) | 16:40 |
| anonymike | :D | 16:40 |
| jimbaker | seriously, it's pretty good, because we need more eyeballs on that python client | 16:40 |
| jimbaker | and this is a good place to try it out | 16:41 |
| jimbaker | and if by chance we have something to demo next week, well i would never complain about that... | 16:41 |
| thomasem | Okay, cool. So, we have some bugs to write, it sounds like. | 16:42 |
| anonymike | I better get started :) haha | 16:42 |
| thomasem | And BPs | 16:42 |
| * sigmavirus nods | 16:42 | |
| jimbaker | yep | 16:42 |
| sigmavirus | might be worth updating the spec template to include a place to log updates | 16:42 |
| jimbaker | except for you anonymike - you can just figure stuff out, and report back to us | 16:42 |
| thomasem | some amendments section | 16:43 |
| sigmavirus | (since it sounds like you'll be updating the pagination-spec and something like that might be useful to track the documents changes for people without their hands on the git repo) | 16:43 |
| sigmavirus | thomasem: exactly | 16:43 |
| anonymike | jimbaker: sure, sounds good | 16:43 |
| thomasem | jimbaker: you want to take the BP for ES/Kibana and feature change for "limit", and I'll take writing up the client bugs? | 16:45 |
| jimbaker | anonymike, at some point, we will also want to look here - https://github.com/noplay/python-mysql-replication | 16:45 |
| jimbaker | thomasem, no BP yet for ES/kibaba | 16:45 |
| thomasem | kk | 16:45 |
| jimbaker | we are going to skunkworks this first, just to get an understanding of what's there | 16:46 |
| thomasem | Lol, okay. It's either/or, to me. I just wanted something to communicate what anonymike is working on, is all, and a place to put notes and such for what's found. | 16:46 |
| thomasem | Anyway, however we wish to run this deal. | 16:47 |
| anonymike | jimbaker: something like this? https://github.com/scharron/elasticsearch-river-mysql | 16:47 |
| jimbaker | anonymike, it is, but rivers are now deprecated in ES | 16:47 |
| anonymike | ha yeah, last update 5 yrs | 16:48 |
| jimbaker | in general, we have to do this at a higher level | 16:48 |
| jimbaker | the notification can say - the underlying doc needs to be updated | 16:48 |
| jimbaker | but it has to understand the model of the system | 16:48 |
| jimbaker | vs just raw table stuff | 16:48 |
| jimbaker | so i think this starts getting more into blueprint level considerations - and why pushing this aspect out a bit | 16:49 |
| thomasem | Alright, we're coming up on 10 minutes left. Any other topics folks want to bring up before we run out of time? | 16:49 |
| jimbaker | but a huge advantage of ES/kibana is we see changes over time | 16:50 |
| jimbaker | thomasem, i'm good | 16:50 |
| thomasem | #topic Open Discussion | 16:50 |
| thomasem | Alright. Sounds like we've got a plan and a lot of good ideas came out of the demo. | 16:51 |
| jimbaker | yep | 16:51 |
| thomasem | Priorities are modifying the loading script and affording for the client to configure `--fields` with some variable selection, too? | 16:52 |
| thomasem | Once existing priorities are sorted, of course, being the Alembic stuff, nested var search, and... | 16:52 |
| fsaad | +1 | 16:53 |
| thomasem | I think that's it that's remaining? | 16:53 |
| thomasem | From Monday's meeting. | 16:53 |
| thomasem | So let's finish it up. | 16:53 |
| thomasem | And move on to the priorities from our demo | 16:53 |
| fsaad | thomasem / all: so does https://bugs.launchpad.net/python-cratonclient/+bug/1675410 mean we don't need to wait for support to provide a list of default fields to show in host-list ? | 16:53 |
| openstack | Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New] | 16:53 |
| jimbaker | right such as fix client up with respect to --limit=1 to whatever - no change needed in rest api | 16:53 |
| fsaad | or still would be nice to have ? | 16:53 |
| jimbaker | fsaad, i think we can skip that for the moment | 16:54 |
| thomasem | fsaad: it'd be nice to have, but I'm hoping it'll make it so they can get whatever variables they care about themselves. | 16:54 |
| thomasem | whatever fields/variables | 16:54 |
| jimbaker | thomasem and i were discussing that this gets into a more general conf system | 16:54 |
| jimbaker | for client expectations | 16:54 |
| jimbaker | otherwise it feels sort of ad hoc | 16:55 |
| thomasem | Imagine how bad --details will look in a host-list, though? | 16:55 |
| thomasem | assuming that causes it to include the variables... | 16:56 |
| jimbaker | thomasem, indeed, megabytes! | 16:56 |
| jimbaker | looks great with --format=json | 16:56 |
| jimbaker | however | 16:56 |
| thomasem | Haha, right | 16:56 |
| thomasem | tabled, though... ouch | 16:56 |
| jimbaker | there is no good answer here. but again i think the idea you proposed of putting stuff in columns by --field is a great one | 16:56 |
| thomasem | cool | 16:57 |
| jimbaker | (does it imply details btw) | 16:57 |
| thomasem | I suppose... but isn't all that --details does is include variables? | 16:57 |
| jimbaker | for now | 16:57 |
| thomasem | Gotcha | 16:57 |
| thomasem | Okay | 16:57 |
| jimbaker | it should also include labels etc | 16:57 |
| thomasem | Ahhh right | 16:57 |
| thomasem | okay | 16:57 |
| jimbaker | the underlying thing is ?details=all - with the implication that it could be mean less | 16:58 |
| thomasem | Wells, I'll finish fleshing out the bug(s) and we can discuss from there | 16:58 |
| jimbaker | or some interesting subset | 16:58 |
| jimbaker | what are my role assignments? etc etc | 16:58 |
| thomasem | --fields id, name, roles, var:host_facts.os | 16:58 |
| jimbaker | children... | 16:58 |
| jimbaker | lots of possible stuff to show | 16:58 |
| jimbaker | anyway, probably wrap up. not going to do all that today | 16:59 |
| thomasem | Alright, coming up on close of meeting. | 16:59 |
| thomasem | 3... | 16:59 |
| thomasem | 2... | 16:59 |
| thomasem | 1... | 16:59 |
| thomasem | #endmeeting | 16:59 |
| openstack | Meeting ended Thu Mar 23 16:59:37 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-23-16.00.html | 16:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-23-16.00.txt | 16:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-23-16.00.log.html | 16:59 |
| jimbaker | thanks! | 16:59 |
| thomasem | Cheers! | 16:59 |
| jimbaker | sigmavirus, https://gist.github.com/jimbaker/8b1b6ccbef5368adaf84ff6853833185 | 17:05 |
| jimbaker | so both outputs, from the branch i pulled as well with git review -d Ic4e342cf01be288b7a7c453659580c57169b1022 (looks like identical pulls, but no gerrit expert am i) | 17:06 |
| jimbaker | thomasem, iirc, you have no problem with testing python client on OSX right? | 17:07 |
| jimbaker | (also all CRATON_DEMO_* envvars are not set - so purely going against recorded cassettes) | 17:07 |
| thomasem | jimbaker: I haven't tried. I set up a remote dev box because of all of the problems I ran into running functional tests for craton on OS X. | 17:07 |
| jimbaker | thomasem, ahh, that makes sense | 17:07 |
| jimbaker | yes, it works fine there. well, other than some minor problem with netiface or something like that not building on python 3.5, but that's a completely separate issue | 17:08 |
| jimbaker | i did notice in our tox stuff on python-client however that we only test against 2.7, 3.4, as well as pypy. we should do this for 3.5 at the very least, and presumably 3.6. maybe drop 3.4 | 17:09 |
| thomasem | jimbaker: tox -e py35 always worked for me? | 17:10 |
| jimbaker | netiface also doesn't build on py3.4 for me | 17:10 |
| jimbaker | thomasem, i will give that specific invocation a try on ubuntu | 17:10 |
| jimbaker | however, i would like betamax to run on osx, given that we should work there for client code | 17:10 |
| jimbaker | but i believe for sigmavirus, it works for him. just need to get it to work for at least another person or two | 17:11 |
| jimbaker | git-harry, could you respond to http://lists.openstack.org/pipermail/openstack-dev/2017-March/114425.html ? want to wrap up voting here | 17:12 |
| sigmavirus | jimbaker: 3/4 active cores would generally be good enough for any other project =P | 17:13 |
| jimbaker | sigmavirus, yeah | 17:13 |
| jimbaker | but 4/4 sounds even better to me | 17:13 |
| jimbaker | i will just call the election over in the next hour or two - it's possible git-harry has quite reasonably gone home for the day | 17:14 |
| sigmavirus | if git-harry hasn't, I'll be disappointed | 17:14 |
| sigmavirus | also well within his rights to abstain | 17:14 |
| jimbaker | sigmavirus, good point | 17:14 |
| sigmavirus | also well within his rights to not want to participate on the ml =P | 17:15 |
| openstackgerrit | Michael Porras proposed openstack/craton master: Adding wrapper functions to tools https://review.openstack.org/449230 | 17:18 |
| anonymike | jimbaker: ^ my proposed solution. I can also add something for user defined config locations. But this allows me to swap back and forth between direct and docker with ease | 17:22 |
| jimbaker | anonymike, looking forward to trying out your proposed improvements! | 17:23 |
| * anonymike jumps into elasticsearch | 17:23 | |
| thomasem | lunch | 17:27 |
| sigmavirus | anonymike: did you secure a rope for easy escape? | 17:29 |
| sigmavirus | d34dh0r53: still hasn't escaped the great elasticsearch openjdk leap of 2014 | 17:29 |
| anonymike | sigmavirus: I did not! | 17:29 |
| sigmavirus | RIP anonymike | 17:29 |
| d34dh0r53 | lol, and it just sucked me back in | 17:29 |
| d34dh0r53 | s/ELK/MegaMaid/g | 17:29 |
| sigmavirus | d34dh0r53: I weep for your family | 17:30 |
| tojuvone | jimbaker, should we finish CFP on monday. I'll have time till tuesday morning (my time) to upload. | 17:30 |
| git-harry | jimbaker: done | 17:32 |
| jimbaker | tojuvone, i'm going to spend some time on that today | 17:33 |
| jimbaker | also try out the lab access you gave me | 17:33 |
| jimbaker | git-harry, thanks! | 17:33 |
| sulo | back, o/ | 17:34 |
| sulo | so just reading the scrollback, i wasnt clear what the outcome of the demo was fsaad, what are the action items that we agreed to work on from that ? i see 3 things listed up there | 17:34 |
| sulo | but wasnt clear what was requested as work item | 17:35 |
| jimbaker | sulo, we need to "flip the bit" for thomasem, we now have 4 out of 4 +1s for him to become core! (counting active cores) | 17:35 |
| sulo | jimbaker: awesome | 17:35 |
| jimbaker | now need to read up on docs to actually do this flipping... | 17:36 |
| sulo | jimbaker: i can add him, sec | 17:36 |
| jimbaker | sulo, even better! | 17:36 |
| sulo | done | 17:36 |
| jimbaker | :) | 17:36 |
| sulo | thomasem: ^ | 17:36 |
| jimbaker | thomasem, use your powers for good, and not for evil | 17:37 |
| anonymike | \o/ | 17:38 |
| fsaad | bjoern updated https://bugs.launchpad.net/python-cratonclient/+bug/1675410 btw | 17:38 |
| openstack | Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New] | 17:38 |
| jimbaker | sulo, getting back to your question - i think this hopefully will be clarified in the bug reporting | 17:39 |
| sulo | jimbaker: ah thats even better, its good that folks are creating bug reports as request | 17:39 |
| fsaad | sulo: the demo went really well, the outcome was a few items of feedback to work on, one was host-show (*-show?) being too polluted when many variables exist | 17:39 |
| jimbaker | also i asked anonymike to look into figuring out what it means to do an initial elasticsearch integration - pure prototype, let's see what the issues are | 17:40 |
| fsaad | action items for client load which tim is looking at | 17:40 |
| jimbaker | then we will circle back and figure out this more formally, including blueprints | 17:40 |
| tojuvone | jimbaker, ok | 17:40 |
| fsaad | also for the first, bjoern suggested limiting the output of host-show variables to a subset of all, but believe consensus is to allow flexibility on fields to show by making configurable | 17:40 |
| sulo | fsaad: cool, jimbaker: what are we trying to get out of elasticsearch integration ? | 17:41 |
| sulo | i mean user stories wise | 17:41 |
| jimbaker | better search, chance of using kibana as a dashboard onto craton | 17:41 |
| fsaad | sulo: also, the search was noticeable slow while demoing, especially to only have had one customer | 17:41 |
| fsaad | so looking into that I believe prompted the elasticsearch conversation | 17:41 |
| jimbaker | so such things as approx search | 17:42 |
| sulo | fsaad: heh, thats expected | 17:42 |
| fsaad | granted we don't know what the demo box specs were, so a lot may improve just by tuning mysql, it was out of the box install conf I know that | 17:42 |
| jimbaker | we can solve the speed issue with indexing | 17:42 |
| fsaad | cool I imagined that was the case, just not familiar with json fields in mysql | 17:42 |
| sulo | fsaad: iirc we dont index properly right now | 17:42 |
| jimbaker | i think nested var search as implemented is fine for automation (assuming we do obvious speedups) | 17:42 |
| fsaad | sounds like low hanging fruit | 17:42 |
| sulo | yeah, some of it will be solved by index | 17:42 |
| jimbaker | but elasticsearch gives us a better support experience | 17:43 |
| jimbaker | so both are good | 17:43 |
| anonymike | people love dashboards | 17:44 |
| sulo | so elasticsearch just for variables ? | 17:44 |
| jimbaker | sulo, everything | 17:44 |
| sulo | hmm .. whats stopping anyone from just doing that now ? | 17:44 |
| jimbaker | but we will see what anonymike can do in terms of feeding elasticsearch in such a way that it's useful | 17:44 |
| sulo | es has good api and search support | 17:44 |
| jimbaker | sulo, i think it's very straightforward | 17:44 |
| jimbaker | we have the resource listing on the craton client side; and ES has a good python client, including bulk loading | 17:45 |
| jimbaker | so it's a pretty small investment of our time to try it out and see what the actual issues are. i'm mostly thinking schema type considerations | 17:46 |
| jimbaker | there are other aspects such as integrating with our rbac; plus driving changes into ES; but let's postpone those for the moment | 17:46 |
| sulo | jimbaker: i am not doubting it will work, or that is pretty simple, i am just saying whats stopping someone who is deploying craton with es to just es for everything - craton | 17:47 |
| jimbaker | sulo, because i think this is a useful integration for us to support | 17:47 |
| jimbaker | as anonymike said, everyone loves a dashboard. and better search would get over the last hump in terms of usability | 17:48 |
| jimbaker | but we can think about this more once we look at what is readily possible to look at. given anonymike's previous work on galaxy, etc, i think he has a good sense of what could be showed | 17:49 |
| jimbaker | especially with respect to defining kibana views | 17:49 |
| sulo | jimbaker: well kibana/grafana is good, but i dont knwo that it would be a great UI for craton .. although, i've seen that get used in public cloud as that | 17:49 |
| jimbaker | sulo, yeah, hence prototype | 17:49 |
| sulo | i personally think, ppl like alerts.othree type dash thats infra specific | 17:49 |
| jimbaker | i'm pretty sure we can get something similar to that in kibaba | 17:50 |
| sulo | so this has become a priority item now ? | 17:50 |
| sulo | or is that becasue of search? | 17:50 |
| jimbaker | search is the driver of the ES discussion | 17:50 |
| sulo | gotcha | 17:50 |
| jimbaker | but i look at it as something that can get anonymike some experience on the python client | 17:51 |
| jimbaker | so easy to justify | 17:51 |
| sulo | sure, i mean do we know what was slow, why, if its fixable on the db side first .. or the decision has been made that its going to be elasticsearch ? | 17:52 |
| jimbaker | sulo, there were certain queries that were not readily expressible using json path | 17:53 |
| jimbaker | that were being raised | 17:53 |
| jimbaker | basically regex type queries. but also more generally approx queries. we can address by saying - load your data better | 17:54 |
| jimbaker | and that's a reasonable answer to a certain extent | 17:54 |
| jimbaker | but at some point we don't want to keep investing a lot of time on making search more powerful. we are not going to put the years of engineering into search that has been done for ES | 17:55 |
| thomasem | sulo: Ultimately, ES is designed for the sort of searching people want to do. I think we'll be wasting our time if we start chasing all of the ways they may want to search the data, when there's already a perfectly good solution in ES. | 17:55 |
| sulo | maybe we should just switch db to something that supports all this .. i mean i like ES .. i like the integration idea too .. but i dont know if this is a good move .. as a systems person .. if you ask me to maintain a mysql cluster + es cluster to run this i will frown | 17:55 |
| anonymike | I agree we need to tweak the db and better explain to users how to query, but i think a few days of work to have a polished ui to demo is important | 17:55 |
| sulo | sure both mysql and es are widely used etc | 17:55 |
| jimbaker | ES is not a config database | 17:55 |
| jimbaker | each has advantages | 17:55 |
| jimbaker | i mean it would be interesting to put craton on crate.io - https://crate.io/ | 17:56 |
| jimbaker | just for the pure fun of that | 17:56 |
| anonymike | whoa | 17:56 |
| jimbaker | but crate.io is eventually consistent | 17:56 |
| jimbaker | and all of my instincts related to config databases are, this is a bad idea | 17:57 |
| sulo | thomasem: jimbaker: its easy to make it so, so like i was saying myabe we need to explore casandra, mongodb etc ? | 17:57 |
| jimbaker | i don't think mongodb or cassandra are good ideas for what we are doing | 17:58 |
| anonymike | I <3 mongo but that'd be quite the refactor no? | 17:58 |
| jimbaker | completely awesome for other use cases | 17:58 |
| jimbaker | fwiw, here's what i'm hoping we can use long term | 17:58 |
| jimbaker | https://github.com/pingcap/tidb | 17:59 |
| jimbaker | sort of like cockroachdb, but with mysql interface | 18:00 |
| * anonymike has not heard of half these dbs | 18:00 | |
| anonymike | tidb looks cool | 18:01 |
| jimbaker | although cockroachdb apparently is somewhat similar to postgresql - https://www.cockroachlabs.com/blog/building-application-cockroachdb-sqlalchemy-2/ | 18:01 |
| jimbaker | so some possibility there | 18:01 |
| thomasem | I can't say I know enough about these to express an opinion either way yet. | 18:01 |
| jimbaker | tidb is written in a combo of go and rust | 18:01 |
| jimbaker | rust for the distributed kv store | 18:02 |
| jimbaker | go for the mysql frontend | 18:02 |
| jimbaker | thomasem, all i know is that we have some choices for strongly consistent, scaled out SQL databases | 18:02 |
| jimbaker | multiple implementations of the F1 paper from google | 18:02 |
| jimbaker | https://research.google.com/pubs/pub41344.html | 18:03 |
| thomasem | What's the beef with running ES, though? Is there really that much of an operational burden? | 18:03 |
| thomasem | Could be supplementary, but not required. | 18:03 |
| jimbaker | thomasem, agreed | 18:04 |
| anonymike | +1 | 18:04 |
| thomasem | For environments where folks want that sort of searchability | 18:04 |
| anonymike | and visualization | 18:04 |
| thomasem | Yeah | 18:04 |
| jimbaker | it's exactly the sort of thing that's easy to put together with docker compose | 18:04 |
| jimbaker | anonymike, if you have a few extra minutes, feel free to do that too. so we can all play | 18:05 |
| jimbaker | ;) | 18:05 |
| sulo | thomasem: jimbaker: ah so it optional ? | 18:05 |
| jimbaker | sulo, absolutely optional | 18:05 |
| sulo | thomasem: yes it is opertional burden , by adding es we are asking folks to run es cluster | 18:05 |
| jimbaker | i want to address a specific support question | 18:05 |
| thomasem | What's involved in running an ES cluster? | 18:05 |
| sulo | thomasem: same as running mysql cluster | 18:05 |
| sulo | you need to maintain it .. nothing wrong with it | 18:06 |
| sulo | but it needs to be maintained | 18:06 |
| anonymike | jimbaker: sounds good, a lot to learn! :) | 18:06 |
| jimbaker | anonymike, i think you are going to be doing this anyway, that's all. at least standup ES with docker | 18:07 |
| anonymike | yep yep, that was the plan | 18:07 |
| jimbaker | anonymike, so the next obvious step is to just say, standup two docker thingies (ES and craton). hence docker compose | 18:07 |
| thomasem | Why don't we ask the support folks? I wonder what the operational cost of, say, Cassandra is compared to MySQL + ES? | 18:08 |
| sulo | thomasem: yes good idea | 18:08 |
| thomasem | I am not an operator of any of these, so I don't lknow. | 18:08 |
| thomasem | know* | 18:08 |
| thomasem | Would be useful information. | 18:08 |
| jimbaker | let's just not have cassandra be an option. it's not | 18:08 |
| thomasem | But, there are consistency concerns. | 18:08 |
| sulo | thomasem: i come from ops world, so i love to run as less as possible | 18:08 |
| jimbaker | perfectly valid option for timeseries data | 18:09 |
| jimbaker | when running at scale, you have to multiple systems | 18:09 |
| jimbaker | optimized by capability | 18:09 |
| thomasem | sulo: The problem to me comes more down to a question of right tool for the right job. | 18:09 |
| thomasem | But, I can see how those would be at odds in this scenario. When Craton goes to a more distributed model, I really only imagine one ES cluster at the global level. | 18:09 |
| thomasem | For support | 18:09 |
| sulo | thomasem: sure, and thats what i am syaing too .. what is the problem, can we solve it without adding anything ... etc | 18:10 |
| jimbaker | mysql 8 does have some enhanced functionality that's nice such as builtin memcached support | 18:10 |
| jimbaker | i don't know if we want to use that vs redis | 18:10 |
| sulo | thomasem: not sure CDC model will works with that | 18:10 |
| jimbaker | so CDC doesn't have this issue | 18:11 |
| thomasem | sulo: why not? Isn't it all supposed to be replicating up to a global Craton? | 18:11 |
| thomasem | And, if so, could we not pull from that to populate ES? | 18:11 |
| jimbaker | only need ES where there's support | 18:11 |
| sulo | thomasem: yes ,, but we are adding this so folks can search | 18:11 |
| sulo | thomasem: search is local ? | 18:11 |
| jimbaker | search should not be local | 18:11 |
| jimbaker | in CDC case | 18:11 |
| jimbaker | CDC only needs to run automation | 18:12 |
| sulo | jimbaker: how do you mean ? | 18:12 |
| jimbaker | people are not going to be seaching there, dashboarding stuff | 18:12 |
| sulo | jimbaker: why not ? | 18:12 |
| thomasem | They're already constrained on what can be run in CDC... do any of these other solutions even work for that case? | 18:12 |
| jimbaker | if they do, then assume greater costs. i don't see RPC doing this | 18:12 |
| sulo | thomasem: yeah good question | 18:12 |
| jimbaker | it really doesn't make sense | 18:12 |
| jimbaker | CDC by definition is already small | 18:13 |
| sulo | jimbaker: i think its more of a restriction question | 18:13 |
| sulo | jimbaker: na cant assume that | 18:13 |
| jimbaker | we don't want support personnel having to log into a CDC to do a search | 18:13 |
| sulo | jimbaker: and afaik, its not | 18:13 |
| jimbaker | sulo, when i say small, i mean just one or maybe a few tenants | 18:14 |
| jimbaker | it can be readily handled by a mysql cluster, with proper indexing | 18:14 |
| thomasem | Let's lay out, specifically, the fundamental reasons for not doing Mongo, Cassandra, Redis, etc. be it consistency concerns, scalability, what have you... and put that information somewhere so we can avoid this chat we've had several times already. :P | 18:14 |
| sulo | thomasem: +1 | 18:14 |
| jimbaker | and most importantly, people are not directly looking at the craton instance. the whole point of this is we aggregate up to an overall support cluster, so they can run queries there | 18:14 |
| jimbaker | it's in the use case from support :) | 18:15 |
| jimbaker | UC1? | 18:15 |
| jimbaker | sulo, thomasem, worthwhile doing for sure | 18:15 |
| jimbaker | i'm always glad for assumptions to be challenged however | 18:15 |
| thomasem | And it'd be good to reference specific documentation of the problem that is at odds with Craton's goals. | 18:15 |
| thomasem | Because one person says consistency, another says it's an operational problem, another says X, but we're never going to put it to bed until it's documented. | 18:16 |
| sulo | i would really like to understand the problem fist, see options etc ... we can continue with es stuff .. but i feel like this just came about because one seach was slow .. the data for search right now is tiny .. smaller than tiny | 18:17 |
| jimbaker | it is not about the speed | 18:17 |
| jimbaker | we already know that the query can be optimized | 18:17 |
| sulo | jimbaker: so whats it then ? | 18:17 |
| sulo | searchability ? | 18:17 |
| jimbaker | it's about the human aspect of search | 18:17 |
| sulo | example ? | 18:18 |
| jimbaker | being able to do an approx type of search | 18:18 |
| thomasem | It seemed like support was worried about not having all of the information. So, they'd like a full-text search approach. Fuzzy searching, not knowing full values... | 18:18 |
| jimbaker | again, no way do we even have time to do this type of investment in searching | 18:18 |
| jimbaker | we have many other issues to solve | 18:19 |
| jimbaker | and support directly using craton in this way was not an original goal. instead automation was the goal. (hence why strong consistency. fortunately we can have cake & eat too | 18:20 |
| jimbaker | with a little integration work | 18:20 |
| sulo | well, adding a new component to solve this does not sould like the right approach to me | 18:20 |
| sulo | not saying its not the right solution | 18:21 |
| thomasem | Because of the operational overhead? | 18:21 |
| sulo | just saying .. we need more info before jumping in | 18:21 |
| sulo | thomasem: thats one apect for sure | 18:21 |
| jimbaker | it's worthwhile to point out that this is the openstack approach, namely searchlight; although searchlight is tightly bound to horizon as i understand it | 18:21 |
| thomasem | I think that's fair. We could use more information. It was kind of a passing comment yesterday. | 18:21 |
| jimbaker | having said that, any ES work could support searchlight in the future | 18:22 |
| jimbaker | so we are not at all treading on new ground here | 18:22 |
| jimbaker | ok, i think we are good for now. biab, going to get a run in! | 18:24 |
| sulo | anyway, good discussion here ... i would definitely like to see what support wants written somehere, with examples etc ... and what thomasem said ^ re: why not mongo etc | 18:25 |
| jimbaker | http://mysqlserverteam.com/indexing-json-documents-via-virtual-columns/ | 18:26 |
| jimbaker | thomasem, something we can look into | 18:26 |
| jimbaker | but first, we should reload data using the alembic update i have | 18:26 |
| jimbaker | ok, really going for that run now :) | 18:26 |
| openstackgerrit | Michael Porras proposed openstack/craton master: Adding wrapper functions to tools https://review.openstack.org/449230 | 18:32 |
| anonymike | grabbing lunch and heading home | 18:40 |
| anonymike | brb | 18:40 |
| sigmavirus | jimbaker: did you figure out your issues with betamax yet? | 19:28 |
| jimbaker | sigmavirus, not yet, but haven't looked since posting that gist | 19:54 |
| jimbaker | i'm ok with it just works on ubuntu for now | 19:54 |
| thomasem | my brother just got in, so I need to step away. I'll catch up laters! | 21:01 |
| anonymike | later thomasem! | 21:01 |
| jimbaker | thomasem, take care! | 21:02 |
| jimbaker | anonymike, so with respect to the python client | 21:08 |
| anonymike | yes? | 21:08 |
| jimbaker | you can get the dict of the json resource, possibly further extended from lazyloading, with | 21:08 |
| jimbaker | resource.to_dict() | 21:08 |
| jimbaker | sorry, meant to say, of the *crud* resourfce | 21:09 |
| jimbaker | although of course it was originally json for our purposes | 21:09 |
| jimbaker | eg: >>> host.to_dict() | 21:09 |
| jimbaker | {'region_id': 1, 'cloud_id': 1, 'project_id': 'ffc2a2e0-260e-43e8-9b71-207ab54584c9', 'created_at': '2017-03-23T21:05:10.000000', 'active': True, 'parent_id': 1, 'device_type': 'server', 'name': 'host0.ORD135.C0001.C-1.example1.com', 'links': [{'href': 'http://127.0.0.1:7780/v1/network-devices/1', 'rel': 'up'}], 'id': 2, 'note': None, 'ip_address': '192.168.1.5', 'cell_id': 1, 'updated_at': None} | 21:09 |
| jimbaker | so this should be something you can directly post in ES | 21:10 |
| anonymike | oh nice, | 21:10 |
| anonymike | perfect | 21:10 |
| jimbaker | yeah, don't want tedim if we can avoid. also, something like this is perhaps more convenient: | 21:11 |
| jimbaker | >>> hosts = list(craton.hosts.list(details='all')) | 21:11 |
| jimbaker | >>> hosts[0].to_dict() | 21:11 |
| jimbaker | {'region_id': 1, 'cloud_id': 1, 'project_id': 'ffc2a2e0-260e-43e8-9b71-207ab54584c9', 'created_at': '2017-03-23T21:05:10.000000', 'active': True, 'parent_id': 1, 'device_type': 'server', 'name': 'host0.ORD135.C0001.C-1.example1.com', 'links': [{'href': 'http://127.0.0.1:7780/v1/network-devices/1', 'rel': 'up'}], 'id': 2, 'note': None, 'variables': {'openstack_release': 'juno', 'console_host': '10.10.1.100', 'nova_console_type': 'novnc', 'tempest_pub | 21:11 |
| jimbaker | lic_subnet_cidr': '192.168.1.0/22', 'neutron_l2_population': True, 'glance_default_store': 'swift', 'cell_capabilities': 'flavor_classes=performance2'}, 'ip_address': '192.168.1.5', 'cell_id': 1, 'updated_at': None} | 21:11 |
| jimbaker | craton.hosts.list returns a generator (possibly could be better named....) | 21:12 |
| jimbaker | details='all' means grab all variables | 21:12 |
| jimbaker | i suspect in the future we may offer a way to get the "blame" instead | 21:12 |
| jimbaker | which is the specific hierarchy of variables | 21:13 |
| jimbaker | anyway, the point of this is, you need json docs to populate ES | 21:13 |
| jimbaker | and here's the most direct way to do so :) | 21:13 |
| anonymike | lol | 21:13 |
| anonymike | thank you | 21:13 |
| anonymike | i hadn't seen the blame stuff | 21:13 |
| anonymike | interesting | 21:13 |
| jimbaker | anonymike, can be found here: >>> hosts = list(craton.hosts.list(details='all')) | 21:15 |
| jimbaker | >>> hosts[0].to_dict() | 21:15 |
| jimbaker | {'region_id': 1, 'cloud_id': 1, 'project_id': 'ffc2a2e0-260e-43e8-9b71-207ab54584c9', 'created_at': '2017-03-23T21:05:10.000000', 'active': True, 'parent_id': 1, 'device_type': 'server', 'name': 'host0.ORD135.C0001.C-1.example1.com', 'links': [{'href': 'http://127.0.0.1:7780/v1/network-devices/1', 'rel': 'up'}], 'id': 2, 'note': None, 'variables': {'openstack_release': 'juno', 'console_host': '10.10.1.100', 'nova_console_type': 'novnc', 'tempest_pub | 21:15 |
| jimbaker | lic_subnet_cidr': '192.168.1.0/22', 'neutron_l2_population': True, 'glance_default_store': 'swift', 'cell_capabilities': 'flavor_classes=performance2'}, 'ip_address': '192.168.1.5', 'cell_id': 1, 'updated_at': None} | 21:15 |
| jimbaker | oops, wrong paste | 21:15 |
| jimbaker | https://github.com/openstack/craton/blob/master/craton/db/sqlalchemy/models.py#L204 | 21:15 |
| jimbaker | anonymike, that is just a taste of the sort of functionality around "governance" we want to offer in the future | 21:16 |
| jimbaker | who set the var? when? why? what were recent values? etc | 21:16 |
| jimbaker | we think these are important questions to answer in a cmdb | 21:17 |
| anonymike | exactly! something people were asking for in galaxy but I never got around to even investigating | 21:17 |
| jimbaker | yeah, so still stuff to build, but at least a place to hook this onto | 21:17 |
| jimbaker | next up from me, i'm working on rbac so we can control access to read/write at a fine-grained level | 21:18 |
| anonymike | i was reading through the models code a few days ago and missed the polymorphic associations blog, gonna read up on that tonight | 21:18 |
| jimbaker | polymorphic assoc stuff is huge for us | 21:19 |
| jimbaker | significant code minimization, makes queries much easier to write | 21:19 |
| jimbaker | at the cost of an extra assoc table | 21:19 |
| jimbaker | but in practice it's fine, because we really care about where we get the var from - it would be exceedingly tedious to write queries to answer that question without the use of polymorphic assoc | 21:20 |
| jimbaker | anonymike, best bet, go through with a mysql client and see how they are created | 21:21 |
| jimbaker | also highly useful, you can print a query at any time | 21:21 |
| anonymike | good idea | 21:21 |
| jimbaker | should also share with you some code i use to literalize a query from params - could be a useful debugging tool that could live somewhere - just not in craton itself | 21:22 |
| anonymike | I'd love to see that | 21:22 |
| jimbaker | cool, will put together that gist, and we can go from there | 21:23 |
| anonymike | thanks! | 21:23 |
| jimbaker | np! | 21:24 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 22:48 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 22:49 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 22:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
