13:01:58 <HenryG> #startmeeting neutron_db
13:01:59 <openstack> Meeting started Mon Aug  4 13:01:58 2014 UTC and is due to finish in 60 minutes.  The chair is HenryG. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:02:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:02:02 <openstack> The meeting name has been set to 'neutron_db'
13:02:12 <HenryG> #topic Open discussion
13:02:29 <jlibosva> I have one item to discuss
13:02:39 <HenryG> jlibosva: Let's talk about https://bugs.launchpad.net/neutron/+bug/1346658
13:02:40 <uvirtbot> Launchpad bug 1346658 in neutron "All DB model classes should be consolidated into one directory" [Low,In progress]
13:02:48 <jlibosva> yes!
13:03:05 <HenryG> As I suspected, it met resistance. :(
13:03:32 <jlibosva> I worked on some function that will go through the source tree and import models
13:03:57 <HenryG> OK. Did you encounter any problems?
13:04:01 <jlibosva> I made some basic benchmark of three approaches http://paste.openstack.org/show/89984/
13:04:14 <jlibosva> No problems, I think it's just a matter of preference
13:04:52 <HenryG> Good data! Thanks!
13:04:54 <jlibosva> One way is that it goes through directories with os.walk
13:05:17 <jlibosva> then looks into .py files until it hits "model_base.BASEV2"
13:05:25 <jlibosva> if it founds this, it imports the module
13:05:34 <jlibosva> that's the column "import relevant packages "
13:05:58 <jlibosva> approach No. 2: import all python packages from neutron source tree with pkgutil
13:06:08 <jlibosva> that's "import all packages"
13:06:41 <jlibosva> and last is what we currently have and sort of simulates the approach if we would have models under one directory (current head.py)
13:07:13 <HenryG> I am surprised that "import head" uses more memory than "import relevant packages".
13:07:25 <jlibosva> that probably means there are modules that doesn't contain models
13:07:36 <jlibosva> I was surprised too
13:08:11 <HenryG> As expected, import head is by far the fastest.
13:08:37 <jlibosva> yes, but cons is it will need the refactoring if we need to get rid of this file
13:09:42 <jlibosva> I think we need to define the reason of having models consolidated or if automatically collected models are sufficient for our needs
13:10:06 <jlibosva> as far as I understand we need to have metadata with all models just for testing. is there any other use?
13:10:20 <jlibosva> Last meeting salv-orlando said he worked on something and he needed this
13:11:04 <HenryG> I only know of testing uses, but please hunt down salv-orlando and get an answer. ;)
13:11:13 <salv-orlando> jlibosva: I don’t remember what we were talking about, unfortunately
13:11:52 <jlibosva> HenryG: what kind of testing - comparing final schema?
13:12:05 <jlibosva> sorry, I lost the track a bit :-/
13:12:06 <salv-orlando> ah right the thing about consolidated models? and why we need them?
13:12:15 <HenryG> salv-orlando: yup
13:12:15 <jlibosva> yep
13:12:21 <salv-orlando> autogenerate
13:12:50 <salv-orlando> for new revision. You need to import all models. Eitgher we consolidate or we use head.py to that aim
13:13:01 <salv-orlando> I think the latter requires way less code churn.
13:13:03 <HenryG> How could I forget that!
13:13:26 <jlibosva> I forgot that too, weird
13:13:47 <HenryG> It's so easy with "import head", easy to forget. :)
13:14:12 <jlibosva> salv-orlando: what is your opinion on automatically walk through code tree and import all models? I sent benchmark http://paste.openstack.org/show/89984/
13:15:22 <HenryG> jlibosva: For autogenerate IMHO even the worst case memory and times there are insignificant.
13:16:05 <jlibosva> I think that applies for testing too. 80MB doesn't seem that high
13:18:49 <HenryG> The only other argument (not sure if it has any validity) is that neutron will be different from other projects using sqlalchemy. The others have their models consolidated. But Neutron has many more models than other projects, with lots of vendors-specific models.
13:20:27 <jlibosva> what if we create a requirement that every plugin should have all models importable (somehow) by single import and models for core of neutron will be in neutron/db/
13:20:47 <jlibosva> this way head.py will be easier to maintain and also models will be part of plugins
13:21:05 <jlibosva> but still there will be requirement to put import to head.py in case new plugin is introduced
13:22:18 <HenryG> jlibosva: I think tree-walk is still going to encounter least resistance.
13:23:38 <jlibosva> we should choose the best approach not the one with least resistance :)
13:24:35 <jlibosva> I have no strong opinion on the approach. I just don't like wasting resources. But these days megabytes doesn't seem to be considered as a waste of resource
13:24:39 <salv-orlando> jlibosva: I think in this case the best approach is the more maintainable one.
13:25:25 <jlibosva> salv-orlando: I think brute-force-import-everything-we-got is the most maintainable, right?
13:26:26 <salv-orlando> jlibosva: it depends on what brute force is. If brute force means that we need to do a check, possibly manual, every time a new module with models is added, then it’s a bit annoying
13:27:00 <salv-orlando> jlibosva: if you consider the queue of plugins and drivers waiting to be merges, that’s going to be quite a few modules
13:27:02 <jlibosva> salv-orlando: no, no. It's automatically imported whole neutron source tree.
13:27:24 <jlibosva> but there might be a problem
13:27:30 <salv-orlando> jlibosva: ok then unless the code makes people puke it’s fine ;)
13:27:42 <jlibosva> some vendor-plugins use specific libraries which makes some import to fail
13:27:53 <HenryG> Moving all models to one directory (once that is done) is also very maintainable.
13:28:21 <jlibosva> so if there is a model in module that contains some extra package, we can face problem
13:29:11 <salv-orlando> HenryG: The downside of that is that we’ll need to implement automated test to ensure no model is out of that package
13:29:27 <salv-orlando> jlibosva: explain the problem. My brain is on strike today.
13:30:19 <jlibosva> salv-orlando: e.g. embrane plugin uses heleosapi package
13:31:06 <jlibosva> salv-orlando: when importing embrane modules that use this package, import fails. if a module will contain model definition and at the same time special package (heleosapi), it won't import the model
13:31:56 <salv-orlando> that is because heleosapi is not part of requirements.txt?
13:32:02 <jlibosva> yes
13:32:31 <jlibosva> so far there is no such issue. all models are in modules that are importable with installed packages from requirements
13:32:39 <salv-orlando> jlibosva: this is unrelated, but have you checked that it is on pypi? That’s what I agreed with the embrane devs when we reviewed their plugin
13:32:55 <jlibosva> salv-orlando: I did not
13:33:21 <salv-orlando> jlibosva: if it’s not this might a wtf moment ;)
13:34:24 <jlibosva> salv-orlando: sounds like wtf moment
13:34:33 <salv-orlando> jlibosva: anyway, I think this limitation if fine. I think no module of neutron should ever raise an import error.
13:34:40 <jlibosva> but I need to make sure
13:34:48 <salv-orlando> you know what I’m wondering now?
13:35:01 <salv-orlando> jlibosva: I’m wondering how the embrane UT pass
13:35:08 <jlibosva> salv-orlando: they mock the module
13:35:18 <jlibosva> sys.modules['heleosapi'] = mock.Mock()
13:35:45 <jlibosva> I think I saw that somewhere
13:36:39 <HenryG> I know cisco plugins use a package called ncclient that is also not in requirements. But at least it's in pypi.
13:36:53 <salv-orlando> jlibosva: right. Summarizing, I think the approach you’re suggesting is feasible. This problem that you’ve pointed out is actually a problem of the module that should not raise importerror.
13:37:20 <salv-orlando> jlibosva, HenryG: worst case we’ll do a bit of refactoring to move models in their own module
13:37:30 <jlibosva> so final decision is "import everything"?
13:37:30 <salv-orlando> it should be as easy as doing cut & paste
13:37:42 <salv-orlando> jlibosva: final-ish.
13:38:11 <salv-orlando> this is the best proposal that came out of the meeting. Then let’s go to the mailing list and see if somebody has a different opinion.
13:38:25 <salv-orlando> we should ask markmcclain at least, if not else because he did the original code.
13:38:53 <jlibosva> ok, I'll send an email
13:39:26 <HenryG> #action jlibosva to send email about strategy for importing models
13:39:30 <HenryG> Thanks jlibosva !
13:40:05 <HenryG> Any more questions about this item?
13:40:50 <jlibosva> not from me
13:41:04 <HenryG> OK
13:41:15 <HenryG> salv-orlando: Can you take a look at https://review.openstack.org/109952 ?
13:41:28 <HenryG> Mark gave it a -2
13:41:37 <salv-orlando> HenryG: ok
13:41:43 <HenryG> Ann is out this week
13:42:03 <HenryG> If you could talk to Mark before she returns, that would be great
13:43:50 <jlibosva> that could be also interesting topic to discuss
13:47:12 <HenryG> Some discussion can take place in the code review, maybe?
13:48:55 <HenryG> One more thing I would like to bring up is the urgency to clean up autogenerate.
13:49:08 <HenryG> #link https://review.openstack.org/108700
13:50:12 <HenryG> Even though it currently uses head.py, it should get merged as-is for now.
13:51:05 <HenryG> It is almost impossible for developers to use autogenerate at the moment.
13:51:29 <jlibosva> I agree, the attention from core is low on that patch though.
13:51:38 <HenryG> salv-orlando: ^^ hint hint
13:52:22 <HenryG> I will also try to ping Mark for attention
13:52:44 <jlibosva> HenryG++
13:52:49 <salv-orlando> jlibosva: I removed the +2 and the went on holyday
13:53:04 <salv-orlando> because I had one concern about hte line 40/41 you’re removing from env.py
13:53:18 <salv-orlando> and I suspected this would cause to not import the modles anymore
13:53:24 <salv-orlando> modles/models
13:53:32 <jlibosva> salv-orlando: that's replaced by head, it contains all models
13:54:09 <salv-orlando> jlibosva: right… should work
13:54:23 <jlibosva> salv-orlando: it just works!
13:56:09 <HenryG> And the final item on my list is another request to salv-orlando: https://review.openstack.org/#/c/108690/2/neutron/plugins/vmware/dbexts/models.py
13:56:39 <HenryG> Can you use your inside connections to find the right person to check that?
13:58:00 <salv-orlando> HenryG: I should be able to answer all questions, but I need to refresh my mind. Therefore I’ll ping arosen later on today.
13:58:14 <HenryG> salv-orlando: Much appreciated!
13:58:46 <HenryG> That is all from me today/
13:59:30 <HenryG> Time is over. If there is anything else we can talk in #openstack-neutron
13:59:40 <HenryG> Thanks everyone!
13:59:41 <jlibosva> HenryG: salv-orlando thanks!
13:59:48 <HenryG> #endmeeting