2015-02-18T00:00:01 thanks for the help 2015-02-18T00:00:24 np 2015-02-18T00:01:59 *** jaosorior has quit IRC 2015-02-18T00:06:32 Merged openstack/oslo.config: Fixes deprecation warning for oslo.config in cfg.py https://review.openstack.org/156730 2015-02-18T00:15:29 Merged openstack/oslo.config: Support i18n messages in config generator https://review.openstack.org/145280 2015-02-18T00:26:02 Merged openstack/oslo.log: Update Oslo imports to remove namespace package https://review.openstack.org/149090 2015-02-18T00:27:21 *** jaypipes has quit IRC 2015-02-18T00:30:09 dhellmann: I have got myself setup 2015-02-18T00:33:23 *** dims__ has quit IRC 2015-02-18T00:33:54 *** dims__ has joined #openstack-oslo 2015-02-18T00:33:55 *** dims__ has quit IRC 2015-02-18T00:35:14 *** dims__ has joined #openstack-oslo 2015-02-18T00:37:02 Merged openstack/oslo.config: Add a list_all_sections method to ConfigOpts https://review.openstack.org/148936 2015-02-18T00:37:19 which is the development repository for oslo? 2015-02-18T00:41:29 *** bknudson has quit IRC 2015-02-18T00:44:42 *** david-lyle has quit IRC 2015-02-18T01:08:22 *** hemna is now known as hemnafk 2015-02-18T01:17:43 *** achanda has quit IRC 2015-02-18T01:22:12 *** zzzeek has quit IRC 2015-02-18T01:31:59 *** zzzeek has joined #openstack-oslo 2015-02-18T01:38:18 *** sigmavirus24 is now known as sigmavirus24_awa 2015-02-18T01:58:58 *** salv-orlando has quit IRC 2015-02-18T02:09:41 *** stevemar has quit IRC 2015-02-18T02:11:27 *** david-lyle has joined #openstack-oslo 2015-02-18T02:14:43 *** mriedem has joined #openstack-oslo 2015-02-18T02:15:07 *** mriedem has quit IRC 2015-02-18T02:16:15 *** mriedem has joined #openstack-oslo 2015-02-18T02:19:49 Joshua Harlow proposed openstack/taskflow: WIP for having the ability to have tooz find workers+topics https://review.openstack.org/151495 2015-02-18T02:22:46 *** stevemar has joined #openstack-oslo 2015-02-18T02:24:02 *** takedakn has joined #openstack-oslo 2015-02-18T02:26:35 *** takedakn has quit IRC 2015-02-18T02:26:45 *** noelbk has quit IRC 2015-02-18T02:26:55 *** takedakn has joined #openstack-oslo 2015-02-18T02:33:55 *** tsekiyam_ has joined #openstack-oslo 2015-02-18T02:37:26 *** tsekiyama has quit IRC 2015-02-18T02:38:10 *** tsekiyam_ has quit IRC 2015-02-18T02:43:05 *** mtanino has quit IRC 2015-02-18T02:48:25 *** takedakn has quit IRC 2015-02-18T02:59:46 *** salv-orlando has joined #openstack-oslo 2015-02-18T03:03:51 Merged openstack-dev/hacking: H105: also check for Authors and authors https://review.openstack.org/147201 2015-02-18T03:07:36 Merged openstack/oslo-incubator: Link hacking guidelines into dev docs https://review.openstack.org/152238 2015-02-18T03:08:16 Merged openstack/oslo-incubator: Have a little fun with release notes https://review.openstack.org/152680 2015-02-18T03:12:20 *** boris-42 has quit IRC 2015-02-18T03:13:09 *** yamahata has quit IRC 2015-02-18T03:21:52 *** himangi has quit IRC 2015-02-18T03:22:40 *** harlowja_ is now known as harlowja_away 2015-02-18T03:28:48 *** rushiagr_away is now known as rushiagr 2015-02-18T03:35:10 *** david-lyle has quit IRC 2015-02-18T03:35:43 *** david-lyle has joined #openstack-oslo 2015-02-18T03:40:19 *** david-lyle has quit IRC 2015-02-18T03:42:27 *** mriedem has left #openstack-oslo 2015-02-18T03:42:32 *** mriedem has quit IRC 2015-02-18T03:44:30 *** tsekiyama has joined #openstack-oslo 2015-02-18T03:48:04 Merged openstack/oslo.config: Do not import our namespace package https://review.openstack.org/156820 2015-02-18T03:48:39 *** tsekiyama has quit IRC 2015-02-18T03:49:26 *** dims__ has quit IRC 2015-02-18T03:50:21 *** harlowja_at_home has joined #openstack-oslo 2015-02-18T03:51:17 *** salv-orlando has quit IRC 2015-02-18T03:54:52 *** amotoki has joined #openstack-oslo 2015-02-18T04:16:47 Joshua Harlow proposed openstack/taskflow: Tweaks to atom documentation https://review.openstack.org/156844 2015-02-18T04:19:11 *** takedakn has joined #openstack-oslo 2015-02-18T04:20:21 *** stevemar has quit IRC 2015-02-18T04:21:15 *** harlowja_at_home has quit IRC 2015-02-18T04:22:18 *** david-lyle has joined #openstack-oslo 2015-02-18T04:23:15 *** david-lyle is now known as david-lyle_afk 2015-02-18T04:29:59 *** achanda has joined #openstack-oslo 2015-02-18T04:33:32 *** takedakn has quit IRC 2015-02-18T04:36:28 *** stevemar has joined #openstack-oslo 2015-02-18T04:45:32 *** achanda has quit IRC 2015-02-18T04:46:44 Merged openstack/taskflow: Add todo note for kombu pull request https://review.openstack.org/156472 2015-02-18T04:50:01 *** dims__ has joined #openstack-oslo 2015-02-18T04:52:05 Eric Brown proposed openstack/oslo.vmware: Add bandit to tox for security static analysis https://review.openstack.org/156838 2015-02-18T04:52:08 *** achanda has joined #openstack-oslo 2015-02-18T04:54:47 *** dims__ has quit IRC 2015-02-18T04:59:35 *** achanda has quit IRC 2015-02-18T05:00:50 *** achanda has joined #openstack-oslo 2015-02-18T05:02:48 Merged openstack/taskflow: Improve multilock class and its associated unit test https://review.openstack.org/155663 2015-02-18T05:11:47 *** achanda has quit IRC 2015-02-18T05:33:25 *** yamahata has joined #openstack-oslo 2015-02-18T05:36:28 *** salv-orlando has joined #openstack-oslo 2015-02-18T05:43:31 *** zzzeek has quit IRC 2015-02-18T05:50:51 Eric Brown proposed openstack/oslo.vmware: Resolve warnings from Bandit https://review.openstack.org/156889 2015-02-18T05:53:38 *** achanda has joined #openstack-oslo 2015-02-18T06:01:10 OpenStack Proposal Bot proposed openstack/oslo.db: Imported Translations from Transifex https://review.openstack.org/156890 2015-02-18T06:02:11 Joshua Harlow proposed openstack/taskflow: Use the enum library for the retry strategy enumerations https://review.openstack.org/156893 2015-02-18T06:03:37 *** crc32 has quit IRC 2015-02-18T06:04:38 OpenStack Proposal Bot proposed openstack/oslo.vmware: Imported Translations from Transifex https://review.openstack.org/156897 2015-02-18T06:08:34 *** ajo_ has joined #openstack-oslo 2015-02-18T06:09:07 Joshua Harlow proposed openstack/taskflow: Use the enum library for the retry strategy enumerations https://review.openstack.org/156893 2015-02-18T06:11:00 *** ajo has quit IRC 2015-02-18T06:14:04 Joshua Harlow proposed openstack/taskflow: Use the enum library for the retry strategy enumerations https://review.openstack.org/156893 2015-02-18T06:20:56 *** achanda has quit IRC 2015-02-18T06:22:54 *** achanda has joined #openstack-oslo 2015-02-18T06:24:28 *** exploreshaifali has joined #openstack-oslo 2015-02-18T06:32:35 *** achanda has quit IRC 2015-02-18T06:33:59 *** exploreshaifali has quit IRC 2015-02-18T06:35:30 *** ajo_ is now known as ajo 2015-02-18T06:41:29 *** miqui has joined #openstack-oslo 2015-02-18T06:56:37 *** achanda has joined #openstack-oslo 2015-02-18T07:10:09 *** takedakn has joined #openstack-oslo 2015-02-18T07:18:40 *** exploreshaifali has joined #openstack-oslo 2015-02-18T07:31:11 *** vigneshvar has joined #openstack-oslo 2015-02-18T07:39:54 *** miqui has quit IRC 2015-02-18T07:41:06 *** miqui has joined #openstack-oslo 2015-02-18T07:48:21 *** takedakn has quit IRC 2015-02-18T07:49:37 *** miqui has quit IRC 2015-02-18T07:51:29 *** salv-orlando has quit IRC 2015-02-18T07:53:52 *** andreykurilin_ has joined #openstack-oslo 2015-02-18T07:58:53 *** takedakn has joined #openstack-oslo 2015-02-18T08:00:28 *** boris-42 has joined #openstack-oslo 2015-02-18T08:06:34 *** miqui has joined #openstack-oslo 2015-02-18T08:07:11 *** takedakn has quit IRC 2015-02-18T08:15:27 *** takedakn has joined #openstack-oslo 2015-02-18T08:15:59 *** yamahata has quit IRC 2015-02-18T08:16:25 *** yamahata has joined #openstack-oslo 2015-02-18T08:18:45 *** ajo has quit IRC 2015-02-18T08:21:00 *** rushiagr is now known as rushiagr_away 2015-02-18T08:27:45 *** viktors|afk is now known as viktors 2015-02-18T08:28:26 *** rushiagr_away is now known as rushiagr 2015-02-18T08:30:34 *** miqui has quit IRC 2015-02-18T08:31:01 *** stevemar has quit IRC 2015-02-18T08:31:25 *** achanda has quit IRC 2015-02-18T08:33:38 *** inc0 has joined #openstack-oslo 2015-02-18T08:34:20 *** himangi has joined #openstack-oslo 2015-02-18T08:35:33 *** ajo has joined #openstack-oslo 2015-02-18T08:48:43 *** salv-orlando has joined #openstack-oslo 2015-02-18T08:54:15 *** andreykurilin_ has quit IRC 2015-02-18T08:56:18 *** ajo has quit IRC 2015-02-18T09:01:20 *** salv-orlando has quit IRC 2015-02-18T09:01:35 *** ajo has joined #openstack-oslo 2015-02-18T09:01:36 *** salv-orlando has joined #openstack-oslo 2015-02-18T09:04:18 *** i159 has joined #openstack-oslo 2015-02-18T09:10:17 *** salv-orlando has quit IRC 2015-02-18T09:10:50 *** salv-orlando has joined #openstack-oslo 2015-02-18T09:17:25 *** exploreshaifali has quit IRC 2015-02-18T09:18:44 *** dulek has joined #openstack-oslo 2015-02-18T09:30:07 Michal Jastrzebski (inc0) proposed openstack/oslo.versionedobjects: Fixes for heat implementation https://review.openstack.org/154835 2015-02-18T09:30:21 *** takedakn has quit IRC 2015-02-18T09:32:10 *** salv-orlando has quit IRC 2015-02-18T09:32:44 *** salv-orlando has joined #openstack-oslo 2015-02-18T09:35:34 *** salv-orlando has quit IRC 2015-02-18T09:36:10 *** salv-orlando has joined #openstack-oslo 2015-02-18T09:37:38 *** jecarey_ has quit IRC 2015-02-18T09:45:36 *** salv-orlando has quit IRC 2015-02-18T09:46:12 *** salv-orlando has joined #openstack-oslo 2015-02-18T09:49:41 *** ihrachyshka has joined #openstack-oslo 2015-02-18T09:49:47 *** salv-orl_ has joined #openstack-oslo 2015-02-18T09:51:05 *** salv-orl_ has quit IRC 2015-02-18T09:51:10 *** salv-orlando has quit IRC 2015-02-18T09:51:38 *** salv-orlando has joined #openstack-oslo 2015-02-18T09:53:21 Marian Horban proposed openstack/oslo-incubator: Using of waitpid system call was optimized https://review.openstack.org/156345 2015-02-18T09:56:07 *** salv-orlando has quit IRC 2015-02-18T09:56:39 *** salv-orlando has joined #openstack-oslo 2015-02-18T10:03:27 *** exploreshaifali has joined #openstack-oslo 2015-02-18T10:15:59 *** yamahata has quit IRC 2015-02-18T10:24:06 *** takedakn has joined #openstack-oslo 2015-02-18T10:32:38 *** takedakn has quit IRC 2015-02-18T10:34:37 *** takedakn has joined #openstack-oslo 2015-02-18T10:56:22 *** takedakn has quit IRC 2015-02-18T11:09:48 *** kbyrne has quit IRC 2015-02-18T11:11:22 *** exploreshaifali has quit IRC 2015-02-18T11:13:21 *** kbyrne has joined #openstack-oslo 2015-02-18T11:13:42 *** cdent has joined #openstack-oslo 2015-02-18T11:14:35 *** e0ne has joined #openstack-oslo 2015-02-18T11:19:37 *** dims__ has joined #openstack-oslo 2015-02-18T11:20:44 *** amotoki has quit IRC 2015-02-18T11:24:38 *** exploreshaifali has joined #openstack-oslo 2015-02-18T11:28:39 *** exploreshaifali has quit IRC 2015-02-18T11:40:16 *** e0ne is now known as e0ne_ 2015-02-18T11:42:15 ihrachyshka: "Reach out, fix, help. Not complain." - golden words! 2015-02-18T11:50:51 *** e0ne_ has quit IRC 2015-02-18T11:53:32 *** takedakn has joined #openstack-oslo 2015-02-18T12:18:50 *** e0ne has joined #openstack-oslo 2015-02-18T12:20:26 *** salv-orlando has quit IRC 2015-02-18T12:24:28 *** takedakn has quit IRC 2015-02-18T12:37:43 *** amotoki has joined #openstack-oslo 2015-02-18T13:05:00 *** e0ne is now known as e0ne_ 2015-02-18T13:05:47 *** jaypipes has joined #openstack-oslo 2015-02-18T13:09:21 *** himangi has joined #openstack-oslo 2015-02-18T13:09:42 *** e0ne_ is now known as e0ne 2015-02-18T13:09:51 *** rushiagr is now known as rushiagr_away 2015-02-18T13:15:58 *** kgiusti has joined #openstack-oslo 2015-02-18T13:18:42 *** zigo has quit IRC 2015-02-18T13:20:36 *** zigo has joined #openstack-oslo 2015-02-18T13:24:25 *** e0ne_ has joined #openstack-oslo 2015-02-18T13:25:23 *** salv-orlando has joined #openstack-oslo 2015-02-18T13:27:06 *** e0ne has quit IRC 2015-02-18T13:30:35 *** e0ne has joined #openstack-oslo 2015-02-18T13:33:00 *** e0ne_ has quit IRC 2015-02-18T13:37:16 *** gordc has joined #openstack-oslo 2015-02-18T13:49:30 *** trown has quit IRC 2015-02-18T13:49:52 *** vigneshvar has quit IRC 2015-02-18T13:50:30 *** himangi has quit IRC 2015-02-18T13:50:50 *** trown has joined #openstack-oslo 2015-02-18T13:51:01 *** jecarey has joined #openstack-oslo 2015-02-18T13:59:17 *** rluediger has joined #openstack-oslo 2015-02-18T14:01:55 *** rluediger has quit IRC 2015-02-18T14:03:59 *** jecarey has quit IRC 2015-02-18T14:04:31 *** exploreshaifali has joined #openstack-oslo 2015-02-18T14:07:19 *** rushiagr_away is now known as rushiagr 2015-02-18T14:15:35 *** dims__ has quit IRC 2015-02-18T14:20:03 *** dims__ has joined #openstack-oslo 2015-02-18T14:20:35 *** dims__ has quit IRC 2015-02-18T14:21:08 *** dims__ has joined #openstack-oslo 2015-02-18T14:25:22 *** dims__ has quit IRC 2015-02-18T14:29:32 *** amrith is now known as _amrith_ 2015-02-18T14:31:28 *** mriedem has joined #openstack-oslo 2015-02-18T14:35:48 *** Guest37356 has joined #openstack-oslo 2015-02-18T14:38:21 *** jgrimm is now known as zz_jgrimm 2015-02-18T14:49:01 *** Guest37356 is now known as dims__ 2015-02-18T14:51:09 *** ajo has quit IRC 2015-02-18T14:51:29 *** ajo has joined #openstack-oslo 2015-02-18T14:52:55 inc0: replied to your comments 2015-02-18T14:53:09 yeah, I saw it, removing -1 :) 2015-02-18T14:53:19 heh, okay 2015-02-18T14:53:20 also, I've added this patch as requirements to my 2015-02-18T14:53:26 mine* 2015-02-18T14:54:43 inc0: okay, on the timezone thing, what you have looks fine, but since it's part of the api: 2015-02-18T14:55:40 * dansmith thinks 2015-02-18T14:56:00 what you have here lets me store a tz-aware or a tz-unaware datetime in the field if ensure_tzinfo=False 2015-02-18T14:56:35 which means that you could get two Foo objects, one with an aware datetime and one without, and then be unable to compare them 2015-02-18T14:56:43 yeah, basically what it do is "what you put in, you'll get out" 2015-02-18T14:57:04 dansmith: there was a question from harlowja_away in https://review.openstack.org/#/c/156398/ 2015-02-18T14:57:26 I wonder if it would be better to have the flag be "is_tz_aware", so that you either have tz-aware datetimes, forced to utc if they're unaware in the positive case, and tzinfo is stripped out if negative 2015-02-18T14:57:53 I don't see point of removing tzinfo if its there 2015-02-18T14:58:32 it's just to try to ensure consistency of what you have stored, since you can't compare aware and non-aware datetimes 2015-02-18T14:58:36 I don't think its good eighter...its deleting data from object 2015-02-18T14:58:51 well, you could just raise an exception instead of stripping it 2015-02-18T14:59:03 what I meant is, ensure that the awareness matches the flag 2015-02-18T14:59:07 that would be better 2015-02-18T14:59:42 allright, I'll do that 2015-02-18T15:00:06 *** salv-orlando has quit IRC 2015-02-18T15:00:37 inc0: okay, after that, I think we should merge that and I promise to stop arguing about it :) 2015-02-18T15:00:39 inc0: is your primary focus using versionedobjects in heat? (any other project?) 2015-02-18T15:01:05 dansmith, sounds good 2015-02-18T15:01:17 dims__, I've started from heat 2015-02-18T15:01:33 but now I'm helping guys from cinder with implementation 2015-02-18T15:01:44 and other to get specs for L going;) like glance, neutron 2015-02-18T15:02:14 *** salv-orlando has joined #openstack-oslo 2015-02-18T15:03:02 inc0: nice 2015-02-18T15:03:23 *** jecarey_ has joined #openstack-oslo 2015-02-18T15:03:24 *** mtanino has joined #openstack-oslo 2015-02-18T15:04:16 *** daniel3_ has joined #openstack-oslo 2015-02-18T15:05:18 inc0: waiting for a few more reviews from you on the series - https://review.openstack.org/#/c/156662/ 2015-02-18T15:05:24 when you get a chance 2015-02-18T15:06:24 I did like half of them before my manager stormed in and called a meeting;) I'll get back to it as soon as I fix my patch 2015-02-18T15:06:33 haha, thanks 2015-02-18T15:06:42 inc0: this one right? https://review.openstack.org/#/c/154835/ 2015-02-18T15:07:22 yup 2015-02-18T15:09:12 *** cdent_ has joined #openstack-oslo 2015-02-18T15:09:38 *** cdent has quit IRC 2015-02-18T15:09:38 *** cdent_ is now known as cdent 2015-02-18T15:15:08 dansmith, on the closer look, I'll go for removing tzinfo, because timeutils injects it anyway;) 2015-02-18T15:15:51 so by this flag I'll just ensure both ends are eighter tzinfo aware, or not, wile inside it will always be tzinfo aware 2015-02-18T15:16:56 *** sigmavirus24_awa is now known as sigmavirus24 2015-02-18T15:18:43 *** _amrith_ is now known as amrith 2015-02-18T15:18:53 *** e0ne is now known as e0ne_ 2015-02-18T15:20:02 *** e0ne_ is now known as e0ne 2015-02-18T15:22:12 *** zz_jgrimm is now known as jgrimm 2015-02-18T15:24:31 *** himangi has joined #openstack-oslo 2015-02-18T15:24:55 *** stevemar has joined #openstack-oslo 2015-02-18T15:27:17 Michal Jastrzebski (inc0) proposed openstack/oslo.versionedobjects: Fixes for heat implementation https://review.openstack.org/154835 2015-02-18T15:27:46 inc0: heh, okay, will look in a sec 2015-02-18T15:27:48 good morning, all 2015-02-18T15:30:32 thanks 2015-02-18T15:30:42 hello dhellmann 2015-02-18T15:30:44 dims__: do you have some time this morning for some policy reviews? I have a few cleanups queued up and if we can get them in today I'd like them to be in the first release 2015-02-18T15:30:46 hi, inc0 2015-02-18T15:31:00 dims__: that's oslo.policy, sorry, wasn't clear 2015-02-18T15:31:05 dhellmann: in an hour, yes. will do 2015-02-18T15:31:12 dims__: great, thanks! 2015-02-18T15:32:26 *** devlaps has joined #openstack-oslo 2015-02-18T15:32:34 *** exploreshaifali has quit IRC 2015-02-18T15:34:46 dims__, https://review.openstack.org/#/c/156397/ I think we should explicitly say that its for tests 2015-02-18T15:35:01 hmf...maybe just file naming? 2015-02-18T15:37:04 like test_utils.py or fakes.py? what do you think? 2015-02-18T15:37:11 Merged openstack/oslo.versionedobjects: Disable some unstable tests until they are generalized https://review.openstack.org/156660 2015-02-18T15:43:05 inc0: we have used fixture.py in other oslo projects 2015-02-18T15:43:10 dims@dims-mac:~/openstack/oslo$ find . -name fixture.py 2015-02-18T15:43:10 ./oslo.config/oslo/config/fixture.py 2015-02-18T15:43:10 ./oslo.config/oslo_config/fixture.py 2015-02-18T15:43:12 ./oslo.context/oslo_context/fixture.py 2015-02-18T15:43:14 ./oslo.i18n/oslo/i18n/fixture.py 2015-02-18T15:43:16 ./oslo.i18n/oslo_i18n/fixture.py 2015-02-18T15:43:18 ./oslo.utils/oslo_utils/fixture.py 2015-02-18T15:43:21 *** zzzeek has joined #openstack-oslo 2015-02-18T15:43:23 than maybe stick to this name? 2015-02-18T15:43:37 but i am flexible, dont want to mandate 2015-02-18T15:44:05 well, I'm ok with fixture, especially if its present in rest of oslo 2015-02-18T15:44:51 my point is, testing alltogether has its own nomenclature, and I'd like to stick to it when we produce something test related 2015-02-18T15:45:21 (but I'm not stubborn about it) 2015-02-18T15:46:18 Merged openstack/oslo.versionedobjects: Generalize the indirection_api interface https://review.openstack.org/156394 2015-02-18T15:46:26 Merged openstack/oslo.versionedobjects: Remove Nova objects module registration code https://review.openstack.org/156395 2015-02-18T15:46:32 Merged openstack/oslo.versionedobjects: Add conditional object registration https://review.openstack.org/156396 2015-02-18T15:46:37 Merged openstack/oslo.versionedobjects: Generalize remote testing infrastructure https://review.openstack.org/156397 2015-02-18T15:47:02 inc0: i hate to suggest renaming stuff in the middle of a patch series as well :) 2015-02-18T15:47:29 yeah, we can do it as another patch, works for me 2015-02-18T15:48:02 this is just my idea of possible improvement, not mistake by itself 2015-02-18T15:49:12 *** hemnafk is now known as hemna 2015-02-18T15:49:15 *** jecarey_ is now known as jecarey 2015-02-18T15:53:55 *** jaosorior has joined #openstack-oslo 2015-02-18T15:58:07 *** yamahata has joined #openstack-oslo 2015-02-18T16:02:02 *** viktors is now known as viktors|afk 2015-02-18T16:02:20 Doug Hellmann proposed openstack/oslo.policy: Clean up configuration option management https://review.openstack.org/157044 2015-02-18T16:06:33 dims__, would you do the honors?:) https://review.openstack.org/#/c/154835/ 2015-02-18T16:08:18 *** belliott has left #openstack-oslo 2015-02-18T16:09:35 *** inc0 has quit IRC 2015-02-18T16:14:33 *** harlowja_at_home has joined #openstack-oslo 2015-02-18T16:16:14 Steve Martinelli proposed openstack/oslo-incubator: Remove policy from oslo-incubator https://review.openstack.org/152812 2015-02-18T16:16:31 Steve Martinelli proposed openstack/oslo-incubator: Remove policy from oslo-incubator https://review.openstack.org/152812 2015-02-18T16:17:01 Steve Martinelli proposed openstack/oslo-incubator: Prevent update.py from updating policy https://review.openstack.org/152813 2015-02-18T16:18:15 *** tsekiyama has joined #openstack-oslo 2015-02-18T16:24:31 *** pradk has joined #openstack-oslo 2015-02-18T16:27:06 dims__, thanksfor reviewing 2015-02-18T16:28:05 *** david-lyle_afk is now known as david-lyle 2015-02-18T16:34:14 stevemar: waiting for one more review (jobs still running) 2015-02-18T16:43:23 *** harlowja_at_home has quit IRC 2015-02-18T16:43:35 do you know if eventlet supports reentrant lock? 2015-02-18T16:46:21 *** noelbk has joined #openstack-oslo 2015-02-18T16:46:43 haypo, no idea about eventlet specifically, but for lockutils, it seems all our locking mechanisms are not aware of the same threads locking semaphores. so double lock locks the thread 2015-02-18T16:47:54 ihrachyshka: it's writing an article about dead locks in threads, asyncio and eventlet ;) 2015-02-18T16:53:34 Merged openstack/oslo.versionedobjects: Generalize object hash-based change detection https://review.openstack.org/156398 2015-02-18T16:53:36 Merged openstack/oslo.versionedobjects: Generalize object dependency change detection https://review.openstack.org/156399 2015-02-18T16:58:28 Merged openstack/oslo.versionedobjects: Add a test to ensure subclassibility of the object registry https://review.openstack.org/156662 2015-02-18T17:03:10 Merged openstack/oslo.policy: Create the temporary files needed for tests https://review.openstack.org/156811 2015-02-18T17:03:16 Merged openstack/oslo.policy: Change default set of tox environments https://review.openstack.org/156812 2015-02-18T17:09:55 Merged openstack/oslo.versionedobjects: Fixes for heat implementation https://review.openstack.org/154835 2015-02-18T17:10:06 Merged openstack/oslo.policy: Fix i18n imports https://review.openstack.org/156813 2015-02-18T17:10:23 *** amotoki has quit IRC 2015-02-18T17:10:31 Merged openstack/oslo.policy: Update comments about tox configuration https://review.openstack.org/156836 2015-02-18T17:11:10 *** i159 has quit IRC 2015-02-18T17:15:54 dims__: hey, you merged all my patches! 2015-02-18T17:17:57 *** daniel3_ has quit IRC 2015-02-18T17:18:19 *** daniel3_ has joined #openstack-oslo 2015-02-18T17:18:22 *** ihrachyshka has quit IRC 2015-02-18T17:18:30 *** daniel3_ has quit IRC 2015-02-18T17:18:30 *** daniel3_ has joined #openstack-oslo 2015-02-18T17:22:42 *** e0ne is now known as e0ne_ 2015-02-18T17:24:16 *** e0ne_ is now known as e0ne 2015-02-18T17:27:14 Ben Nemec proposed openstack/oslo.config: Log a warning when deprecated opts are used https://review.openstack.org/148020 2015-02-18T17:31:36 *** e0ne has quit IRC 2015-02-18T17:32:56 *** dulek has quit IRC 2015-02-18T17:33:12 *** bknudson has joined #openstack-oslo 2015-02-18T17:33:13 *** dulek has joined #openstack-oslo 2015-02-18T17:34:11 dansmith: is that good or bad? :) 2015-02-18T17:34:20 dims__: good :) 2015-02-18T17:34:25 bnemec: someone made a comment yesterday that https://review.openstack.org/#/c/154642/1/specs/eventlet-best-practices.rst,cm seemed like a poor fit for the specs repo, because it's not really proposing a 'change' per se. I wonder if it fits better in the oslo.concurrency docs? 2015-02-18T17:35:45 dansmith: mind if i propose a rename of checks.py -> fixture.py 2015-02-18T17:35:56 dhellmann: I would be fine with it living there, but I think it needs cross-project buy-in. 2015-02-18T17:36:01 dims__: we have a fixtures thing already, maybe they should be merged 2015-02-18T17:36:17 dansmith: ok, i'll take a stab 2015-02-18T17:36:20 dhellmann: Also, it's not really about how people use oslo.concurrency. Those rules would apply regardless of whether you're using Oslo at all. 2015-02-18T17:36:21 dims__: I'm proposing more changes to checks, so I'll slap a merge after I'm done 2015-02-18T17:36:24 bnemec: yeah, that makes sense, I just wanted to mention it before I forgot 2015-02-18T17:36:34 dims__: or I can let you do it 2015-02-18T17:36:39 bnemec: yeah, we don't have a good place for docs like this 2015-02-18T17:36:40 dansmith: cool, not in a hurry 2015-02-18T17:36:45 *** daniel3_ has quit IRC 2015-02-18T17:36:47 dims__: okay 2015-02-18T17:37:30 *** daniel3_ has joined #openstack-oslo 2015-02-18T17:42:58 *** harlowja_away is now known as harlowja_ 2015-02-18T17:43:17 *** exploreshaifali has joined #openstack-oslo 2015-02-18T17:47:02 Doug Hellmann proposed openstack/oslo.policy: Clean up configuration option management https://review.openstack.org/157044 2015-02-18T17:57:16 haypo afaik ya, it should work fine with reentrant locks (at least in 2.7/2.6) ; https://hg.python.org/releasing/2.7.9/file/753a8f457ddc/Lib/threading.py#l168 is replaced by eventlet (_get_ident() there) 2015-02-18T17:57:46 whether it works in py3.x i'm not sure 2015-02-18T17:58:22 * https://github.com/eventlet/eventlet/blob/master/eventlet/green/threading.py#L11 2015-02-18T17:58:53 so that prevents the eventlet sempahore (which replaces the threading lock) from being acquired twice by the same greenthread 2015-02-18T18:07:58 bnemec: looking at https://review.openstack.org/#/c/148020 -- I wondered why you didn't use versionutils to emit the warning? 2015-02-18T18:09:56 *** dulek has quit IRC 2015-02-18T18:10:16 *** rushiagr is now known as rushiagr_away 2015-02-18T18:12:34 dhellmann: I don't think I realized there was anything but the decorator in there. That should eliminate the duplication logic though. 2015-02-18T18:12:39 *** yamahata has quit IRC 2015-02-18T18:12:55 bnemec: yeah, it also gives us the "die on a deprecated option" behavior that some operators like 2015-02-18T18:13:39 bnemec: comment left -- I just wanted to make sure there wasn't a circular dependency reason or something 2015-02-18T18:13:41 dhellmann: Oh, also we have no incubator code in oslo.config today. 2015-02-18T18:13:53 *** palendae has left #openstack-oslo 2015-02-18T18:14:21 that's not an issue by itself 2015-02-18T18:14:22 I guess we have the cross-test script. 2015-02-18T18:14:34 Yeah, I'll add versionutils. 2015-02-18T18:14:39 there might be a circular dependency issue, though, since that module uses oslo_config 2015-02-18T18:14:43 well, hang on 2015-02-18T18:15:00 bnemec: if versionutils is going to end up in a library other than oslo.config, we'd have an issue with dependencies :-/ 2015-02-18T18:16:07 * dhellmann needs lunch before puzzling this one out 2015-02-18T18:25:42 Oh, update.py isn't aware of the namespace change yet. It's still putting files in oslo.config and hosing up the imports. :-( 2015-02-18T18:26:40 Ah, not update.py. Just openstack-common.conf. That's an easier fix. :-) 2015-02-18T18:29:18 Merged openstack/oslo.db: Imported Translations from Transifex https://review.openstack.org/156890 2015-02-18T18:34:41 Merged openstack/oslo.config: Better check for integer range to account for 0 https://review.openstack.org/151940 2015-02-18T18:41:19 Ben Nemec proposed openstack/oslo.config: Log a warning when deprecated opts are used https://review.openstack.org/148020 2015-02-18T18:41:54 dhellmann: Actually there is a circular dependency with versionutils, but it's okay because we don't need it at import time anyway. 2015-02-18T18:43:06 dims__: so our existing fixtures thing (obj_fixtures) is in tests/ and it's things the objects tests need 2015-02-18T18:43:30 dims__: just to be clear, you're suggesting having a fixture.py outside the tests directory, just s/checks/fixtures/ right? 2015-02-18T18:43:50 dansmith: yep 2015-02-18T18:44:11 these are fixtures to be used in other projects 2015-02-18T18:44:48 dims__: okay 2015-02-18T18:47:30 Joshua Harlow proposed openstack/taskflow: Tweaks to atom documentation https://review.openstack.org/156844 2015-02-18T18:52:58 *** cdent has quit IRC 2015-02-18T18:54:36 dansmith: checks.py has mock, but mock is only in test-requirements 2015-02-18T18:55:19 dims__: hmm, is it legit for me to add something like mock to requirements? 2015-02-18T18:55:44 trying to check other fixtures 2015-02-18T18:56:00 dansmith: only one we do that so far is oslotest 2015-02-18T18:56:22 dims__: hmm 2015-02-18T18:56:23 *** achanda has joined #openstack-oslo 2015-02-18T18:57:42 *** e0ne has joined #openstack-oslo 2015-02-18T18:57:42 *** e0ne is now known as e0ne_ 2015-02-18T18:57:52 *** e0ne_ is now known as e0ne 2015-02-18T18:58:52 dhellmann: wdyt? versionedobjects has a checks.py which needs mock (it has fixtures to be used by other projects). where do we specify mock library versions (requirements.txt vs test-requirements.txt) 2015-02-18T19:03:50 *** vigneshvar has joined #openstack-oslo 2015-02-18T19:07:22 *** pradk has quit IRC 2015-02-18T19:08:31 bnemec: with the circular dep we can't have the 2 libraries depending on each other 2015-02-18T19:09:27 dims__: what do we do with other libs fixtures and the fixtures library 2015-02-18T19:09:50 dhellmann: can't find any other fixture.py that needs mock 2015-02-18T19:10:12 dims__: but they would all use the "fixtures" library, right? 2015-02-18T19:10:25 dims__: https://pypi.python.org/pypi/fixtures 2015-02-18T19:12:02 dims__: oslo.concurrency depends on fixtures directly 2015-02-18T19:13:08 dhellmann: ah 2015-02-18T19:14:01 *** achanda has quit IRC 2015-02-18T19:14:04 I think since this is part of the public API of this library, the dependency can go into requirements.txt 2015-02-18T19:14:22 dansmith: checks.py imports fixtures, so we follow the same pattern as in oslo.concurrency 2015-02-18T19:14:24 *** bknudson has quit IRC 2015-02-18T19:14:25 dhellmann: ack 2015-02-18T19:14:29 if we get push-back from anyone, we can put it in test-requirements.txt and just expect that projects that use it will install those as well 2015-02-18T19:14:53 dims__: how about writing this up as a policy for our specs repo? 2015-02-18T19:15:11 dims__: so fixtures and mock need to go in requirements, right? 2015-02-18T19:15:20 dansmith: y 2015-02-18T19:15:22 dansmith: yeah 2015-02-18T19:16:30 dhellmann: is there another spec review like this i can pattern it on? since this is not a code related thing 2015-02-18T19:17:05 okay cool 2015-02-18T19:17:09 dims__: all of the ones in the policy subdir 2015-02-18T19:17:21 dhellmann: ack will look/submit 2015-02-18T19:17:24 dims__: I just approved a bunch yesterday that had been up for review for a while without any -1 2015-02-18T19:19:08 Joshua Harlow proposed openstack/debtcollector: By default mutate the docstring(s) for deprecations https://review.openstack.org/155988 2015-02-18T19:20:37 Joshua Harlow proposed openstack/debtcollector: By default mutate the docstring(s) for deprecations https://review.openstack.org/155988 2015-02-18T19:21:02 *** achanda has joined #openstack-oslo 2015-02-18T19:25:04 Davanum Srinivas (dims) proposed openstack/oslo.versionedobjects: Cleanup comment and H803 hacking https://review.openstack.org/157114 2015-02-18T19:25:44 *** pradk has joined #openstack-oslo 2015-02-18T19:27:18 Joshua Harlow proposed openstack/debtcollector: By default mutate the docstring(s) for deprecations https://review.openstack.org/155988 2015-02-18T19:31:25 *** sreshetnyak has quit IRC 2015-02-18T19:39:07 Merged openstack/oslo.policy: Clean up configuration option management https://review.openstack.org/157044 2015-02-18T19:45:55 *** vigneshvar has quit IRC 2015-02-18T19:46:06 Joshua Harlow proposed openstack/debtcollector: By default mutate the docstring(s) for deprecations https://review.openstack.org/155988 2015-02-18T19:48:52 *** crc32 has joined #openstack-oslo 2015-02-18T19:56:18 Sridhar Gaddam proposed openstack/oslo.utils: Utility API to generate EUI-64 IPv6 address https://review.openstack.org/137774 2015-02-18T19:59:38 *** bknudson has joined #openstack-oslo 2015-02-18T20:00:34 *** andreykurilin_ has joined #openstack-oslo 2015-02-18T20:02:14 *** vigneshvar has joined #openstack-oslo 2015-02-18T20:04:14 Dan Smith proposed openstack/oslo.versionedobjects: Generalize compatibility testing https://review.openstack.org/157125 2015-02-18T20:04:15 Dan Smith proposed openstack/oslo.versionedobjects: Generalize the object relationships test https://review.openstack.org/157126 2015-02-18T20:04:15 Dan Smith proposed openstack/oslo.versionedobjects: Rename checks to fixture and update requirements https://review.openstack.org/157127 2015-02-18T20:09:19 bnemec: are you around to discuss the circular dep issue? I'm stumped. 2015-02-18T20:09:21 *** achanda has quit IRC 2015-02-18T20:09:59 dhellmann: I'm actually in a debugging session with someone right now. I'll let you know when I'm free. 2015-02-18T20:13:07 *** exploreshaifali has quit IRC 2015-02-18T20:15:11 *** andreykurilin_ has quit IRC 2015-02-18T20:17:01 *** achanda has joined #openstack-oslo 2015-02-18T20:22:26 Davanum Srinivas (dims) proposed openstack/oslo-specs: Policy for specifying external library in requirements.txt https://review.openstack.org/157135 2015-02-18T20:26:39 dhellmann: Okay, crisis averted for now. :-) 2015-02-18T20:27:03 *** devlaps has quit IRC 2015-02-18T20:30:21 bnemec: so I have 3 alternatives for what we do with versionutils, and none seems great 2015-02-18T20:30:33 1. move it to oslo.utils, introducing a dependency between oslo.utils and oslo.config 2015-02-18T20:30:47 2. move it to its own library 2015-02-18T20:30:51 3. leave it in the incubator indefinitely 2015-02-18T20:31:09 only option 3 lets us avoid a circular dependency between libraries :-/ 2015-02-18T20:34:59 dhellmann: Hmm, I'm not coming up with any terrific options either. 2015-02-18T20:35:22 I guess alternative 4 is to put the code in oslo.config, but ew. 2015-02-18T20:35:29 We could copy just report_deprecated_feature into oslo.config, but then we have a potential opt mismatch if we ever change the versionutils opt definition. 2015-02-18T20:35:42 right 2015-02-18T20:36:40 the only options I'm coming up with for decoupling them involve complex callback registry schemes 2015-02-18T20:38:35 bnemec: of all the times I think you would want to exit the app on a deprecation warning, a config option seems the most likely 2015-02-18T20:38:44 dhellmann: Maybe we could make the oslo.config dep in versionutils optional? Default to non-fatal if it's not present? 2015-02-18T20:39:17 bnemec: hmm, or the other way around. If oslo.config can't import the versionutils code, it just uses LOG.warn() 2015-02-18T20:39:59 Yeah, that could also work. 2015-02-18T20:40:00 versionutils can't go in oslo.log? 2015-02-18T20:40:17 harlowja_: oslo.config and oslo.log can't have a circular dependency either 2015-02-18T20:40:22 Eric Brown proposed openstack/oslo.vmware: Resolve warnings from Bandit https://review.openstack.org/156889 2015-02-18T20:40:37 hmmm 2015-02-18T20:40:38 harlowja_: The problem is I just changed https://review.openstack.org/#/c/148020/ to use versionutils. 2015-02-18T20:40:40 *** yamahata has joined #openstack-oslo 2015-02-18T20:40:49 * dhellmann suggested it 2015-02-18T20:40:53 hmmmmssss 2015-02-18T20:41:20 That change has a bad record with reviewer suggestions. They all turn out to be deep, deep rabbit holes. :-) 2015-02-18T20:41:31 heh 2015-02-18T20:42:05 the callback idea was to give oslo.config a function or method like set_deprecation_reporting_func() 2015-02-18T20:42:29 bnemec its cursed, i'll stay away, ha 2015-02-18T20:42:30 the value would default to LOG.warn, but an app could pass in oslo.versionutils.whatever-that-function-si 2015-02-18T20:42:42 but that feels a bit heavy-handed for one function 2015-02-18T20:43:20 dhellmann agreed, would work but does feel sorta odd 2015-02-18T20:43:44 it also feels pretty odd to put an "I'm going to exit" call in a library 2015-02-18T20:43:54 so maybe the right answer really is to just leave versionutils in the incubator indefinitely 2015-02-18T20:44:17 dhellmann: It doesn't exit though. It just raises an exception. 2015-02-18T20:44:20 although that's sort of a technicality 2015-02-18T20:44:28 bnemec does anyone catch it :-P 2015-02-18T20:44:38 oh, true, I guess I just assumed it wasn't being caught 2015-02-18T20:44:44 I'm betting there are lots of except Exceptions that would catch it. :-) 2015-02-18T20:44:44 *** devlaps has joined #openstack-oslo 2015-02-18T20:44:51 true 2015-02-18T20:45:48 And it's off by default. The user has to explicitly ask for that behavior. 2015-02-18T20:46:03 yeah 2015-02-18T20:48:45 bnemec: ok, I give up. Let's go back to the version where you were just using LOG.warn() directly. It should be possible to recover that from gerrit so you don't have to reconstruct it. And if someone complains about the missing exception in the future we can think about it again. Maybe by then we'll have solved another case like this and have a pattern to follow. 2015-02-18T20:49:56 dhellmann: Sounds good. It's a lot better than the nothing we're doing with deprecated opts today. :-) 2015-02-18T20:50:21 put oslo.log,oslo.config into super repo+package called oslo-incubator2; profit :-P 2015-02-18T20:51:22 harlowja_: it would have been technically easier to release the incubator as one big library, except maintenance would have been a nightmare 2015-02-18T20:51:27 :) 2015-02-18T20:52:25 * dhellmann is startled to see blue sky and snow flurries through the same window 2015-02-18T20:53:33 Ben Nemec proposed openstack/oslo.config: Log a warning when deprecated opts are used https://review.openstack.org/148020 2015-02-18T20:54:12 bnemec: see my other comment on PS6 about the implementation of _check_deprecated() though, with the names 2015-02-18T20:54:40 Ah, yeah. I should have fixed that first. 2015-02-18T20:54:55 Would have saved me having to trick Gerrit into letting push a duplicate commit. :-) 2015-02-18T20:55:03 :-) 2015-02-18T20:57:44 dims__, stevemar, morganfainberg, ayoung : I think we're ready for an oslo.policy release. Is there any reason to wait? 2015-02-18T21:02:57 *** mriedem1 has joined #openstack-oslo 2015-02-18T21:03:09 * dhellmann throws caution to the wind and releases oslo.policy 0.1.0 2015-02-18T21:03:18 *** mriedem has quit IRC 2015-02-18T21:04:58 dhellmann: yay! 2015-02-18T21:07:19 hmm, our release notes script doesn't handle this case 2015-02-18T21:09:05 dims__: I guess the last time we did a new lib we didn't announce the release to give us time to make sure the API worked out, right? 2015-02-18T21:09:23 dhellmann: yes 2015-02-18T21:09:30 ok, we'll do the same thing here then 2015-02-18T21:09:32 we did that for oslo.log 2015-02-18T21:09:34 no announcement email 2015-02-18T21:09:40 right, that's the one 2015-02-18T21:09:49 we have so many libraries now! 2015-02-18T21:10:12 dhellmann: there's a bug against oslo.policy asking where the release was, we could just close that for now 2015-02-18T21:11:13 someone eager to try it out, 2015-02-18T21:11:15 I thought I already closed that as invalid 2015-02-18T21:11:31 Ben Nemec proposed openstack/oslo.config: Log a warning when deprecated opts are used https://review.openstack.org/148020 2015-02-18T21:11:42 ah ok dhellmann 2015-02-18T21:12:25 dims__: I waffled, so I had to check, but yes, I did. 2015-02-18T21:13:09 u had a waffle 2015-02-18T21:14:11 harlowja_: now I'm hungry 2015-02-18T21:14:15 me too 2015-02-18T21:14:16 lol 2015-02-18T21:15:53 bnemec: isn't an empty group name the same as DEFAULT? 2015-02-18T21:16:31 \o/ 2015-02-18T21:16:57 *** amrith is now known as _amrith_ 2015-02-18T21:17:52 dhellmann: I don't think so: https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L1538 2015-02-18T21:18:23 bnemec: I sure hope the default is None, then :-) 2015-02-18T21:19:52 dhellmann: Yeah, I was tempted to try it, but decided that was a rabbit hole too far. ;-) 2015-02-18T21:20:42 hi oslo. is this perhaps fallout from the namespace rename? 2015-02-18T21:20:48 https://www.irccloud.com/pastebin/hb74zcUv 2015-02-18T21:22:28 dougwig: maybe? is that code in octavia? 2015-02-18T21:22:56 Merged openstack/oslo-specs: Add revision history sections https://review.openstack.org/152618 2015-02-18T21:23:04 dougwig: it looks like maybe someone went a little too far with a sed conversion of oslo.config to oslo_config 2015-02-18T21:23:08 yes. i've never touched that except via oslo-incubator/update.py, and it seems that oslo_config got text replaced to oslo.config somehow 2015-02-18T21:23:16 oh, hmm 2015-02-18T21:23:22 maybe we need to fix up update.py then 2015-02-18T21:24:31 Merged openstack/oslo-specs: Add adoption timeline to incubator policy https://review.openstack.org/153682 2015-02-18T21:28:28 dougwig: I can reproduce that when I run update.py. I'm looking at it. 2015-02-18T21:28:38 dougwig: have you already filed a bug? 2015-02-18T21:32:50 *** andreykurilin_ has joined #openstack-oslo 2015-02-18T21:34:39 Dan Krause proposed openstack/taskflow: add get_flow_details and get_atom_details to all backends https://review.openstack.org/157153 2015-02-18T21:34:53 Eric Brown proposed openstack/oslo.vmware: Resolve warnings from Bandit https://review.openstack.org/156889 2015-02-18T21:38:12 bnemec: when you say “We have multiple lines of communication available that don't require annoying all our users with unnecessary log messages.”, whats the line of communication for this kind of thing, the code doing something they should ideally correct for? how does an INFO message convey that this particular condition is less than ideal ? 2015-02-18T21:38:40 zzzeek: /join openstack-cinder? 2015-02-18T21:38:51 Or open a bug. Or send something to the list tagged [cinder] 2015-02-18T21:39:12 bnemec: OK so, I should remove this patch then, and instead have it raise an error 2015-02-18T21:39:13 ? 2015-02-18T21:39:38 bsically youre saying , there’s no such class of behavior such as, “this is less than ideal but we can correct for it" 2015-02-18T21:39:50 dhellmann: not yet. want one? 2015-02-18T21:40:17 zzzeek: Right, we've corrected for it. We don't need to log a warning that's going to cause a bunch of operators to have to chase down the reason for it. 2015-02-18T21:40:19 dougwig: yeah, if you have a minute. I think I have a fix for you but need to run some tests. 2015-02-18T21:40:28 Not when we can open a bug against cinder and say "fix this". 2015-02-18T21:40:31 ok, standby, meatspace intruding. 2015-02-18T21:40:32 bnemec: what if we are correcting for many things that affect performance negatively? 2015-02-18T21:40:43 bnemec: we just keep stuffing corrections in, hiding them, and thats that ? 2015-02-18T21:40:58 zzzeek: Tell the people who can fix it, not the operators. 2015-02-18T21:41:01 That's all I'm saying. 2015-02-18T21:41:13 bnemec: and they say, “Theres no warning, theres nothing we can see, so we dont care” 2015-02-18T21:41:13 * dhellmann pictures a side of beef barging into dougwig's office 2015-02-18T21:41:21 bnemec: also what about warnings.warn() ? 2015-02-18T21:41:37 bnemec: sqlalhcemy has a crapload of warnings.warn() in it, all kinds of htings that might work the way someone is using them, but should be corrected 2015-02-18T21:41:42 those should all be…hidden ? 2015-02-18T21:41:49 I'd be fine with warnings. They don't go to the logs by default. 2015-02-18T21:41:56 bnemec: OK so can i cahgne this to warnings.warn() ? 2015-02-18T21:42:03 bnemec: because it really isnt INFO and it really hsouldnt be hidden 2015-02-18T21:42:27 zzzeek: That's why I'm suggesting debug. That's the developer-oriented log level. 2015-02-18T21:42:33 But yeah, I'm fine with the warnings module. 2015-02-18T21:42:40 That seems developer oriented as well. 2015-02-18T21:42:45 bnemec: who ever reads DEBUG, and if they did, how do they tell things that are “normal” from things that are errors? 2015-02-18T21:45:58 sdague: ^Just FYI, it seems we could use some more clarification on the target audience in the logging spec. 2015-02-18T21:46:19 We seem to have a difference of interpretation. 2015-02-18T21:47:08 can someone summarize the question? 2015-02-18T21:47:13 bnemec: I really dont care waht the interpretation is but that spec doesnt seem to make any room for “the code is doing a bad thing the developers should fix”. if it shoudl be DEBUG with a special prefix, great. whats the prefix ? 2015-02-18T21:47:26 sdague: “the code is doing a bad thing that we can recover from, but the develoipers should fix it" 2015-02-18T21:47:28 sdague: how to log ? 2015-02-18T21:47:49 Full discussion here: https://review.openstack.org/#/c/156725/2/oslo_db/sqlalchemy/session.py 2015-02-18T21:48:08 which developers are the target? 2015-02-18T21:48:18 sdague: poeple who commit to cinder, for example 2015-02-18T21:48:30 sdague: someone who would make it so that they call engine.dispose() before launching a child process 2015-02-18T21:48:46 well, wihtin the child proc. 2015-02-18T21:49:18 warnings.warn() might be the most appropriate place 2015-02-18T21:49:32 you are throwing an exception already though right? Isn't that the "don't do this thing" indication to developers 2015-02-18T21:49:38 but warnings.warn() has the major issue that you cant put dynamic inforamtion in it without the chance of leaking memory 2015-02-18T21:49:57 dhellmann: lol 2015-02-18T21:49:57 sdague: no, we can recover from it and continue. i can make it just throw instead, but that would be more disruptive 2015-02-18T21:50:32 bnemec: in this specific case, we also could decide that this behavior is normal, i might be leaning towards that anyway. but im more curious about this warnings thing in general 2015-02-18T21:51:06 zzzeek: Yeah, it's a good discussion. 2015-02-18T21:51:12 zzzeek: yeh, so here's the thing we've discovered. developers basically don't read the logs. 2015-02-18T21:51:25 so it's not actually developer communication 2015-02-18T21:51:40 sdague: well this is the thing with warnings, people complain about them, so they get noticed 2015-02-18T21:51:47 what it actually turns into is "dear god, my cloud is busted" 2015-02-18T21:51:52 sdague: sure 2015-02-18T21:51:56 sdague: no middle ground huh 2015-02-18T21:52:03 Also, I actually think this behavior is broken anyway. Who knows what other file descriptors they've accidentally passed to child processes? 2015-02-18T21:52:25 Maybe we could add a flag to make this fatal in the gate? 2015-02-18T21:52:27 bnemec: OK…how would they guard against that across-the-board ? 2015-02-18T21:52:32 That would get noticed by devs. :-) 2015-02-18T21:52:47 bnemec: yeh, that's why I was wondering about the raise 2015-02-18T21:53:04 dhellmann: https://bugs.launchpad.net/oslo-incubator/+bug/1423361 2015-02-18T21:53:04 Launchpad bug 1423361 in oslo-incubator "incubator update.py converts oslo_config in cache module method name" [Undecided,New] 2015-02-18T21:53:04 zzzeek: Fork their workers before doing things like opening files and db connections? 2015-02-18T21:53:17 because that seems to be the thing that we want to do, figure out how to signal the right audience to address it 2015-02-18T21:53:20 this is really an odd one. these are database conecvtions, they’re buried in a connection pool people dont realize is there, and the patch easily just corrects for it by reopening the first time it hits in the child proc 2015-02-18T21:53:54 bnemec: OK….but this seems to be some kind of ad-hoc process spawning thing that oslo.concurrency is doing 2015-02-18T21:53:59 bnemec: it looks to be designed to work this way 2015-02-18T21:54:23 bnemec: these arent long running processes 2015-02-18T21:54:31 zzzeek: Well, processutils was more intended for doing things like running iptables from a service. 2015-02-18T21:54:53 bnemec: OK so, I can make this jsut raise. want me to do that? 2015-02-18T21:54:59 bnemec: then cinder has to fix it on their end 2015-02-18T21:55:11 bnemec: but that means, new oslo.db comes out, every app doing this somehwere breaks 2015-02-18T21:55:20 zzzeek: Yeah, exactly. 2015-02-18T21:55:26 dhellmann: are you watching this convo 2015-02-18T21:55:31 If we want this to be fatal, I think it has to be _only_ in the gate. 2015-02-18T21:55:51 Like the fatal deprecations thing in versionutils. 2015-02-18T21:55:52 bnemec: OK and if we’re not in the gate…then what ? silent recovery ? 2015-02-18T21:56:00 * bnemec brings the conversation full circle ;-) 2015-02-18T21:56:05 bnemec: so, honestly, I personally think having the library not try to recover is the right philosophical thing to do 2015-02-18T21:56:28 because I actually see issues like that pop up a bunch 2015-02-18T21:56:37 like the embedded retry in execute 2015-02-18T21:56:40 I actually agree. This will also break horribly under py3. 2015-02-18T21:56:55 *** e0ne has quit IRC 2015-02-18T21:57:00 Unfortunately in py2 it's an intermittent failure, apparently. 2015-02-18T21:57:02 bnemec: b.c. just any open FD will raise immediately ? 2015-02-18T21:57:15 bnemec: if i have a proces, and i open an FD, then say fork(), it breaks immediately in py3? 2015-02-18T21:57:19 zzzeek: Because open fds get closed by default on forks. 2015-02-18T21:57:30 py3 sets O_CLOEXEC on file descriptors 2015-02-18T21:57:34 bnemec: this recovery I have will work fine in py3k 2015-02-18T21:57:38 bnemec: the descriptor is never touched 2015-02-18T21:58:39 zzzeek: Right, because you detect the problem and reopen it? 2015-02-18T21:59:13 bnemec: i detect that the container for the conection was copied from a parent proc, i throw away the descriptor immediately and yes it has the invalidate flag set so it reopens immediately 2015-02-18T21:59:57 zzzeek: if you do decide to leave the message in here, I'd get pretty explicit it, and figure out the process name you are running on and say "this is a bug with project XXXX, please file a bug at ... url" 2015-02-18T22:00:07 sdague: OK what log level 2015-02-18T22:00:23 if you get that explicit, WARN seems fine to me 2015-02-18T22:00:28 DOH @ 2015-02-18T22:00:32 heh 2015-02-18T22:00:40 sdague: well thats waht the whole argument was about :) 2015-02-18T22:00:44 * dims__ gets some popcorn :) 2015-02-18T22:00:54 * dhellmann reads scrollback 2015-02-18T22:01:02 * zzzeek has no opinion! does not care! 2015-02-18T22:01:13 zzzeek: right, but in order to do that, you need to really give the ops a straight line to the right dev place to complain 2015-02-18T22:01:33 sdague: OK do you want to just make a comment on the review then, https://review.openstack.org/#/c/156725/2/oslo_db/sqlalchemy/session.py 2015-02-18T22:01:52 because in it's current state as an op I'm going to look at it, not really understand if it's an issue, see that it happens a bunch, then whitelist it in my grep filters for not real issues 2015-02-18T22:01:54 its between you and bnemec what you want to do, he was opposed to WARN 2015-02-18T22:02:03 Yeah, we did that before and got a lot of complaints. 2015-02-18T22:02:17 bnemec: where did you used to do it? 2015-02-18T22:02:23 "Why are you warning me about this thing I can't fix?" 2015-02-18T22:02:33 oh, yeh, that, agreed 2015-02-18T22:02:53 you know what woudl be great. we decide what we want here. then we *update that spec* to be clear about this 2015-02-18T22:02:54 if you are going to flag it that high you need to provide a path to fixing it 2015-02-18T22:03:01 That was the traditional sql mode thing, and we basically just turned it on by default and prayed. 2015-02-18T22:03:29 We got less complaints about that than the warning, oddly enough. 2015-02-18T22:03:36 Apparently we were already pretty traditional-safe. 2015-02-18T22:03:51 bnemec: I know I bitched about that one :) 2015-02-18T22:04:02 * sdague waiting for dhellmann to weigh in 2015-02-18T22:04:04 Heh 2015-02-18T22:04:10 * dhellmann has almost caught up 2015-02-18T22:04:28 * dhellmann wonders why walking away at 4:55 always results in so much scrollback 2015-02-18T22:04:35 dhellmann, harlowja_ : i wrote a new version of my asyncio spec, because i think that many people misunderstood the purpose of the spec. https://review.openstack.org/#/c/153298/3/specs/asyncio.rst 2015-02-18T22:04:53 sorry, each version of the spec is longer than the previous spec! 2015-02-18T22:05:15 sdague, bnemec, zzzeek : so the options are log a message intended for devs, use warnings.warn, or throw an exception? 2015-02-18T22:05:20 the most interesting change is a new section on "race conditions" 2015-02-18T22:05:32 dhellmann: or just consider this particular thing as “OK” 2015-02-18T22:05:40 dhellmann: I think the options are 2015-02-18T22:05:43 zzzeek: that's the "let the library recover" option? 2015-02-18T22:05:53 1) don't recover from the error, put the burden on the caller 2015-02-18T22:06:01 dhellmann: yes. 2015-02-18T22:06:30 harlowja_: ah yes, monkey-patched threading.RLock may work in eventlet 2015-02-18T22:06:38 * dhellmann waits for sdague to finish enumerating 2015-02-18T22:06:42 2) log something and recover 2015-02-18T22:06:44 if 2) 2015-02-18T22:07:18 then why are we logging? if it's because we are expecting someone to change something, we need to make sure it gets to the right audience 2015-02-18T22:07:40 WARN "Invalidating connection detected as shared to new... " 2015-02-18T22:07:42 what about logging and not recovering? then the log message will be in the job that fails 2015-02-18T22:07:54 er, in the output for the job that fails 2015-02-18T22:07:55 is mostly going to just confuse people reading the message 2015-02-18T22:08:24 Also, it sounds like this happens every run, but only intermittently causes failures. 2015-02-18T22:08:30 yeah, let's assume we can come up with a message that isn't confusing to developers if we're going to log it and figure out the wording separately 2015-02-18T22:08:46 bnemec: well for sure, it causes random race conditions 2015-02-18T22:09:09 bnemec: but yes its on every run if certain options in cinder are turned on 2015-02-18T22:09:23 bnemec: that make it go off and use oslo.concurrency when it starts up 2015-02-18T22:09:24 Yeah, so we need all or nothing. Either recover, or fail immediately. 2015-02-18T22:09:47 OK well, that gerrit has “the code”. you all can make it do whatever :) 2015-02-18T22:09:49 how about adding just the log message for now and cutting a release so we can measure how often it happens in projects other than cinder 2015-02-18T22:09:57 dhellmann: OK. What level ? :) 2015-02-18T22:10:01 that way we can tell whether it's safe to make it fail immediately 2015-02-18T22:10:01 so I think that's actually the crux of the problem really 2015-02-18T22:10:11 zzzeek: what level do we log at for developers in gate jobs? 2015-02-18T22:10:18 dhellmann: shrugs ? 2015-02-18T22:10:23 debug, I would think 2015-02-18T22:10:27 because it's like "oh, this thing is wrong, it's your problem, except, I'll fix it for you" 2015-02-18T22:10:27 dhellmann: OK ! 2015-02-18T22:10:30 I don't know if we actually turn the logging up that high 2015-02-18T22:10:31 dhellmann: debug it is 2015-02-18T22:10:39 yes, we use DEBUG in the gate 2015-02-18T22:10:48 dhellmann: any wording in the DEBUG that make it clear that “hey, you should probably see why this is happening” ? 2015-02-18T22:10:51 dhellmann: like a prefix ? 2015-02-18T22:10:54 It's not safe to make it fail immediately. That will break all existing Cinder code if they happen to grab the new oslo.db. 2015-02-18T22:11:05 and …can this be added to the spec ?? to avoid 30 min IRC discussion ? 2015-02-18T22:11:13 sdague: right, I completely agree that if we could we'd just make it fail. However, if we know this is going to break things and piss people off, we should go a little more slowly. 2015-02-18T22:11:30 zzzeek: I believe the spec already says that developer messages should be logged at debug 2015-02-18T22:11:47 dhellmann: OK but how do we distinguish “normal operations” DEBUG from “HEY! this is BAD!” DEBUG ? 2015-02-18T22:12:04 dhellmann: just need the lingo 2015-02-18T22:12:11 dhellmann: if someoen wants to just pop it onto the gerrit here 2015-02-18T22:12:15 zzzeek: and no, just write a clear message explaining what happened. Something like "This process forked while a database connection was open. That is likely to cause race conditions." 2015-02-18T22:12:30 Sort of serious suggestion: LOG.debug('WARNING: ...') 2015-02-18T22:12:32 zzzeek: operators should not need to read debug messages 2015-02-18T22:12:35 dhellmann: OK….but we recovered, so there wont be any race conditions 2015-02-18T22:12:37 dhellmann: that is fine 2015-02-18T22:12:41 zzzeek: I'd actually prefix with "BUG: " 2015-02-18T22:12:48 dhellmann: OK, so verbiage here is….ah “BUG:”, sure 2015-02-18T22:12:48 zzzeek: I'm suggesting that you remove the fix and leave the race condition for now 2015-02-18T22:12:57 and I think it's a good point, and we should add to the spec 2015-02-18T22:13:07 dhellmann: wow really. UM, wow. OK 2015-02-18T22:13:09 so JUST log, and let's release a version of the lib that does that and see what we get in logstash 2015-02-18T22:13:09 I'll throw a patch up tomorrow 2015-02-18T22:13:18 zzzeek: hold your horses 2015-02-18T22:13:28 hey i have to pick up my wife, takes 20 minutes can you folks update me via gerrit / etc? 2015-02-18T22:13:28 I suggest a 2 phase approach. 2015-02-18T22:13:33 Couldn't we fix the race and log the message to see what we get in logstash? 2015-02-18T22:13:43 or we can all talk at once 2015-02-18T22:13:53 * zzzeek has to bbl, sorry 2015-02-18T22:13:59 phase 1: log only the condition that is the problem, without fixing it 2015-02-18T22:14:22 phase 2: based on how many apps we find with that error, decide whether to fix it by recovering or introduce an exception 2015-02-18T22:14:32 "zzzeek: Because open fds get closed by default on forks." *_CLOEXEC only ask to close the fd on exec(), not at fork 2015-02-18T22:14:46 if we introduce an exception, then we go to the app(s) who will break and let them know so they can fix their code before we do that 2015-02-18T22:14:48 dhellmann: debug is not indexed in logstash, so that plan isn't going to work 2015-02-18T22:15:00 ok, well, then we'll have to log at info or something so we can measure it 2015-02-18T22:15:08 we can change it to debug as part of phase 2 2015-02-18T22:15:32 the gate is only one access pattern though 2015-02-18T22:15:52 true 2015-02-18T22:15:53 so I don't think that's enough data to make the final decision on what to do, you should figure out where you'd like to be, and head there. 2015-02-18T22:16:17 ok, well, it seems we want the error. I just don't want to deal with the fallout of introducing a fatal error like that knowingly. 2015-02-18T22:16:44 mailing list thread? make sure to cc ptls of known effected projects? 2015-02-18T22:17:09 we don't know which projects are affected, though, that's the point 2015-02-18T22:17:13 so far we only know about cinder 2015-02-18T22:17:14 Cinder is the only known affected right now. 2015-02-18T22:17:32 if it's only cinder, why is this even an oslo bug? 2015-02-18T22:18:10 my guess is that we'll find other projects 2015-02-18T22:18:22 can we find them through looking at code instead of running gate jobs? 2015-02-18T22:18:27 dhellmann: i'd suspect neutron as well 2015-02-18T22:18:29 It's not really an oslo bug, but it ends up looking like one. 2015-02-18T22:18:31 so if you expose on the ML list as a more general thread, maybe you'll get other people chiming in 2015-02-18T22:18:52 or paying attention, I could be overly optimistic there 2015-02-18T22:19:01 Probably :-) 2015-02-18T22:19:08 sdague: yeah, I'm a bit pessimistic about that but it's worth a try 2015-02-18T22:19:11 but that does seem like the cross dev community communication bus 2015-02-18T22:19:24 I'd like to at least agree on an approach to put in that email,t hough 2015-02-18T22:19:52 and frankly, I expect most people to want the faster horse instead of the car 2015-02-18T22:20:06 i.e., recover quietly and leave me alone 2015-02-18T22:20:22 Yeah, same thing with the monkey patching 2015-02-18T22:20:27 dhellmann: my vote would be on "don't recover from the error, put the burden on the caller" 2015-02-18T22:20:31 dhellmann: maybe, it ends up hiding a ton of misuse issues 2015-02-18T22:20:41 sdague: oh, I agree, I'm just making predictions 2015-02-18T22:20:50 zzzeek: "Cinder is spawning a subprocess using oslo.concurrency and is failing to call create_engine() local to that process" i don't understand. the child process inherits file descriptors? does the child process use these file descriptors? 2015-02-18T22:20:53 yeh, my inclinations is that libraries shouldn't do recovery like this 2015-02-18T22:21:07 +1 2015-02-18T22:21:08 haypo: yeah, the child process is reusing the open database connection 2015-02-18T22:21:20 sdague: right, it doesn't tend to work out well doing it at that low a level 2015-02-18T22:21:39 dhellmann: how does the child process pick the file descriptor if it is now aware that it inherited unwanted file descriptors? 2015-02-18T22:21:41 bnemec: can you start the mailing list thread? you have more background on this than I do 2015-02-18T22:21:44 Ooh, what about a deprecation period for the current behavior? 2015-02-18T22:22:01 haypo: https://review.openstack.org/#/c/156725/2/oslo_db/sqlalchemy/session.py 2015-02-18T22:22:09 We start by logging the warning, then next cycle or whenever we make this fail fast? 2015-02-18T22:22:23 bnemec: I think we want the bug fixed this cycle, though 2015-02-18T22:22:39 bnemec: 1 release of oslo.db, not one full cycle 2015-02-18T22:22:39 We can do the current workaround during the deprecation period. 2015-02-18T22:22:59 dims__: I'd want a full cycle to give people a chance to find and fix all the problems. 2015-02-18T22:24:22 * bnemec hears crickets 2015-02-18T22:24:39 I'm not keen on a long grace period for this fix. 2015-02-18T22:25:01 bnemec: does not help, if people don't heed the warning, then they will never change 2015-02-18T22:25:11 they will change only when we break them 2015-02-18T22:25:11 I really do want more info about how many projects have the issue, though, because if it's just one then the answer becomes much simpler. 2015-02-18T22:25:16 right dhellmann 2015-02-18T22:25:20 dims__: Right, but oslo releases can happen within a couple of weeks. 2015-02-18T22:25:45 bnemec: we can hold the line on oslo.db 2015-02-18T22:26:13 Yeah, until we merge another critical bug fix. :-) 2015-02-18T22:26:26 haha 2015-02-18T22:27:01 I also don't want to set the precedent that we're going to hold up releases for an extended period of time for situations like this. 2015-02-18T22:28:33 *** jecarey has quit IRC 2015-02-18T22:28:36 I guess my thing is that I don't think we can break people without any warning, even if Cinder does turn out to be the only one. 2015-02-18T22:29:33 bnemec: did you agree to start that email thread? 2015-02-18T22:30:01 bnemec: if cinder is the only case where this happens, then we don't need to make any changes other than the log message in the db library 2015-02-18T22:30:05 bnemec: mine is that we need to get this under control before we ship out kilo even if we have to break them in the interim, dont want known issues like this in production 2015-02-18T22:30:12 dhellmann: I guess so 2015-02-18T22:30:29 we could, but we don't *need* to 2015-02-18T22:30:56 *** achanda has quit IRC 2015-02-18T22:31:46 I'm still not clear that we have a consensus on an approach though. Are we going to start with only the log message? 2015-02-18T22:31:50 * harlowja_ man this is a lively conversation, lol 2015-02-18T22:32:03 :) harlowja_ 2015-02-18T22:32:15 Logging. Serious business. 2015-02-18T22:32:23 haha bnemec 2015-02-18T22:32:32 ;) 2015-02-18T22:32:54 dhellmann: i read the change but i don't understand how a connection record can be inherited from a parent to a child process when the subprocess module 2015-02-18T22:33:08 bnemec: that's what we need the thread to figure out, I guess. 2015-02-18T22:33:20 haypo i think its because sqlalchemy engines are pooled (?) 2015-02-18T22:33:31 *** noelbk has quit IRC 2015-02-18T22:33:31 haypo: I'm not sure they are using subprocess(), but you'd have to ask zzzeek because he's the one that found the problem 2015-02-18T22:33:53 dhellmann: Okay, I'll just try to list the options we've discussed then. 2015-02-18T22:34:08 bnemec: yeah 2015-02-18T22:34:11 if its just subprocess why not just off all open file descriptors? 2015-02-18T22:34:12 * bnemec puts on his flame retardant suit just in case he gets any of them wrong ;) 2015-02-18T22:34:42 haypo: this thread may have more detail: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057097.html 2015-02-18T22:35:32 * harlowja_ goes back to being quiet (to much detail i dont know about, ha) 2015-02-18T22:35:37 ^ be quiet josh, ha 2015-02-18T22:35:38 harlowja_: the app should be doing that, but isn't 2015-02-18T22:35:47 that's the actual bug 2015-02-18T22:35:52 dhellmann: i guess that cinder uses fork, not subprocess.Popen 2015-02-18T22:36:03 dhellmann: thanks for the links 2015-02-18T22:36:05 haypo: that may be the case 2015-02-18T22:36:52 haypo: Specifically this http://lists.openstack.org/pipermail/openstack-dev/2015-February/057184.html 2015-02-18T22:37:02 The eventlet stuff at the beginning of the thread seems to have been a red herring. 2015-02-18T22:37:59 i've seen stuff like https://github.com/thesharp/daemonize/blob/master/daemonize.py#L116 (wonder why we aren't doing something similar) 2015-02-18T22:39:11 or * https://github.com/BlueDragonX/detach/blob/master/detach.py#L61 (and many others) 2015-02-18T22:39:24 Doug Hellmann proposed openstack/oslo-incubator: Fix the regex for turning project.lib back to oslo.lib https://review.openstack.org/157178 2015-02-18T22:40:26 ^ all little fork 'helpers' 2015-02-18T22:40:37 *** noelbk has joined #openstack-oslo 2015-02-18T22:42:05 oh, i remember with sileht explained me that the Service class is based on fork() and does crazy things to workaround fork() issues 2015-02-18T22:42:45 ya, but i wonder why we didn't follow similar patterns to other daemonizeing things i've seen (that afaik all by default close fds) 2015-02-18T22:42:51 *** mriedem1 has quit IRC 2015-02-18T22:42:57 maybe something idk (or some history there) 2015-02-18T22:43:14 harlowja_: Service design is completly crazy :) 2015-02-18T22:43:23 haypo no disagreement there :-P 2015-02-18T22:43:32 some crazy history is highly likely, lol 2015-02-18T22:44:08 since service.py is still in the incubator, it's not too late to fix it ;) 2015-02-18T22:44:24 well, aren't there cases where forking and keeping the file descriptors open is valid? 2015-02-18T22:44:41 dhellmann probably, but maybe that should be the exception and not the default? 2015-02-18T22:45:00 harlowja_: quite possibly 2015-02-18T22:45:24 although i do wonder how cinder is executing code before service.py does stuff 2015-02-18T22:45:30 bnemec: fixing the service module to close file descriptors after a fork by default is another approach we could take. That would let apps sync the fix when they are ready, rather than when we release oslo.db. 2015-02-18T22:45:39 most usages i've seen the service.py stuff is the first thing to run (and nothing else runs before it) 2015-02-18T22:46:03 ya, shutoff all file-descriptors (add exceptions to this as needed) 2015-02-18T22:46:34 *** achanda has joined #openstack-oslo 2015-02-18T22:46:54 aw, sweet, we shoudl use this one 2015-02-18T22:46:55 https://github.com/Gamocosm/minecraft-server_wrapper/blob/master/daemon.py#L62 2015-02-18T22:47:07 and turn service.py into a minecraft server when its idle 2015-02-18T22:47:07 lol 2015-02-18T22:47:21 ^ joke... 2015-02-18T22:48:39 anyway, i contributed to https://bugs.launchpad.net/cinder/+bug/1417018 with my theory on fork 2015-02-18T22:48:39 Launchpad bug 1417018 in Cinder "Cinder encounters dbapi error: NoSuchColumnError: "Could not locate column in row for column '%(140070887032400 anon)s.volumes_id'"" [Undecided,New] - Assigned to Mike Bayer (zzzeek) 2015-02-18T22:48:44 *** crc32 has quit IRC 2015-02-18T22:49:10 dhellmann: Yeah, if we could fix it in the service code that would be nice. 2015-02-18T22:49:16 Especially since that hasn't graduated yet. 2015-02-18T22:49:42 right 2015-02-18T22:49:57 zzzeek doesn't say which process in cinder has the issue 2015-02-18T22:50:14 dhellmann: it has to do with ceph 2015-02-18T22:50:19 dhellmann: i had some log output in the email thread 2015-02-18T22:50:24 zzzeek: which service process was it? 2015-02-18T22:50:41 * harlowja_ trying to connect it to https://github.com/openstack/cinder/tree/master/cinder/cmd (and which one here) 2015-02-18T22:51:03 hmmm, i guess it could be https://github.com/openstack/cinder/blob/master/cinder/cmd/volume.py#L62 2015-02-18T22:51:14 *** yamahata_ has quit IRC 2015-02-18T22:51:21 dhellmann: CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LVM_SYSTEM_DIR=/etc/cinder vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix stack-volumes-lvm-1" 2015-02-18T22:51:26 'server = service.Service.create' activating different backends (that may be doing bad things) 2015-02-18T22:51:30 *** isq has joined #openstack-oslo 2015-02-18T22:51:51 rootwrap stuff all goes through subprocess right? 2015-02-18T22:51:56 *via execute() 2015-02-18T22:52:05 *via process utils execute 2015-02-18T22:52:18 harlowja_: the "CMD ..." log comes from execute() which uses subprocess.Popen with close_fds=True 2015-02-18T22:52:35 zzzeek: which process was calling that rootwrap command, I mean 2015-02-18T22:52:54 dhellmann: hard to say 2015-02-18T22:52:56 I'm trying to understand which daemon it is, so I can look at that daemon's startup code 2015-02-18T22:52:59 dhellmann: oh 2015-02-18T22:53:03 dhellmann: well its cinder-volume 2015-02-18T22:53:11 dhellmann: i have step-by-step in the bug report 2015-02-18T22:53:13 to reproduce 2015-02-18T22:54:13 zzzeek: the bug isn't linked from https://review.openstack.org/#/c/156725/2 is it in the email thread somewhere? 2015-02-18T22:56:21 dhellmann: oh. hm 2015-02-18T22:56:26 hmm, this looks like a fundamental issue with the way service works 2015-02-18T22:56:39 create() takes the name of a manager class, and the instantiates it 2015-02-18T22:56:48 dhellmann: its in one of my comments there its https://bugs.launchpad.net/cinder/+bug/1417018 2015-02-18T22:56:48 in __init__ 2015-02-18T22:56:48 Launchpad bug 1417018 in Cinder "Cinder encounters dbapi error: NoSuchColumnError: "Could not locate column in row for column '%(140070887032400 anon)s.volumes_id'"" [Undecided,New] - Assigned to Mike Bayer (zzzeek) 2015-02-18T22:57:05 zzzeek: next revision it would be good to add that to the commit message 2015-02-18T22:57:24 dhellmann: yeah i didnt know the system to add bugs from a different project? 2015-02-18T22:57:28 dhellmann: just full hyperlink ? 2015-02-18T22:57:43 zzzeek: no, just "Closes-Bug: #xxx" or whatever 2015-02-18T22:57:55 dhellmann: OK but that bug number is local to cinder, not oslo 2015-02-18T22:58:08 launchpad bugs can be associated with more than one project 2015-02-18T22:58:12 dhellmann: cross-project bug linking basically 2015-02-18T22:58:16 dhellmann: OK 2015-02-18T22:58:24 dhellmann: but..this is an oslo.db bug ? :) 2015-02-18T22:58:44 zzzeek: reload the bug report, now it's associated with both cinder and oslo.db 2015-02-18T22:58:59 dhellmann: yes. so this is an oslo.db bug also? 2015-02-18T22:59:07 dhellmann: menaing, “oslo.db does not guard against XYZ” ? 2015-02-18T22:59:25 zzzeek: the root cause should be investigated IMO 2015-02-18T22:59:29 zzzeek: no, not really. I think the right thing to do here is fix the service code so we close file descriptors before forking 2015-02-18T22:59:32 haypo: the root cause is exactly known 2015-02-18T22:59:43 zzzeek: ah? what is it? 2015-02-18T22:59:49 zzzeek: in general I don't like the libraries trying to recover from issues that came from elsewhere 2015-02-18T22:59:56 haypo: what I stated in the bug report 2015-02-18T23:00:08 *** openstackgerrit has quit IRC 2015-02-18T23:00:14 haypo: file descriptor is traveling to a new child process 2015-02-18T23:00:15 zzzeek: I don't think it's that subprocess call, though. 2015-02-18T23:00:24 zzzeek: see my comment: i don't undestand your analysis 2015-02-18T23:00:24 *** openstackgerrit has joined #openstack-oslo 2015-02-18T23:00:27 dhellmann: well if I make that call fail, then the bug goes away 2015-02-18T23:00:27 subprocess closes file descriptors 2015-02-18T23:00:43 haypo: why dont you run my step-by-step reproucition steps, pdb into it and let me know what you think ? 2015-02-18T23:01:04 dhellmann: does it close FDs that are open by native -C code like MySQLdb ? 2015-02-18T23:01:14 dhellmann: but actually i can reproduce it with pymysql also 2015-02-18T23:01:19 zzzeek: it's around midnight, i'm not supposed to work :) maybe tomorrow if you can help me to prepare the setup to reproduce the issue 2015-02-18T23:01:24 dhellmann: bnemec tells me subproces only closes FDs in python 3 2015-02-18T23:01:32 zzzeek: yes, close_fds=True closes all file descriptors 2015-02-18T23:01:34 haypo: the steps are all there, do you know how to run devstack ? 2015-02-18T23:01:51 zzzeek: sure, i have a devstack which is supposed to work :-) 2015-02-18T23:02:13 zzzeek: http://git.openstack.org/cgit/openstack/oslo.concurrency/tree/oslo_concurrency/processutils.py#n213 2015-02-18T23:02:20 zzzeek: oslo.concurrency sets explicitly close_fds to True, so it also close FDs on Python 2 2015-02-18T23:02:32 haypo: OK, so the bug does not exist, lets close it ! :) 2015-02-18T23:02:37 that's why i'm asking if the issue doesn't come from fork 2015-02-18T23:03:01 haypo: whats the other way for a linux proces to make a child process besides fork ? 2015-02-18T23:03:30 zzzeek: Popen uses fork+exec. the Service class only uses fork 2015-02-18T23:03:34 zzzeek: I suspect this is our real issue: http://git.openstack.org/cgit/openstack/oslo-incubator/tree/openstack/common/service.py#n308 2015-02-18T23:03:52 because the service class is set up with a database connection before the launcher forks 2015-02-18T23:03:55 dhellmann: i love fork so much 2015-02-18T23:04:08 * dhellmann should be putting a fork into his dinner soon 2015-02-18T23:04:30 * haypo prefers spoons 2015-02-18T23:04:41 zzzeek: for example, if cinder-volume starts multiple workers, they would *all* share that initial database connection 2015-02-18T23:04:46 * bnemec might be on a liquid diet after this discussion 2015-02-18T23:05:00 dhellmann: I only narrow it down to oslo.concurrency b.c. if i make that specific part of it fail, the issue goes away, and the error always occurs right after that log line in oslo.concurrency with “CMD” 2015-02-18T23:05:02 https://hg.python.org/releasing/2.7.9/file/753a8f457ddc/Lib/subprocess.py#l1181 (looks similar to what the other daemon code i've seen do) ; looks ok in 2.7 at least, ha 2015-02-18T23:05:33 if you all like, I can log the process id in oslo.concurrency there and prove its the same number in the error condition 2015-02-18T23:05:34 zzzeek: yeah, I believe you, I'm just not sure I understand why it happens there 2015-02-18T23:05:51 dhellmann: gonna say close_fds does not do waht we think it does in py2k or the flag is not set to True 2015-02-18T23:06:10 that would be interesting to see, too. Maybe cinder is passing false for that. 2015-02-18T23:06:22 dhellmann: "if cinder-volume starts multiple workers ..." do you mean the at application class is created once and then we get multiple processes using os.fork? 2015-02-18T23:06:26 It's not configurable. The only time it's false is on windows. 2015-02-18T23:06:29 zzzeek: how many workers does devstack create for cinder-volume with that configuration? 2015-02-18T23:06:38 dhellmann: that's completly insane. how is it possible that it worked before? :) 2015-02-18T23:06:43 dhellmann: haven’t analyzed that. seemed like it was doing “a bunch” 2015-02-18T23:07:35 https://hg.python.org/releasing/2.7.9/file/753a8f457ddc/Lib/subprocess.py#l1238 is all the stuff that happens in 2.7 in the child 2015-02-18T23:07:38 haypo: ah, I hadn't noticed that 2015-02-18T23:07:49 * zzzeek has just this one point. He ran the whole thing from scratch to see it happening, and he put the full steps to reprocue. so if folks want to confirm, please run the steps 2015-02-18T23:07:57 oops, that was for bnemec ^^ 2015-02-18T23:08:47 zzzeek: do you have any reports of this problem in projects other than cinder? 2015-02-18T23:08:53 dhellmann: nope 2015-02-18T23:08:54 *** pradk has quit IRC 2015-02-18T23:08:58 ok, that's something 2015-02-18T23:09:07 dhellmann: however, the symptoms are extremely obtuse and distant from the acvtual problem 2015-02-18T23:09:12 dhellmann: very occasional MySQL screwups 2015-02-18T23:09:22 dhellmann: a friend told me that after fork, the POSIX standard only allows a very low number of syscalls, something like read, write, exec 2015-02-18T23:09:29 so we outlined a few options, and bnemec is going to start a mailing list thread on it. I'm inclined to leave this as a cinder bug and not have the library recover from the problem. I know it's easy, but I don't want to hide issues like this. 2015-02-18T23:09:33 dhellmann: and in fact here its quite difficult to reproduce. I had to patch cinder to run the one query 100 times in a row to get the race to collide consistently 2015-02-18T23:09:40 I'm not sure if I want to throw an error or not 2015-02-18T23:09:44 dhellmann: fork is only supposed to be used to execute fork, that's all :) 2015-02-18T23:09:47 oops 2015-02-18T23:09:51 dhellmann: fork is only supposed to be used to call exec(), that's all :) 2015-02-18T23:09:55 maybe zzzeek your machine got bombarded by space rays when this happened? 2015-02-18T23:09:59 and it flipped a bit 2015-02-18T23:10:08 harlowja_: and the two other folks who reported the issue, yup 2015-02-18T23:10:13 haypo: that's not true, processes fork() to daemonize themselves all the time 2015-02-18T23:10:17 space rays could of hit them too zzzeek 2015-02-18T23:10:29 *** daniel3_ has quit IRC 2015-02-18T23:11:38 zzzeek: for the record, I believe there is a problem and that it's related to forking somehow and that your patch would "fix it" -- I'm just not sure that's the best solution long-term. I'd like to understand why this is only coming up in cinder, for example. That's why I suggested that we start by logging the situation and measure where we see it and how often. 2015-02-18T23:11:47 dhellmann: ah maybe. but instanciate the application after the fork would avoid many issues 2015-02-18T23:12:04 dhellmann: Do we still want the thread now, or do we want to look into the Service thing? Because if we can fix it in Service then it may affect my preferences. 2015-02-18T23:12:12 haypo: yes, I agree. I'll have to try to wrap my head around the service code better to see if that's possible. 2015-02-18T23:12:20 its probably a good idea to have service.py shutoff fds right? 2015-02-18T23:12:33 dhellmann: okey doke, as I’ve maintained since….yesterday, I can make it A. raise B. do nothing C. correct for it, D. choose not to decide but we stil have made a choice, so... 2015-02-18T23:12:40 bnemec: let's mention the service option, but I think that's going to be a longer-term fix no matter what. The folks that wrote that code aren't around any more 2015-02-18T23:12:40 *since that seems to match the behavior of all other libraries that fork (and subprocess itself) 2015-02-18T23:12:45 * zzzeek does not care! 2015-02-18T23:13:12 zzzeek u should do Z 2015-02-18T23:13:16 option Z please, thx 2015-02-18T23:13:25 * zzzeek does the happy this-is-not-his-code dance 2015-02-18T23:13:26 zzzeek: I'm leaning towards B' (do nothing other than log a debug message) but I just don't want you to get frustrated by us taking some time to understand the issue 2015-02-18T23:13:28 * harlowja_ runs aawy 2015-02-18T23:14:00 dhellmann: i dont need this issue fixed. a coworker asked me to look at it, and it was totalyl mysterious and disturbing, like the connection pool was broke or something, so im jstu super glad its nothing on my end :) 2015-02-18T23:14:16 zzzeek: cool, like I said, just making sure the delay isn't annoying :-) 2015-02-18T23:14:25 id encourgage everyone to jump on the bug report and do their worst 2015-02-18T23:14:27 FYI I'm interested to help to redesign the service class. but it may be difficult to change it in stable branches 2015-02-18T23:15:10 so zzzeek specific issue maybe need other changes in stable branches 2015-02-18T23:15:35 haypo: if other proejcts are forking and not disposing their engines and we decide not to correct for it in the pool, then yes 2015-02-18T23:15:37 we may need a new service class which has a sane design, to not break applications using the old class 2015-02-18T23:16:31 zzzeek: i don't know what is the best place to workaround the fork issue, i don't know enough how SQLAchemy is used 2015-02-18T23:17:33 i have to go, bye 2015-02-18T23:18:45 it's time for me to head out, too 2015-02-18T23:19:04 Good night everyone. 2015-02-18T23:23:20 i'm still around (now all lonely) :( 2015-02-18T23:24:29 *** noelbk has quit IRC 2015-02-18T23:28:56 *** noelbk has joined #openstack-oslo 2015-02-18T23:49:35 *** dims_ has joined #openstack-oslo 2015-02-18T23:50:09 *** andreykurilin_ has quit IRC 2015-02-18T23:51:05 *** dims__ has quit IRC 2015-02-18T23:51:29 *** vigneshvar has quit IRC