Wednesday, 2018-08-08

*** namnh has joined #openstack-nova00:00
*** namnh has quit IRC00:01
*** ccamacho has quit IRC00:10
openstackgerritMatt Riedemann proposed openstack/nova master: Fix bad links for admin-guide  https://review.openstack.org/58025900:12
openstackgerritMerged openstack/os-vif master: Update reno for stable/rocky  https://review.openstack.org/58610600:24
mriedemi have no idea how this can be happening https://review.openstack.org/#/c/588455/00:26
mriedemif only we had some multi-node pci ci00:26
openstackgerritMerged openstack/nova stable/queens: Add recreate test for RT.stats bug 1784705  https://review.openstack.org/58792100:32
openstackbug 1784705 in OpenStack Compute (nova) queens "ResourceTracker.stats can leak across multiple ironic nodes" [High,In progress] https://launchpad.net/bugs/1784705 - Assigned to Matt Riedemann (mriedem)00:32
*** ccamacho has joined #openstack-nova00:33
*** gyee has quit IRC00:38
*** Nel1x has joined #openstack-nova00:45
*** hongbin has joined #openstack-nova00:53
*** licanwei has joined #openstack-nova01:01
*** mriedem has quit IRC01:06
*** hongbin has quit IRC01:13
*** hongbin has joined #openstack-nova01:13
openstackgerritzhufl proposed openstack/nova master: xx_instance_type_id in list_migrations should be integer  https://review.openstack.org/58848101:23
*** mhen has quit IRC01:26
*** mhen has joined #openstack-nova01:27
*** bbbbzhao_ has joined #openstack-nova01:38
*** Dinesh_Bhor has joined #openstack-nova01:50
*** mrsoul has joined #openstack-nova01:58
alex_xuefried: yea, I'm working on it https://review.openstack.org/569011, we will discuss this in the PTG02:08
*** gbarros has joined #openstack-nova02:35
*** yonglihe has joined #openstack-nova02:42
*** Dinesh_Bhor has quit IRC02:54
*** dave-mccowan has quit IRC02:57
*** dklyle has quit IRC02:58
*** Nel1x has quit IRC02:58
*** zhurong has joined #openstack-nova03:01
*** Dinesh_Bhor has joined #openstack-nova03:04
*** lei-zh has joined #openstack-nova03:08
*** itlinux_ has quit IRC03:13
*** psachin has joined #openstack-nova03:14
openstackgerritMerged openstack/nova stable/queens: Make ResourceTracker.stats node-specific  https://review.openstack.org/58797603:18
openstackgerritMerged openstack/nova master: Fix none-ascii char in doc  https://review.openstack.org/58842203:18
openstackgerritMerged openstack/nova master: Add microversion info in the os-server-groups API samples  https://review.openstack.org/58900603:19
openstackgerritMerged openstack/nova master: api-ref: Add descriptions for rebuild  https://review.openstack.org/58893103:19
*** gbarros has quit IRC03:25
*** gbarros has joined #openstack-nova03:36
*** gbarros has quit IRC03:49
*** hongbin has quit IRC03:51
*** erlon has quit IRC03:53
*** links has joined #openstack-nova03:56
*** jaosorior has joined #openstack-nova03:59
*** udesale has joined #openstack-nova04:00
*** lei-zh has quit IRC04:03
*** ratailor has joined #openstack-nova04:06
*** jaypipes has quit IRC04:11
*** jaypipes has joined #openstack-nova04:11
*** tetsuro has joined #openstack-nova04:18
*** zhurong has quit IRC04:20
*** jenglisch has quit IRC04:27
*** janki has joined #openstack-nova04:34
*** vivsoni has quit IRC04:54
*** vivsoni_ has joined #openstack-nova04:55
*** jarod_ has quit IRC05:07
*** takashin has joined #openstack-nova05:11
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] api-ref: add description for 1.29  https://review.openstack.org/58940705:18
*** stakeda has joined #openstack-nova05:25
*** jaosorior has quit IRC05:27
*** hoonetorg has quit IRC05:31
*** ccamacho has quit IRC05:32
*** Bhujay has joined #openstack-nova05:35
*** udesale has quit IRC05:42
*** hoonetorg has joined #openstack-nova05:52
*** zhurong has joined #openstack-nova05:57
gmannAPI office hour time06:01
gmann#startmeeting nova api06:02
openstackMeeting started Wed Aug  8 06:02:14 2018 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.06:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.06:02
openstackThe meeting name has been set to 'nova_api'06:02
gmannwho all here today?06:02
gmannPING List: gmann, alex_xu06:02
alex_xuo/06:03
*** hemna_ has quit IRC06:03
*** nicolasbock has joined #openstack-nova06:03
gmannalex_xu: let's start06:04
gmann#topic Priorities06:04
gmann#link https://etherpad.openstack.org/p/rocky-nova-priorities-tracking06:04
gmannit will be quick update as Rocky FF already done06:04
*** dpawlik has joined #openstack-nova06:05
gmannthere are 4 pending item from this which we  need to continue in Stein06:05
gmannand after PTG we will update the priority for stein06:05
*** hoonetorg has quit IRC06:06
gmanni will continue the policy granular things as default roles are done in keystone now06:06
gmann#link this one https://review.openstack.org/#/c/547850/06:06
gmannalex_xu: anything you have on this topic ?06:07
alex_xugmann: why that depends on default roles in keystone?06:08
gmannthat is do to the policy granular along with some meaningful defaults roles06:08
*** dpawlik has quit IRC06:09
alex_xuok, i see06:09
gmannfor example, in this spec i am proposing to make GET policy separate than POST/PUT so new GET policy should go with new default like reader, auditor etc06:09
*** dpawlik has joined #openstack-nova06:09
gmannwhich make sense06:09
gmannotherwsie we end up doing policy change twice06:09
alex_xuok06:11
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Use oslo_utils.uuidutils  https://review.openstack.org/58971706:11
gmannlet's move next?06:11
alex_xuyea, nothing from me06:12
gmann#topic Bug Triage/Discussion06:12
gmann#link https://etherpad.openstack.org/p/nova-api-weekly-bug-report06:12
gmannupdated this week report06:12
gmannthere was 1 new bug which we can discuss here and keypair in-use bug06:13
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Use uuidutils of oslo.utils  https://review.openstack.org/58971706:13
gmann#link https://bugs.launchpad.net/nova/+bug/164445706:13
openstackLaunchpad bug 1644457 in OpenStack Compute (nova) "keypair quota error" [Medium,Confirmed] - Assigned to Vishakha Agarwal (vishakha.agarwal)06:13
gmannalex_xu: i agree with doc only update as of now on keypair things, and later if we need we should go with barbican integration as matt mentioned via sdague mail06:14
gmanni just updated the comment there.06:14
alex_xugmann: yea, that the thing we can do now06:14
gmannalex_xu: +106:15
gmannvishakha: ^^06:15
gmannnext bug i want to discuss is new one show up -06:15
gmann#link https://bugs.launchpad.net/nova/+bug/178333806:15
openstackLaunchpad bug 1783338 in OpenStack Compute (nova) "Unexpected exception in API method: ValueError: year is out of range" [Medium,Confirmed] - Assigned to Ghanshyam Mann (ghanshyammann)06:15
vishakhagmann yes I will update the doc06:15
gmanni commented and confirmed that.06:16
gmannvishakha: thanks06:16
vishakhagmann: thanks for the help06:16
gmannvishakha: you can just make clear in api-ref that in-use field will always be zero for keypairs.06:16
vishakhagmann: ok sure06:16
gmannand some more clarification like keypair is per user only not per projects06:17
gmannalex_xu: on second bug, error is that start or end dates in GET /simple-tenant-usage API is not iso8601 format.06:17
gmannit says year is out of range which i do not know why as attached screenshot in bug show year as '2018'06:18
vishakhagmann : ok06:18
*** ccamacho has joined #openstack-nova06:18
gmannone thing we are surly missing is to handle the value error which leads to 500 in this API06:18
alex_xugmann: yea, we can update the query schema06:21
gmannas of now we just check string there06:21
gmann#link https://github.com/openstack/nova/blob/e8a73c400846b5be0f12a2eea32bd2c5c5aeb370/nova/api/openstack/compute/schemas/simple_tenant_usage.py#L2206:21
alex_xuyea06:22
gmannneed to check if iso8601 format there in json schema or we can do via custom format_checker also06:22
alex_xuI remember we have somewhere use it06:22
alex_xuchange-since in server list API i guess06:23
alex_xugmann: https://github.com/openstack/nova/blob/e8a73c400846b5be0f12a2eea32bd2c5c5aeb370/nova/api/openstack/compute/schemas/servers.py#L59206:24
gmannah yeah06:24
gmann#link https://github.com/openstack/nova/blob/e8a73c400846b5be0f12a2eea32bd2c5c5aeb370/nova/api/validation/validators.py#L4706:24
gmannalex_xu:  thanks, that will be quick06:24
alex_xubut the 2018 is out of range is very strange06:24
gmanni will update the patch for that06:24
alex_xugood to check that06:25
gmannyeah, not sure nova get 2018 or something else. ll get horizon and try out this one and will update on bug06:25
alex_xubut it only say 'year out of range', maybe the code doesn't see the 2018, it will be another value06:25
alex_xuyea06:25
gmannlet's move to next bug06:26
gmannthere is one high priority bug for href things06:26
gmann#link https://bugs.launchpad.net/nova/+bug/170566306:26
openstackLaunchpad bug 1705663 in OpenStack Compute (nova) "Wrong href link returned when providing non-existed version in GET version API" [High,In progress] - Assigned to Zhenyu Zheng (zhengzhenyu)06:26
gmanni have not got chance to review Kevin_Zheng patch yet, alex_xu did you?06:27
gmanni think he update the PS06:27
*** pcaruana has joined #openstack-nova06:27
alex_xugmann: no, I forget that one, probably try it today06:27
Kevin_Zhengo/06:27
gmannalex_xu: thanks, i will also check but tomorrow only.06:27
*** BrinZhang has joined #openstack-nova06:29
gmannthat's all from me on bug things, anything from your side ?06:29
alex_xugmann: nothing from me06:29
gmann#topic Open Discussion06:30
gmannthere is nothing on agend under open. if nothing we can close office houre early06:30
*** ShilpaSD has joined #openstack-nova06:34
gmannalex_xu: anything else ?06:35
alex_xunothing from me06:37
gmannlet's close then. thanks06:37
gmann#endmeeting06:37
openstackMeeting ended Wed Aug  8 06:37:32 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)06:37
openstackMinutes:        http://eavesdrop.openstack.org/meetings/nova_api/2018/nova_api.2018-08-08-06.02.html06:37
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/nova_api/2018/nova_api.2018-08-08-06.02.txt06:37
openstackLog:            http://eavesdrop.openstack.org/meetings/nova_api/2018/nova_api.2018-08-08-06.02.log.html06:37
*** pooja_jadhav_ has joined #openstack-nova06:38
*** takashin has left #openstack-nova06:39
*** Cardoe_ has joined #openstack-nova06:42
*** vivsoni_ has quit IRC06:42
*** vivsoni__ has joined #openstack-nova06:42
*** Cardoe has quit IRC06:43
*** Cardoe_ is now known as Cardoe06:43
*** jaosorior has joined #openstack-nova06:44
*** s10 has joined #openstack-nova06:58
*** luksky has joined #openstack-nova06:59
*** Dinesh_Bhor has quit IRC07:03
openstackgerritTetsuro Nakamura proposed openstack/nova master: api-ref: fix min_version for parent_provider_uuid in responses  https://review.openstack.org/57957707:05
openstackgerritTetsuro Nakamura proposed openstack/nova master: [placement] api-ref: add description for 1.29  https://review.openstack.org/58940707:05
*** adrianc has joined #openstack-nova07:05
*** s10 has quit IRC07:08
*** maciejjozefczyk has joined #openstack-nova07:28
gibigood morning nova07:30
*** rcernin has quit IRC07:35
*** jaosorior has quit IRC07:38
*** jaosorior has joined #openstack-nova07:38
*** udesale has joined #openstack-nova07:40
*** jaosorior has quit IRC07:41
*** jaosorior has joined #openstack-nova07:41
*** jaosorior has quit IRC07:42
*** jaosorior has joined #openstack-nova07:43
*** jpena|off is now known as jpena07:46
*** chason has quit IRC07:49
*** adrianc has quit IRC07:53
*** sahid has joined #openstack-nova07:57
*** moshele has joined #openstack-nova08:02
*** tetsuro_ has joined #openstack-nova08:06
*** tetsuro has quit IRC08:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client  https://review.openstack.org/58366708:10
ivvehey guys.. the new migrate_configure_max_speed in guest.py (fix in pike https://bugs.launchpad.net/nova/+bug/1783635) but from queens.. bandwidth=0 but is capped @ 1024KB/sec08:14
openstackLaunchpad bug 1783635 in OpenStack Compute (nova) pike "Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'" [High,Fix committed] - Assigned to Lee Yarwood (lyarwood)08:14
ivvei've applied it manually for pike. 0 should be unlimited?08:15
ivvedef migrate(self, destination, migrate_uri=None, params=None, flags=0,08:15
ivve                domain_xml=None, bandwidth=0):08:15
ivveseems to set itself to 108:16
ivveim no good at debugging this so asking for help :)08:16
ivvenote: cap is for entire host, not per vm08:18
ivveinstance..08:18
*** cdent has joined #openstack-nova08:20
lyarwoodsahid: ^ I think bandwidth=0 is unlimited right?08:21
lyarwoodhttps://libvirt.org/html/libvirt-libvirt-domain.html#virDomainMigrateSetMaxSpeed doesn't help08:21
ivveit should be, but my migrations are 1024KB/sec capped08:22
ivvejust evacuated a host and watched my bridge.. 1024KB08:22
ivveso it seems to set itself to 108:22
ivve1MB/sec08:22
lyarwoodivve: live_migration_bandwidth isn't set in nova.conf?08:24
ivveis it meant to be set via nova configuration and/or metadata for hosts/instances?08:24
ivvelyarwood: let me check!08:24
ivvekinda stumbled upon this when i updated a pike08:25
lyarwoodivve: https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6939-L6942 - you should see it logged here as well btw08:25
*** zhurong has quit IRC08:26
ivveso im guessing more issues will follow, not only the fix in bug 178363508:26
openstackbug 1783635 in OpenStack Compute (nova) pike "Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'" [High,Fix committed] https://launchpad.net/bugs/1783635 - Assigned to Lee Yarwood (lyarwood)08:26
ivvewell actually im wrong08:27
sahidlyarwood: yes but that can be overidden by the api call migrateToUri08:27
ivvethats for openstack-ansible to fix08:27
ivvei guess08:27
*** tssurya has joined #openstack-nova08:27
lyarwoodivve: that's the only other fix on top of 16.1.4 to do with migration AFAIK08:28
ivvelyarwood: you are right, the option is not even in nova.conf08:28
ivvesetting it now and restarting, sec for confirmation08:29
lyarwoodivve: it should default to 008:30
lyarwoodivve: I was trying to ensure it wasn't set to 108:30
*** stakeda has quit IRC08:31
lyarwoodsahid: it defaults to 0 there as well so I have no idea where 1 is coming from for ivve08:31
ivveit wasn't in the libvirt stanza at all08:31
ivveso im guessing defaults to 1 ? :D08:31
sahid1 is comming is you are using linux bridge08:31
sahidto slow the migration08:31
sahidto let neutron configure the network on destination node08:32
sahidthen neutron is sending an event to nova08:32
sahidand the migration can run at full speed08:33
lyarwoodhttps://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6913-L6922 - yup my bad08:33
lyarwoodsahid: does LB send events back once it's done?08:34
lyarwoodsahid: is that why we are stuck at 1MB/s08:34
ivveyeah im using linux bridges08:36
ivveit seems it never speeds up08:38
ivvei will confirm08:39
*** tetsuro_ has quit IRC08:41
sahidthere are some fix in neutron as well08:41
ivveperhaps neutron doesn't send the event. did it used to in pike 16.0.16?08:42
ivveaye feels like im missing some dep. fixes :D08:42
ivvei went from 16.0.16 -> 16.0.1708:42
ivvethere is also some issue between CPU i think08:43
ivvedifferent cpu hosts are whining and don't want to migrate at all ;)08:43
lyarwoodivve: which version of neutron do you have?08:44
lyarwoodivve: as sahid says this needs >=11.0.508:44
lyarwoodivve: https://review.openstack.org/#/c/510013/08:44
*** amarao has joined #openstack-nova08:44
*** derekh has joined #openstack-nova08:46
ivveneutron-server 11.0.6.dev4708:47
ivvepossibly some configuration then?08:48
lyarwoodsahid: ^?08:49
lyarwoodivve: are you seeing the debug line logged by the agent? `Port %s not found, will not notify nova.`08:49
mdboothowalsh: Did you see the Trilio thing on the ml?08:54
ivvechecking08:55
sahidivve: this part is in the agent, did you well upgrade neutron on the compute nodes?08:58
ivveyes it seems to be the same version08:59
ivve11.0.6.dev4708:59
ivveah wait09:00
ivveyou mean agent version09:00
ivveall agents pointing to that version09:01
ivve11.0.6.dev4709:01
ivvehowever since i think backport was missed to pike on the migration guest.py perhaps more things were missed?09:01
sahidhum..09:01
sahidcan you confiugre nova.conf in debug adn then start a live migration, then share the logs09:02
ivvei can't see any port not found error messages related to the migrations...09:02
ivveaye09:02
sahidthanks09:02
mdboothlyarwood: Did I see some scuttlebutt around nominations for membership of the CIX tiger-tag-team?09:02
lyarwoodmdbooth: wrong chat window ;)09:04
mdboothlyarwood: Ah, so it is.09:04
*** _pewp_ has quit IRC09:06
*** _pewp_ has joined #openstack-nova09:06
ivvehttps://hastebin.com/epafobulet.http09:08
ivveand the other09:10
ivvehttps://hastebin.com/kevuhihuje.coffeescript09:10
ivvethe first was receiving end, second was sending09:10
ivveon these two nodes i have configured the setting live_migration_bandwidth = 009:13
ivveeven if it is assumed default09:13
openstackgerritKevin Zhao proposed openstack/nova master: Skip CPU comparison in Aarch64  https://review.openstack.org/58976909:15
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/58951309:21
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Add workaround to stop use of qemu-img by the RT  https://review.openstack.org/58956709:21
*** alexchadin has joined #openstack-nova09:28
*** efoley has joined #openstack-nova09:33
openstackgerritOpenStack Release Bot proposed openstack/osc-placement stable/rocky: Update .gitreview for stable/rocky  https://review.openstack.org/58977309:34
openstackgerritOpenStack Release Bot proposed openstack/osc-placement stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky  https://review.openstack.org/58977409:34
openstackgerritOpenStack Release Bot proposed openstack/osc-placement master: Update reno for stable/rocky  https://review.openstack.org/58977509:34
*** vivsoni__ has quit IRC09:39
*** luksky has quit IRC09:44
*** priteau has quit IRC09:45
*** priteau has joined #openstack-nova09:45
sahidivve: line 217 we can see Nova reonfiguring libvirt to use 009:45
*** edmondsw has joined #openstack-nova09:45
sahidcan you try to configure live_migration_bandwidth to a large value instead of 009:47
*** edmondsw has quit IRC09:49
*** ratailor_ has joined #openstack-nova09:56
*** ratailor_ has quit IRC09:56
*** ratailor has quit IRC09:58
*** vivsoni has joined #openstack-nova10:00
*** ratailor has joined #openstack-nova10:01
mardimHello guys one question10:01
mardimI booted an instance succesfully10:02
mardimBut when I am trying to resize it I get novalid host was found error10:02
mardimI am using PIke10:02
mardimI am seeing the logs in scheduler and the ComputeFilter returns 0 hosts10:03
*** ratailor has quit IRC10:03
mardimI have only one compute node10:03
mardimAny idea ?10:03
*** ratailor has joined #openstack-nova10:03
mardimartom ^10:03
mardimmelwitt, ^10:04
*** ratailor has quit IRC10:04
openstackgerritChris Dent proposed openstack/nova master: Add explicit functional-py36 tox target  https://review.openstack.org/58982510:05
lyarwoodmardim: is allow_resize_to_same_host=True ?10:05
cdentgibi ^10:05
mardimYes it is10:05
mardimAnd I definitely have enough resources10:06
mardimbecause my compute node has 16 cpus10:06
mardimin hyperthreading10:06
lyarwoodl~N10:14
lyarwoodwait, this isn't mutt10:15
lyarwoodmardim: kk, looks like people did use scheduler_default_filters=AllHostsFilter10:15
lyarwoodmardim: in addition to help here, no idea if that's still required10:16
lyarwoodmardim: http://madorn.com/resize-on-single-compute.html#.W2rC13ZKhhE10:16
mardimthose are my openstack package : https://hastebin.com/useyohiwek.css10:16
lyarwoodbeen a while since I tried this10:16
mardimAlso I found this10:16
mardimhttps://bugs.launchpad.net/nova/+bug/176259610:16
openstackLaunchpad bug 1762596 in OpenStack Compute (nova) "controller nova resize instance dont' work openstack Pike" [Undecided,Invalid]10:16
*** tonyb has quit IRC10:16
mardimmaybe is realted10:16
mardimrelated10:16
*** luksky has joined #openstack-nova10:16
mardimlyarwood, I will try only with Allhosts10:17
mardimnad see what happens10:17
mardimThanks10:17
*** ratailor has joined #openstack-nova10:20
*** tonyb has joined #openstack-nova10:23
*** vivsoni has quit IRC10:23
*** vivsoni has joined #openstack-nova10:23
ivvesahid: sorry i was afk for lunch a while there10:25
ivvesahid: testing large value, 102410:25
*** vivsoni_ has joined #openstack-nova10:27
ivvesahid: no change, still 1024kb/sec10:28
*** vivsoni has quit IRC10:29
*** udesale has quit IRC10:30
*** mdbooth has quit IRC10:31
*** mdbooth has joined #openstack-nova10:31
*** efoley has quit IRC10:33
openstackgerritBoxiang Zhu proposed openstack/nova master: Refactor cell_type in compute/api.py  https://review.openstack.org/58983310:38
*** erlon has joined #openstack-nova10:38
openstackgerritSergii Golovatiuk proposed openstack/nova master: Fix URI for IPv6  https://review.openstack.org/58954810:43
*** BrinZhang has quit IRC10:53
*** BrinZhang has joined #openstack-nova10:53
cdentthanks gibi10:57
*** jaosorior has quit IRC10:58
*** dave-mccowan has joined #openstack-nova10:59
*** jaosorior has joined #openstack-nova10:59
gibicdent: totally make sense, I had a similar change locally testing some py36 fix11:01
cdentyeah, I got sick of adding one every so often11:02
*** rmart04 has joined #openstack-nova11:02
openstackgerritSergii Golovatiuk proposed openstack/nova master: Fix URI for IPv6  https://review.openstack.org/58954811:02
openstackgerritSergii Golovatiuk proposed openstack/nova master: Fix URI for IPv6  https://review.openstack.org/58954811:05
*** ratailor has quit IRC11:09
*** BrinZhang has quit IRC11:16
*** BrinZhang has joined #openstack-nova11:16
openstackgerritChen proposed openstack/nova master: Update ssh configuration doc  https://review.openstack.org/58984411:20
openstackgerritChen proposed openstack/nova master: Update ssh configuration doc  https://review.openstack.org/58984411:23
*** tbachman has joined #openstack-nova11:23
lyarwoodowalsh: re https://review.openstack.org/#/c/589548/2/nova/virt/libvirt/driver.py do we not have any code in nova or oslo that can do this for us already?11:33
*** jpena is now known as jpena|lunch11:35
owalshlyarwood: did occur to me... and now that I actually look -  https://docs.openstack.org/oslo.utils/latest/reference/netutils.html#oslo_utils.netutils.escape_ipv611:37
lyarwoodowalsh: noice, sorry wasn't looking at the latest PS but still we should use that11:38
lyarwoodowalsh: I can respin that now with tests11:38
owalshlyarwood: cool, thanks11:38
openstackgerritMerged openstack/nova master: Fix resize revert to use non-legacy alloc handling  https://review.openstack.org/58942511:42
*** alexchadin has quit IRC11:47
*** chason has joined #openstack-nova11:52
*** alexchadin has joined #openstack-nova11:54
*** zhangbailin_ has joined #openstack-nova11:56
*** BrinZhang has quit IRC11:59
*** zhangbailin_ has quit IRC12:00
*** BrinZhang has joined #openstack-nova12:00
*** udesale has joined #openstack-nova12:05
*** udesale has quit IRC12:05
*** udesale has joined #openstack-nova12:06
*** udesale has quit IRC12:07
*** udesale has joined #openstack-nova12:08
*** udesale has quit IRC12:10
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Always escapte IPv6 addresses when used in migration URI  https://review.openstack.org/58954812:13
lyarwoodowalsh: ^ needs a bug but that should be better12:13
*** tssurya has quit IRC12:13
*** panda|rover is now known as panda|ruck12:25
*** slagle has joined #openstack-nova12:33
*** jpena|lunch is now known as jpena12:35
*** tssurya has joined #openstack-nova12:36
*** holser_ has joined #openstack-nova12:42
holser_lyarwood12:43
artomOK, this is starting to get annoying, and consistently reproducible - I can't get a devstack compute to "join" another devstack allinone - it errors out with "Didn't find service registered by hostname after 60 seconds"12:43
artomWhat am I doing wrong?12:43
lyarwoodholser_: \o_ hey12:43
* holser_ nods12:43
lyarwoodholser_: https://review.openstack.org/589548 - can you also create a bug for this so we can backport it into stable?12:43
holser_concerning https://review.openstack.org/#/c/589548/ the valid case is u'[2001:0db8:85a3:0000:0000:8a2e:0370]:7334'12:43
holser_sure I will create it now12:43
lyarwoodholser_: thanks12:43
lyarwoodholser_: in terms of what is being passed in or expected?12:44
holser_what being passed12:44
holser_we have comment from sahid which is valid12:45
holser_also  def _migrate_uri(dest): should be covered also12:45
holser_uri = 'tcp://%s' % dest12:45
lyarwoodholser_: kk, I'll leave that to you then12:46
holser_sure...12:46
* holser_ is writing bug report12:46
*** slagle has quit IRC12:48
*** tbachman has quit IRC12:50
*** mriedem has joined #openstack-nova12:59
*** BrinZhang has quit IRC13:04
*** edmondsw has joined #openstack-nova13:05
mdboothlyarwood: test_instance_action passes :)13:06
mdboothlyarwood: Question is: what else did I break in the process? ;)13:06
lyarwoodmdbooth: welcome to the house of cards that is that fixture :)13:07
*** maciejjozefczyk has quit IRC13:11
*** psachin has quit IRC13:12
*** alexchadin has quit IRC13:14
mdboothlyarwood: Ok, everything in test_instance passes13:15
*** gbarros has joined #openstack-nova13:15
mdboothlyarwood: Mind if I push it to let CI see what explodes?13:15
mdboothActually, lemme just run the python2 tests first.13:16
* mdbooth only ran python3 so far13:16
mdboothHas to be a first for me13:16
lyarwoodmdbooth: yeah go for it when you're ready13:16
lyarwoodmdbooth: add yourself as a co-author while you're at it as well, assuming you've had a change things13:17
lyarwoodto change things* even13:17
*** tbachman has joined #openstack-nova13:19
*** alexchadin has joined #openstack-nova13:20
*** alexchadin has quit IRC13:22
openstackgerritMatthew Booth proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow  https://review.openstack.org/58701313:23
mdboothlyarwood: ^^^13:23
*** tbachman has quit IRC13:23
mdboothEnded up having to go a bit back to basics.13:23
mdboothBecause the bugs weren't in the fixture code, but rather in the simplifications it was intentionally making. Like using volume_id instead of attachment_id all over the place.13:25
mdboothAs soon as we started asserting things about that, everything broke.13:25
lyarwoodmdbooth: yup indeed, thanks for that, I'll rebase the other changes on to this once it passes and sort the other nits out13:26
*** maciejjozefczyk has joined #openstack-nova13:26
*** maciejjozefczyk has quit IRC13:28
*** maciejjozefczyk has joined #openstack-nova13:30
*** alexchadin has joined #openstack-nova13:31
*** tbachman has joined #openstack-nova13:31
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] api-ref: add description for 1.29  https://review.openstack.org/58940713:37
openstackgerritMatthew Booth proposed openstack/nova master: fixtures: Fail deleting non-existent attachment in CinderFixtureNewAttachFlow  https://review.openstack.org/58990013:37
mdboothlyarwood: Incidentally ^^^ removes a workaround which was previously required. No idea if it's still required, will find out.13:38
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] api-ref: add description for 1.29  https://review.openstack.org/58940713:38
lyarwoodmdbooth: kk, we can merge it in if it passes now13:42
mdboothlyarwood: ack13:43
*** alexchadin has quit IRC13:44
*** alexchadin has joined #openstack-nova13:45
openstackgerritMatt Riedemann proposed openstack/nova master: Update the parameter explain when updating a volume attachment  https://review.openstack.org/56518113:46
openstackgerritMatt Riedemann proposed openstack/nova master: Update the parameter explain when updating a volume attachment  https://review.openstack.org/56518113:48
*** _ix has joined #openstack-nova13:49
openstackgerritMerged openstack/python-novaclient master: Fix server strings in reboot operation  https://review.openstack.org/58898113:53
mdboothlyarwood: functional failed again. Coffee, then I'll take another look.13:57
*** alexchadin has quit IRC14:00
*** alexchadin has joined #openstack-nova14:00
*** alexchadin has quit IRC14:03
*** maciejjozefczyk has quit IRC14:07
*** awaugama has joined #openstack-nova14:07
*** _ix has quit IRC14:13
*** Bhujay has quit IRC14:18
*** hongbin has joined #openstack-nova14:18
mriedemlyarwood: want to address the nit in https://review.openstack.org/#/c/589513/ and then i'll +214:19
lyarwoodmriedem: yeah was about to ask if there was anything else before I hit it14:20
*** HW-Peter has joined #openstack-nova14:21
*** _ix has joined #openstack-nova14:22
*** maciejjozefczyk has joined #openstack-nova14:22
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/58951314:22
dansmithmriedem: fwiw, this was my idea: https://review.openstack.org/#/c/589567/314:24
dansmithmriedem: but I agree it would be nicer to do the thing that 95% of people need by default14:24
lyarwoodyeah sorry the commit just had it the wrong way around14:25
dansmithmriedem: I just don't think jumping through a bunch of hoops to collect correct info that doesn't matter efficiently is worth it14:25
lyarwoodyou only need the workaround if you are using DiskFilter etc14:25
dansmithlyarwood: right, that'd be my preference14:25
*** moshele has quit IRC14:29
mriedem+2 on the bottom change14:30
mriedemdid s10 say if that fixes the majority of the perf regression?14:31
lyarwoodthanks I'll sort this other mess out now14:31
lyarwoodmriedem: just brings the time down by 50%, 20 seconds without this, 10 seconds with this, previously 2 seconds.14:32
lyarwoodmriedem: so the workaround should get us back to 2 seconds14:32
openstackgerritMatthew Booth proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow  https://review.openstack.org/58701314:33
openstackgerritMatthew Booth proposed openstack/nova master: fixtures: Fail deleting non-existent attachment in CinderFixtureNewAttachFlow  https://review.openstack.org/58990014:33
mdboothlyarwood: Attempt 2 ^^^14:33
mdboothNothing architectural: just a couple of external tests asserting a variable I'd deleted14:34
efriedmriedem: o process guru, what do you think about swapping bug numbers here: https://review.openstack.org/#/c/401009/14//COMMIT_MSG@1614:36
*** Roamer` has quit IRC14:40
*** ccamacho has quit IRC14:43
*** ccamacho has joined #openstack-nova14:45
*** priteau has quit IRC14:53
* mriedem is talking to plumbers14:54
mriedemefried: yeah i noticed that as well when i looked at that bug,14:55
mriedemhttps://review.openstack.org/#/c/447355/ marked it as closing the bug but it didn't really, it should have been partial14:55
efriedmriedem: If you agree, I'll flip it and fast approve14:55
efriedmriedem: Either that or mark the second bug as a dup14:56
mriedemlet me look at what changed since i last +2ed it14:56
melwitt.14:59
mriedemefried: commented15:01
efriedack15:01
openstackgerritEric Fried proposed openstack/nova master: Fix host validity check for live-migration  https://review.openstack.org/40100915:02
efriedmriedem: Done and +A15:02
*** priteau has joined #openstack-nova15:04
openstackgerritMatthew Booth proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow  https://review.openstack.org/58701315:05
*** tbachman has quit IRC15:05
openstackgerritMatthew Booth proposed openstack/nova master: fixtures: Fail deleting non-existent attachment in CinderFixtureNewAttachFlow  https://review.openstack.org/58990015:05
mdboothlyarwood: Attempt 3: I feel good about this one15:06
*** chason has quit IRC15:06
*** chason has joined #openstack-nova15:07
*** chason has quit IRC15:07
*** chason has joined #openstack-nova15:07
*** links has quit IRC15:10
melwittdansmith: there are more updates to this bug from gnuoy, having to do with the redirection api being used https://bugs.launchpad.net/nova/+bug/178523515:11
openstackLaunchpad bug 1785235 in OpenStack Compute (nova) "metadata retrieval fails when using a global nova-api-metadata service" [Undecided,Incomplete] - Assigned to Liam Young (gnuoy)15:11
dansmithmelwitt: okay never heard back from him here that he moved to the non-allinone metadata service15:13
melwittlooking back at the IRC log from 2018-08-03, it looks like you said to run the metadata service with "the regular api server, and just not enable the osapi api if you want to only run metadata"15:13
dansmithright15:13
melwittit seems like they haven't done that and I don't know how to tell him to do that, like what command and conf to use15:14
gnuoyhi there15:14
gnuoyI have tried moving away from a standalone server and that does work for me.15:14
gnuoyHowever, I was hoping to get to the bottom of the issue with the standalone service15:15
melwittI thought we explained why that doesn't work, the database connection and message queue settings are intentionally not sent over RPC15:16
dansmithgnuoy: the standalone service is really just for running on compute nodes in multihost n-net mode, which doesn't make sense for a neutron and multi-cell environment15:16
dansmithwhich is why I said this is a doc bug if the regular api works for you15:16
gnuoyok, then thats a misunderstanding on my part. I didn't appreciate the situation was explicit as: not expected to work with neutron and cells15:17
gnuoyin that case I'll update the bug saying that the central api service works for me.15:17
*** ccamacho has quit IRC15:17
gnuoysorry for the noise.15:17
dansmithno problem, glad to hear that it works as we expect though :)15:18
*** dpawlik has quit IRC15:19
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] api-ref: add description for 1.29  https://review.openstack.org/58940715:21
*** efoley has joined #openstack-nova15:22
*** amarao has quit IRC15:23
*** luksky has quit IRC15:24
openstackgerritChris Dent proposed openstack/nova master: [placement] Avoid rp.get_by_uuid in allocation_candidates  https://review.openstack.org/58994115:29
*** hemna_ has joined #openstack-nova15:34
*** pcaruana has quit IRC15:34
*** tbachman has joined #openstack-nova15:34
openstackgerritLiam Young proposed openstack/nova master: Remove Neutron MetaAPIProxy from cellsv2-layout  https://review.openstack.org/58852515:35
openstackgerritSergii Golovatiuk proposed openstack/nova master: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/58954815:35
*** gbarros has quit IRC15:36
gnuoydansmith, melwitt, fwiw I've had a go at updating the nova metadata verbage  https://review.openstack.org/58852515:36
melwittcool, thanks15:39
*** jlvillal is now known as jlv-sick15:40
*** rmart04 has quit IRC15:42
openstackgerritJay Pipes proposed openstack/nova master: get provider IDs once when building summaries  https://review.openstack.org/58994515:42
kosamaraefried: Since yaml seems a better way forward, the spec I would propose would include most of the points of your spec. The larger differences I see so far mostly amount to device discovery and no auto-generated traits (following jaypipes' comment). Also, I would focus on generating the RPs and split the interactions with instances to a follow-up spec.15:43
efriedkosamara: Sounds good to me. As you're doing your writeup, it would be nice if you're careful to distinguish the responsibilities of the virt driver versus the compute manager. So for instance, the decision about whether to generate traits could/should (IMO) be left up to the individual driver.15:45
mdboothIs zuul grumpy about something?15:46
mdbooth"Tenant openstack isn't ready"15:46
kosamaraefried: OK. Thanks! I also left you some questions on your spec.15:48
efriedkosamara: Okay, nice, thank you. I'll get to them this afternoon.15:48
*** masayukig has quit IRC15:50
*** ShilpaSD has quit IRC15:53
*** dklyle has joined #openstack-nova15:55
*** gbarros has joined #openstack-nova15:56
*** tssurya has quit IRC15:57
*** david-lyle has joined #openstack-nova15:59
*** dklyle has quit IRC16:01
*** david-lyle is now known as dklyle16:02
*** sambetts|afk is now known as sambetts16:03
*** gbarros has quit IRC16:06
mriedemmelwitt: were you planning on closing out https://blueprints.launchpad.net/nova/+spec/versioned-notification-transformation-rocky since we're past FF?16:07
mriedemi assumed we'd close that out and pick up with a stein bp16:07
gibimriedem, melwitt: I agree. I can create a new bp for stein16:07
melwittmriedem: I was planning to close them tomorrow on RC day. are they supposed to be closed sooner than that usually?16:08
*** derekh has quit IRC16:08
melwittthe notification one and the mox one16:08
dansmithmelwitt: did you see my comment about rpc aliases on the rc1 pad?16:08
*** derekh has joined #openstack-nova16:09
melwittdansmith: I saw a comment yes, saying rc2 if we have one, else just before rc116:09
dansmithyup16:09
melwittI don't think we're anticipating a rc2, are we mriedem?16:10
dansmithin the past we had an obligatory rc2 for translations or something, but not sure that happens anymore16:11
dansmithwhich is why I said that16:11
melwittoh, I see16:11
melwittI didn't know about that16:11
mriedemwe shouldn't have an rc2 unless something gets reported as a major regression at the last minute16:12
mriedemthings have actually been too quiet compared to what kind of stuff we'd had between FF and RC1 in previous relases (like ocata and pike)16:12
mriedemgibi: i added an item to the ptg etherpad about legacy notification deprecation16:13
gibimriedem: thanks. I thought about that too as we have a good chance to finish the transformation in Stein16:13
mriedemunfortunately getting the projects consuming nova's notifications switched over to versioned would likely rest on our shoulders16:14
gibimriedem: yeah, I understand. I don't know how will we have time for such work16:15
openstackgerritMatt Riedemann proposed openstack/nova master: Add the guideline to write API reference  https://review.openstack.org/56905816:15
gibimriedem: besides that we communicate the deprecation and help answering questions16:16
mriedemwe probably won't16:16
mriedemsearchlight is in maintenance mode16:16
mriedemdesignate has like 2 active contributors16:16
mriedemtelemetry is in the same boat?16:16
mriedemnot sure about mistral16:16
mriedemthere are likely other projects consuming nova's notifications that i'm not thinking of, maybe blazar and masakari?16:16
mriedemi wonder if we could write some kind of conversion middleware16:17
gibimaybe watcher too but yeah16:17
melwitthave we sent a dev ML mail before about it? to find out who might be interested in switching over?16:17
mriedemfor projects as a crutch until they can consume versioned notifications natively16:17
mriedemgibi had an etherpad at one point16:17
*** gbarros has joined #openstack-nova16:17
mriedemno other project has talked about switching over as far as i know,16:18
*** gbarros has quit IRC16:18
mriedembecause as noted, they are short-staffed projects and doing that switch would be very low priority for them16:18
melwittyeah. I wasn't sure if they knew about the new stuff or not16:18
gibimelwitt: there was couple of mails on the ML about the fact that we are woring on such transformation and I got positive response from telemetery at least16:18
melwitta-ha, cool16:19
*** masayukig has joined #openstack-nova16:19
*** dpawlik has joined #openstack-nova16:19
gibimelwitt: but I'm totally agree to repeate the mail about the new interface, asking for current willingness to transform16:20
melwittgibi: yeah, I know you've been sending the regular mails about notification work but I wasn't sure how much that translated to other projects realizing there are new features available that they might be interested in using. just a thought16:21
gibimelwitt: yeah. So I can write a mail to the bigger audience (even tagging the project mriedem listed about in the subject) about the current state of the work and asking them about the willingnes to transition to the new interface16:22
melwittmight be moot since as was said, the projects we can think of that consume notifications are really short-staffed at this point. other than maybe telemetry16:22
melwittgibi: cool, might be worth a shot16:23
gibimelwitt: I made a note on my desk, I will do the mail tomorrow.16:24
melwittk16:24
*** dpawlik has quit IRC16:24
*** holser_ has quit IRC16:27
*** sahid has quit IRC16:27
*** moshele has joined #openstack-nova16:30
melwittmriedem: should I close out the notifications and mox blueprints now? when do you usually close them?16:31
*** efoley has quit IRC16:31
*** imacdonn has quit IRC16:38
*** imacdonn has joined #openstack-nova16:39
mriedemat FF16:39
melwittok16:41
*** dklyle has quit IRC16:43
melwittversioned notifications bp from queens was closed at RC1, so I was following that https://blueprints.launchpad.net/nova/queens16:45
*** moshele has quit IRC16:45
melwittnext time I'll do FF16:45
melwittmox and versioned notifications bps are now closed for rocky16:47
*** gyee has joined #openstack-nova16:52
melwittdansmith: I think we're a go for the RPC version aliases since we're not anticipating a RC217:07
dansmithalright17:08
dansmithwow, zero compute rpc changes in rocky17:09
dansmithand we haven't updated the non-compute aliases in a while17:11
openstackgerritDan Smith proposed openstack/nova master: Update compute rpc version alias for rocky  https://review.openstack.org/58997217:12
melwittare we supposed to update them every release?17:13
melwittyeah, I see conductor last one was ocata17:13
dansmithwell, our docs say that you can use release names in the [upgrade_levels] things,17:13
dansmithbut we don't really support anything other than compute being backlevel, so..17:13
melwittoh, ok17:14
dansmithalso, we have't had rpc bumps in the other services in a long time,17:14
dansmithwhich also means that once you're a couple versions into the unchanged-ness, there's nothing really needing to be pinned to an old version17:14
dansmithso, all that is to say.. meh I guess?17:14
melwitthey, it helps me understand it :)17:15
*** derekh has quit IRC17:18
*** dklyle has joined #openstack-nova17:21
melwittany major rpc version bumps we want to do? https://wiki.openstack.org/wiki/RpcMajorVersionUpdates17:26
melwittlooking through https://wiki.openstack.org/wiki/Nova/ReleaseChecklist17:26
melwitt"Merge latest translations" what does that mean?17:27
dansmithtranslation patches which we don't have anymore I think17:29
*** sambetts is now known as sambetts|afk17:29
melwittah, ok17:29
dansmithI think we already discussed the major bumps.. I looked around and none seem particularly fruitful at the moment,17:29
dansmithand compute hasn't changed all cycle17:29
melwittokay, thanks. I must not have connected the dots to this checklist17:30
mriedemthe only thing i mentioned to dan earlier (a week or two ago) was scheduler rpcapi,17:32
mriedemthere are some "drop in 5.x" stuff in there17:32
mriedembut nothing critical17:32
dansmithyeah, but I looked at it, I don't think it's worth the trouble17:32
mriedemthis is also gross https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L32517:33
mriedembut meh17:33
mriedemthat code probably dies in stein when i can drop the legacy reqspec compat stuff17:33
* mriedem sharpens the blade17:33
melwittheh17:33
openstackgerritMerged openstack/nova master: conf: Deprecate 'network_manager'  https://review.openstack.org/53092317:40
*** moshele has joined #openstack-nova17:40
*** moshele has quit IRC17:44
*** mdbooth has quit IRC17:57
*** dklyle has quit IRC17:57
openstackgerritMerged openstack/python-novaclient master: Use uuidutils of oslo.utils  https://review.openstack.org/58971718:00
*** jpena is now known as jpena|off18:07
*** dklyle has joined #openstack-nova18:09
*** itlinux has joined #openstack-nova18:12
cdentmy stats thank you mriedem18:19
*** moshele has joined #openstack-nova18:20
*** moshele has quit IRC18:21
openstackgerritmelanie witt proposed openstack/nova master: Add functional test for affinity with multiple cells  https://review.openstack.org/58507318:23
openstackgerritmelanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells  https://review.openstack.org/54025818:23
mriedemcdent: heh because i approved 2 changes in 2 days?18:28
*** ircuser-1 has quit IRC18:30
*** dklyle has quit IRC18:31
*** dklyle has joined #openstack-nova18:34
openstackgerritmelanie witt proposed openstack/nova master: Use TLSv1.2 for secure VNC access  https://review.openstack.org/58999218:35
*** janki has quit IRC18:37
cdentmriedem: no for assigning that ancient bug t me18:46
mriedemoh heh18:47
mriedemyeah doing some late summer cleaning18:47
*** sapd has quit IRC18:49
*** dklyle has quit IRC18:59
dansmithmriedem: melwitt assuming no cells meeting today19:08
openstackgerritMatt Riedemann proposed openstack/nova master: FakeDriver: adding and removing instances on live migration.  https://review.openstack.org/24361319:09
mriedemi don't think i have any major developments since last week,19:09
mriedemas of yesterday, it sounds like tssurya's handling a down cell spec will need to also account for doing server group calculations where members are in a down cell19:09
mriedemand what we do in that case19:09
melwittyeah, that. and the multi-cell affinity bug fix of mine is finally ready for review again, I got a decent functional test working with it https://review.openstack.org/54025819:10
dansmithack, since FF I've been assuming we'll pick that up after things open up19:11
dansmithmelwitt: yeah but not until after rc1 at least right?19:11
melwittand got rid of the unit test malarkey19:11
melwittdansmith: yeah. given the timing and the latentness + multi-cell-only nature, wait until after rc119:11
melwittwas just bringing it up for celly info19:13
mriedemin cells related news, i pushed up a fix for this over the weekend https://review.openstack.org/#/c/588943/19:14
mriedemfollows the same pattern as cleaning up instance mappings and reqspecs for archived instances19:14
dansmithah yeah19:14
melwittfor the multi-cell affinity, I could also see it being, wait for stein and backport to rocky, queens, pike. whatever you all think is best19:17
*** tbachman has quit IRC19:17
mriedemyou already said that was the plan above yeah?19:29
mriedem"yeah. given the timing and the latentness + multi-cell-only nature, wait until after rc1"19:29
mriedemmaybe you meant rc2?19:30
mriedemtbc, we generally shouldn't assume/expect an rc219:30
mriedemotherwise we failed at rc119:30
dansmithI don't think we should do an rc2 for that19:30
mriedemi remember loving explaining how we did rc's upstream to our downstream PMs at ibm since i tried to model the same release process19:30
mriedem"ok so when do we do rc2 and rc3?"19:31
mriedem"we don't, get your fixes into rc1 or don't cut rc1"19:31
mriedem"BUT!!!"19:31
dansmithwell, we did have those couple of cycles where we always had an rc2 for translations on the things we got into rc119:31
mriedemright, but the PMs at ibm planned for multiple candidates because they knew we had a backlog of bugs19:31
mriedemtotal waterfall19:31
dansmithheh yeah, and,19:31
dansmithwhy call it rc1 if there's no rc2 amirite/19:32
mriedemthat's deep19:32
dansmithit's like an outline with one bullet.. no larger sin.19:32
melwittyeah, sorry, I meant rc2 vs wait until stein, that I was asking of you both19:32
mriedemi'd say stein19:32
dansmithfor sure19:32
melwittk, cool19:32
mriedemmake sure there is no immediate regression,19:32
mriedemthen backports19:32
sean-k-mooneymriedem: nova is pretty waterfall too in general that said the runways has made nova mor agile this cycle19:32
* dansmith groans at the a-word19:33
mriedemhey man19:33
melwittso what about the placement perf thing? just try to land it for RC1 and not wait for RC2?19:33
mriedemmelwitt: my understanding on that is there are 2 fixes19:33
mriedemcdent's is the more important of the two and is already approved19:33
melwittthat's correct. one is approved, other is up for review19:33
mriedemi.e. cdent's drops the perf by 50%19:33
mriedemjay's drops that another 50%19:33
sean-k-mooneydansmith: :) most of intel liked that word but followed water-scurm-fall developemnt instead19:34
mriedemi'm pretty sure my first 10 years at ibm the dev model was really code-and-fix19:34
*** tbachman has joined #openstack-nova19:34
mriedemthe bestest of models19:34
mriedemmelwitt: i'll defer to efried and cdent and the placement boyz on how comfortable they are on jay's fix for rc219:35
sean-k-mooneymriedem: by drops the perf by 50% do you mean makes it better or worse19:35
mriedemimproves perf by 50%19:35
sean-k-mooney:)19:35
mriedemit's the dpdk of patches19:36
sean-k-mooneymriedem: really fast and imposible to debug19:36
mriedemheh that's a pretty good analogy for placement19:36
melwittmriedem: okay, sounds fair. efried and cdent, let me know what you think of it once you've reviewed19:37
jaypipesmelwitt: re: https://bugs.launchpad.net/nova/+bug/1746863, I thought we'd always said server groups were restricted to a single cell. is that not the case?19:38
openstackLaunchpad bug 1746863 in OpenStack Compute (nova) "scheduler affinity doesn't work with multiple cells" [High,In progress] - Assigned to melanie witt (melwitt)19:38
cdentmelwitt, mriedem : I'm happy to see them both go in19:38
melwittwhat is your assessment of the risk of the change somehow making the final release worse?19:39
*** tbachman has quit IRC19:39
mriedemjaypipes: it's totally possible to have server group members wind up unintentionally in separate cells19:39
mriedemeven if in an affinity group19:40
cdentmelwitt: we talking about the placement thing on "final release worse"? the risk in those changes is very very low. the value is very very high.19:40
melwittjaypipes: they are, in a sense that affinity means same host (cells or not) and anti-affinity means different hosts. but the bug is that if you land on hostA for your first instance, because we don't look for members in all cells, we won't find that a group member is on hostA and therefore we need to co-locate instance2 to hostA for affinity if you want to add another host. I hope that makes sense19:41
sean-k-mooneymriedem: jaypipes i guess mabe you should use dansmith's pre placement filter stuff to avoid that.19:41
melwittjaypipes: *if you want to add another instance19:41
sean-k-mooney*could use19:41
mriedemsean-k-mooney: how?19:42
melwittcdent: yeah, exactly. thanks for confirming it is low risk19:42
mriedemplacement doesn't know about server groups19:42
mriedemnor cells19:42
dansmithwell, if we had a same-resource-provider thing we could kindof hack up a thing to do it via placement19:42
dansmithbut agree, it's not easy19:43
dansmithit would be trivial to just fail a boot request for affinity if we can't talk to the cell where the other members are19:43
sean-k-mooneymriedem: well the pre filter is for tenant affinity to a cell. i was thicnking if we had a request in a server group we could have a prefiltr that just picks a cell and only trys to place within that cell for the entire group19:43
dansmithsince we clearly can't honor the affinity goal19:43
dansmithsean-k-mooney: that doesn't help us19:44
dansmithsean-k-mooney: you might not be keeping tenants to cells19:44
mriedemdansmith: yeah i think that's what gibi said on mel's patch19:45
sean-k-mooneydansmith: i was not suggesting it was a depency just that if we detected there was an affinity group the only consider 1 cell for the request19:45
*** rmart04 has joined #openstack-nova19:45
dansmithmriedem: ack, haven't looked19:45
sean-k-mooneydansmith: anyway it was just a tought.19:45
mriedemwe can still race our way around affinity and wind up in different cells19:45
*** rmart04 has quit IRC19:45
mriedemif you create the servers at the same time19:45
*** tbachman has joined #openstack-nova19:46
mriedemand i think the late up-call doesn't help you there either19:46
mriedembut it's been awhile since i've thought about this19:46
sean-k-mooneyya ther isent really a way around that if you have multiple schdulers acitve.19:46
dansmithwe should be able to handle the race to multiple cells I think19:46
dansmithif we're the first instance in a group, we schedule to any cell.. if we're not, we restrict to that cell, if we thought we were and then end up being the second one after a no-restriction thing, we reschedule19:47
dansmithshould be easy to check the mappings of other members to see if they're set to catch the last case I think19:47
sean-k-mooneydansmith: would that not reqiure an up call from the compute to the top level cells db19:48
dansmithno19:48
dansmithit's all done in the scheduling superconductor loop19:48
mriedemwe don't reschedule within superconductor...19:48
dansmithbut we can19:49
dansmithwe don't currently because we shouldn't need to19:49
dansmithscheduler re-queries if it can't claim resources, and conductor doesn't know if a compute will fail, so it can't reschedule at that point,19:49
dansmithbut if we have a reason, like we go create the instance in a cell, and then learn we lost the race, we can reschedule before we map that instance19:49
dansmithyou know,19:50
dansmithI was going to spout off another potential way to solve the need for the affinity upcall in a similar way, but I need to think about that for a sec19:52
dansmithbut regardless for the cell race I think we can manage it in superconductor19:52
sean-k-mooneydansmith: is there any reason we dont create placement aggregates for cells out of interest?19:54
*** awaugama has quit IRC19:55
melwittI hope somebody can write down these ideas somewhere so we don't forget in stein19:55
sean-k-mooneyif we did that consider a cell is just a member-of right?19:55
sean-k-mooneyall the info to create the aggregate is already in the cell mapping table19:56
dansmithsean-k-mooney: I don't want to leak cells outside of nova in any way19:56
dansmithsean-k-mooney: and I definitely don't want to auto-create aggregates in nova that align to cells19:56
sean-k-mooneydansmith: aggregates in placement auto-created by nova you mean19:57
dansmithsean-k-mooney: definitely not auto-created in nova, but probably not really in placement either19:57
dansmithwe can already filter hosts by cell in the scheduler19:59
sean-k-mooneywell a placement aggreate is just a group of resouce providers. there is no real inherint semantics beyond that. so you can model cells or regions or host aggregats equally19:59
sean-k-mooneydansmith: true but if there was a agregate associated with placement could ignore everything not in that aggregate20:00
sean-k-mooneyanyway its just an idea.20:00
openstackgerritMerged openstack/nova master: Refactor cell_type in compute/api.py  https://review.openstack.org/58983320:00
*** dklyle has joined #openstack-nova20:05
*** ircuser-1 has joined #openstack-nova20:05
*** luzC has quit IRC20:07
*** luzC has joined #openstack-nova20:09
*** itlinux has quit IRC20:12
*** holser_ has joined #openstack-nova20:13
*** luksky has joined #openstack-nova20:13
melwittmriedem: don't forget to please review these ocata quota fixes before you go on vacay https://review.openstack.org/588416 and the one above it20:21
mriedemoh right20:22
*** dpawlik has joined #openstack-nova20:24
*** cdent has quit IRC20:31
openstackgerritmelanie witt proposed openstack/nova master: Add a prelude release note for the 18.0.0 Rocky GA  https://review.openstack.org/58930320:39
*** slagle has joined #openstack-nova20:39
openstackgerritSergii Golovatiuk proposed openstack/nova master: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/58954820:51
*** dpawlik has quit IRC20:57
*** dklyle has quit IRC20:58
*** tbachman has quit IRC21:00
*** dklyle has joined #openstack-nova21:02
*** gbarros has joined #openstack-nova21:06
openstackgerritSergii Golovatiuk proposed openstack/nova master: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/58954821:07
*** dklyle has quit IRC21:07
*** pcaruana has joined #openstack-nova21:07
*** holser_ has quit IRC21:11
*** holser_ has joined #openstack-nova21:12
*** slagle has quit IRC21:13
openstackgerritJay Pipes proposed openstack/nova master: split gigantor SQL placement query into multiple  https://review.openstack.org/59004121:13
*** slagle has joined #openstack-nova21:14
jaypipesmriedem, dansmith: is there a way to boot a Nova VM on behalf of another user/project?21:16
dansmithI think that has been requested before21:16
jaypipesmriedem: alternately, is there a way to transfer ownership of an instance once its been created?21:16
dansmithalso requested21:16
jaypipespretty sure that latter thing has been requested a few times, yeah21:16
melwitthaha, Oath questions21:16
dansmithyup21:17
jaypipes:)21:17
jaypipesmelwitt: yup. trying to see if we can chuck the "ytenant" patches.21:17
melwittyeah, I kept talking about it for years "on behalf of" but there's not even support for that in keystone either, from what I understand. there's no such thing as a credential that records "I am A and I have obtained a token for B"21:18
jaypipesmriedem, dansmith: BTW, https://review.openstack.org/#/c/590041 is all about that "debugging scheduler filters" thing...21:18
jaypipesmriedem, dansmith: or rather the "NoValidHosts oh noes!" thing.21:19
jaypipesbnemec-pto: https://review.openstack.org/#/c/59004121:19
dansmithboth would be complicated, especially around image, flavor, and affinity things that would be violated by the new tenant21:19
openstackgerritMerged openstack/nova master: xx_instance_type_id in list_migrations should be integer  https://review.openstack.org/58848121:19
melwittyeah, transfer of ownership would be very difficult21:20
*** priteau has quit IRC21:20
melwittbut for "on behalf of", an idea I had was if we were to take a --project-id, --user-id parameter to nova boot, that you could pass if you're admin, then we create the instance with the given project/user instead of the admin's. and we'd have to record in the logs that it happened, "Project/user admin/admin has requested instance create for person/person" so it's traceable in some way21:23
*** slagle has quit IRC21:23
dansmithwell,21:24
dansmithpresumably you get admin's access to images and flavors?21:24
dansmithso what do we do if the user tries to rebuild?21:24
dansmithor resize?21:24
dansmithand do you get admin's affinity or the user's?21:24
melwittthe project/user recorded in the database will be theirs (the user's), not the admins21:24
dansmithwhat about when they rebuild/resize/etc?21:24
melwittafter the admin creates it, no more admin involved21:24
dansmith which means they can't rebuild if the admin built with an image they can't access yeah?21:25
jaypipesdansmith: it's a "let's have the admin hand-hold you for a little while, then after that, you can use it" sort of anti-feature.21:25
melwittdansmith: yeah21:25
dansmithjaypipes: yes, anti-feature is a good name for it :)21:25
melwittyeah. I can't really remember why they want it21:26
dansmithjaypipes: without a lot of inbuilt support for that workflow, it sounds like the kind of thing that will always have a million caveata21:26
dansmithlike shelve21:26
melwittbecause admin can act upon all user's instances already, except for create, if I'm remembering correctly21:26
melwittthat is, an admin can delete a normal user's instance, etc21:27
dansmithwell, that's a terrible reason, but I know what you mean :)21:27
melwitthaha, sorry, wasn't using it as a reason21:27
melwittI mentioned it in the context of, I can't remember why they need it for create, being that admin can already do everything else to normal user instances21:28
dansmithI know21:29
*** gbarros has quit IRC21:29
*** mriedem has quit IRC21:30
*** mriedem has joined #openstack-nova21:31
melwittjaypipes: fwiw, you might be able to solve it at the keystone level, like if you could present keystone with admin creds and then receive back a token for another user (if you have the appropriate role), then it would be transparent to nova. the context nova receives would be for the wanted user21:31
melwittlike if there were a role, "Allowed to act on behalf of project X" that admin type users could be a part of21:32
dansmithor write a half page doc for your users on how to boot their first instance21:33
mriedemdansmith: we have a bug for that with shelve,21:34
melwittyeah, I can't remember what it's for, something automated where a headless user is doing it21:34
mriedemuser creates server,21:34
mriedemadmin shelves it,21:34
mriedemuser unshelves,21:34
mriedemkablammo21:34
dansmithboom21:35
dansmithheh yeah21:35
mriedemb/c user doesn't have access to the admin snapshot21:35
dansmithyar21:35
melwittthat's fun21:35
melwittI guess the keystone idea would solve that issue21:35
mriedemi'm pretty sure it's a relatively straight-forward fix, just haven't ever had the proper motivation to work on it21:35
mriedemno no no21:35
mriedemwhen we snapshot, we make the instance owner (the user) have member access to the image21:35
mriedemglance has a thing for that21:36
mriedemhttps://specs.openstack.org/openstack/glance-specs/specs/api/v1/requesting_image_memberships.html21:36
mriedemhttps://specs.openstack.org/openstack/glance-specs/specs/api/v1/adding_a_member_to_an_image.html21:36
melwittneat21:36
mriedemhttps://bugs.launchpad.net/nova/+bug/167579121:37
openstackLaunchpad bug 1675791 in OpenStack Compute (nova) "Instance created by demo user(non-admin), shelved by admin and unshelved by demo user --> ends up in error state " [Medium,Triaged] - Assigned to Damini Chopra (damini)21:37
openstackgerritMerged openstack/nova master: Add explicit functional-py36 tox target  https://review.openstack.org/58982521:38
openstackgerritMerged openstack/nova master: Add the guideline to write API reference  https://review.openstack.org/56905821:38
mriedemspeaking of changing stuff on unshelve https://review.openstack.org/#/c/559828/21:39
melwittyou've opened the shelve bug bin21:40
mriedemtime to create a shelve sig21:42
mriedem / support group21:42
mriedemfor admins that have enabled that api21:42
melwittlol21:42
*** gbarros has joined #openstack-nova21:46
mriedemmelwitt: dansmith: a couple of thoughts in this ocata-only func test https://review.openstack.org/#/c/588416/21:53
mriedemif you wanted to make it less unit-y21:53
dansmithmriedem: I definitely think this is a unit test, as I said21:55
mriedemi guess you don't actually start conductor when using NoopConductorFixture21:55
melwittargh, I thought I had to write it this way to fit in with the pattern of regression tests21:56
melwittwhichever one it needs to be, I always pick the wrong one :)21:56
openstackgerritMerged openstack/nova master: Remove Neutron MetaAPIProxy from cellsv2-layout  https://review.openstack.org/58852521:56
mriedemi'm a tad confused on the stub in the 2nd test21:58
melwittin what way?21:58
mriedemit says,21:59
mriedem"# Stub out the API to make the instance destroy raise InstanceNotFound,21:59
mriedem        # simulating if conductor already deleted it."21:59
mriedemconductor only deletes the instance if conductor gets build request not found when it goes to delete the build request, right?21:59
*** gbarros has quit IRC21:59
*** imacdonn has quit IRC21:59
mriedemwhich only happens if _attempt_delete_of_buildrequest returns True right?21:59
melwittyes. I glossed over that from writing up all the other comments. O tjoml21:59
melwitt*,I think22:00
melwittyes22:00
mriedemso...22:00
mriedemaren't the first and 2nd tests really the same thing?22:00
mriedemor22:00
mriedemi guess one is saying,22:00
melwittno because the first test will commit the reservations and the second one won't and will fall into the exception handling and rollback and refresh22:00
mriedemthe 2nd is saying,22:00
mriedembetween the time _lookup_instance returns the instance, another request deletes the build request, so conductor deletes the instance and then request 1 fails on instance.destroy()?22:01
melwittthe first test never goes into the except InstanceNotFound block22:01
melwittyes22:01
mriedemi'm not following "no because the first test will commit the reservations and the second one won't and will fall into the exception handling and rollback and refresh"22:02
*** pcaruana has quit IRC22:02
mriedembut both _lookup_instance and instance.destroy() are in the same try/except InstanceNotFound in _delete_while_booting22:02
melwittI just mean the first test will not cover the except InstanceNotFound block22:02
mriedemso the point is, each test is covering two ways we could raise in there22:02
melwitt_lookup_instance does not raise InstanceNotFound22:02
melwittit returns (None, None) if it catches InstanceNotFound22:02
mriedemoh right22:03
mriedemok22:03
melwittso first case is _lookup_instance returns (None, None) because it found no instance. second case is instance was found but then instance.destroy() raises because conductor deleted it out from under us after we deleted the build request but before we got to instance.destroy()22:04
melwitt*we deleted the build request, found the instance, and then it was gone in that small window when we got to destroy22:05
mriedemok22:06
mriedemwell, up to you on if you or dansmith want my comments addressed, if you don't care, then i don't really care much either and i'll +W22:06
*** nicolasbock has quit IRC22:06
*** moshele has joined #openstack-nova22:06
mriedemalso,22:07
mriedemyou're sure you're not hitting this? https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L172922:07
dansmithI don't like the test on multiple fronts,22:07
mriedemb/c i see you're not using the AllServicesCurrent fixture22:07
dansmithbut it seems like the fix is worth getting in, nits on the test aside22:07
melwittI'd rather not have to convert them to unit tests at this point but if you want me to change it, I will22:07
mriedemno i'm not talking about full on re-writing them as unit tests22:08
mriedemi just meant with stuff like the AllServicesCurrent fixture and NoopConductorFixture22:08
mriedembut we must not be hitting https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1729 because your change to the tests in the next patch are ok22:09
melwittyeah ... seems like it would return early because of the service version but the test does pass after the fix, which means it's getting to the quota calls somehow22:09
dansmithbecause there are no services?22:09
mriedemthe api service is running22:09
dansmithI don't see why we want noopconductor over just not starting it22:09
dansmithbut it's all just forced because this should be a unit test where it doesn't matter anyway :)22:10
*** edmondsw has quit IRC22:10
melwittSORRY GUYS for my blasphemous functional test22:10
mriedemi'm just surprised that fixture isn't needed is all i'm saying22:10
mriedemlike, why do we even need it in the first place in other func tests is my point22:11
mriedemthou shalt be smoten with review nit-pickery22:11
melwittwe have definitely needed it before. I'm not sure what's going on here22:11
melwittre: ServicesCurrent22:11
dansmithif you have no services you get back the current I think22:12
dansmithwe used to return zero,22:12
*** imacdonn has joined #openstack-nova22:12
melwittthe api service doesn't count as a service?22:12
melwittos_apicompute22:12
mriedemidk, might have something to do with wsgi-intercept usage in OSAPIFixture now22:13
dansmithyou're not starting an api service for real here22:13
dansmithright22:13
melwittO.o22:13
dansmithbut anyway if this was A UNIT TEST22:13
* dansmith twists the dagger a bit more22:13
mriedemwe used to22:13
mriedemself.osapi = service.WSGIService("osapi_compute")22:13
mriedem        self.osapi.start()22:13
mriedem        self.addCleanup(self.osapi.stop)22:13
melwitt:***(22:13
mriedemwhich is probably why AllServicesCurrent was added before22:13
*** gbarros has joined #openstack-nova22:13
mriedem+W22:15
mriedemmoving onto soul crusher #222:15
* melwitt holds off on celebrating22:16
*** rcernin has joined #openstack-nova22:19
*** moshele has quit IRC22:25
*** holser_ has quit IRC22:26
*** priteau has joined #openstack-nova22:26
mriedembazinga https://review.openstack.org/#/c/582413/822:27
melwittargh22:28
melwittok that's not so bad22:29
mriedemdo you want/need to hold the functional test patch below it out to add checks for core and ram usage?22:29
mriedemor just slap those into this one22:29
melwittI'm fine with adding them to the second one22:30
*** gbarros has quit IRC22:37
*** hamzy_ has joined #openstack-nova22:40
*** hamzy has quit IRC22:42
*** tbachman has joined #openstack-nova22:48
openstackgerritMatt Riedemann proposed openstack/nova master: Docs: Add guide to migrate instance with snapshot  https://review.openstack.org/58444222:50
*** sapcc-bot has joined #openstack-nova22:53
*** hongbin has quit IRC22:54
*** zioproto_ has joined #openstack-nova22:55
*** gmann_ has joined #openstack-nova22:55
*** r-daneel has joined #openstack-nova22:59
mriedemheh https://review.openstack.org/#/c/382853/23:00
mriedemi knew i'd seen this recently23:00
mriedemhttps://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R324523:00
mriedemoct of 201623:00
*** lbragstad has quit IRC23:01
*** amotoki_ has joined #openstack-nova23:01
*** stephenfin_ has joined #openstack-nova23:02
*** sapcc-bot3 has quit IRC23:02
*** evrardjp has quit IRC23:02
*** SamYaple has quit IRC23:02
*** Tahvok has quit IRC23:02
*** kevinbenton has quit IRC23:02
*** adreznec has quit IRC23:02
*** weezhard has quit IRC23:02
*** amrith has quit IRC23:02
*** sean-k-mooney has quit IRC23:02
*** mtreinish has quit IRC23:02
*** penick has quit IRC23:02
*** stephenfin has quit IRC23:02
*** oanson has quit IRC23:02
*** Gorian has quit IRC23:02
*** dulek has quit IRC23:02
*** lucasagomes has quit IRC23:02
*** jogo has quit IRC23:02
*** amotoki has quit IRC23:02
*** dosaboy has quit IRC23:02
*** gmann has quit IRC23:02
*** zioproto has quit IRC23:02
*** gmann_ is now known as gmann23:02
*** zioproto_ is now known as zioproto23:02
*** Tahvok has joined #openstack-nova23:03
*** kevinbenton has joined #openstack-nova23:03
*** mtreinish has joined #openstack-nova23:04
openstackgerritmelanie witt proposed openstack/nova stable/ocata: [stable only] Handle quota usage during create/delete races  https://review.openstack.org/58241323:04
*** mhen has quit IRC23:05
*** mhen has joined #openstack-nova23:08
*** jogo has joined #openstack-nova23:09
openstackgerritMatt Riedemann proposed openstack/nova master: Update nova network info when doing rebuild for evacuate operation  https://review.openstack.org/38285323:11
openstackgerritMatt Riedemann proposed openstack/nova master: Update nova network info when doing rebuild for evacuate operation  https://review.openstack.org/38285323:12
mriedemwow so that evacuate fix was approved in march of 2017,23:15
mriedembut apparently needed a rebase23:15
mriedemand then no one ever mentioned needing re-approval23:15
melwittyikes23:17
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Fix instance evacuation with PCI devices  https://review.openstack.org/59005923:18
*** luksky has quit IRC23:26
melwittI'm going to propose stable releases tomorrow. I meant to do it last week but didn't get to it23:29
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Update nova network info when doing rebuild for evacuate operation  https://review.openstack.org/59006223:33
openstackgerritMerged openstack/nova master: get provider IDs once when building summaries  https://review.openstack.org/58994523:34
*** slaweq has quit IRC23:37
*** slaweq has joined #openstack-nova23:38
*** BrinZhang has joined #openstack-nova23:49
*** BrinZhang has quit IRC23:54
*** BrinZhang has joined #openstack-nova23:54
*** zhangbailin_ has joined #openstack-nova23:54
*** moshele has joined #openstack-nova23:55
openstackgerritMerged openstack/nova master: libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/58951323:57
openstackgerritMerged openstack/nova master: Add tempest-slow job to run the tempest slow tests  https://review.openstack.org/56769723:57
openstackgerritMerged openstack/nova master: Fix bad links for admin-guide  https://review.openstack.org/58025923:57
openstackgerritMerged openstack/nova master: [placement] Avoid rp.get_by_uuid in allocation_candidates  https://review.openstack.org/58994123:57
openstackgerritMerged openstack/nova master: api-ref: fix min_version for parent_provider_uuid in responses  https://review.openstack.org/57957723:57
openstackgerritMerged openstack/nova master: doc: mark the max microversion for rocky  https://review.openstack.org/58959823:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!