Tuesday, 2018-07-24

*** shu-mutow has joined #openstack-sdks00:12
*** markvoelker has quit IRC00:19
*** markvoelker has joined #openstack-sdks00:47
*** JudeC has quit IRC00:48
openstackgerritOpenStack Proposal Bot proposed openstack/os-service-types master: Updated from OpenStack Service Type Authority  https://review.openstack.org/58506200:50
openstackgerritMerged openstack/python-openstackclient master: Do not require port argument when updating floating IP  https://review.openstack.org/57505700:53
*** chenyb4 has joined #openstack-sdks01:02
*** slaweq has joined #openstack-sdks01:11
*** slaweq has quit IRC01:15
*** edmondsw has joined #openstack-sdks01:27
*** edmondsw has quit IRC01:31
*** mhen has quit IRC01:40
*** mhen has joined #openstack-sdks01:41
*** slaweq has joined #openstack-sdks02:11
*** slaweq has quit IRC02:16
*** dave-mccowan has quit IRC02:31
*** samP has joined #openstack-sdks02:57
*** samP has quit IRC02:59
*** samP has joined #openstack-sdks03:05
*** samP has quit IRC03:10
*** samP has joined #openstack-sdks03:13
*** samP has quit IRC03:15
*** edmondsw has joined #openstack-sdks03:15
*** edmondsw has quit IRC03:19
*** samP has joined #openstack-sdks03:19
*** samP has quit IRC03:26
*** samP has joined #openstack-sdks03:39
*** samP has quit IRC03:39
*** samP has joined #openstack-sdks03:46
*** samP has quit IRC03:48
*** samP has joined #openstack-sdks03:55
*** samP has quit IRC03:55
*** samP has joined #openstack-sdks03:57
*** samP has quit IRC03:58
*** samP has joined #openstack-sdks04:03
*** samP has quit IRC04:05
*** samP has joined #openstack-sdks04:09
*** samP has quit IRC04:10
*** slaweq has joined #openstack-sdks04:11
*** JudeC has joined #openstack-sdks04:15
*** slaweq has quit IRC04:16
*** samP has joined #openstack-sdks04:16
*** samP has quit IRC04:16
*** samP has joined #openstack-sdks04:22
*** samP has quit IRC04:23
*** samP has joined #openstack-sdks04:28
*** samP has quit IRC04:28
*** samP has joined #openstack-sdks04:29
*** samP has quit IRC04:29
*** samP has joined #openstack-sdks04:38
*** samP has quit IRC04:39
*** e0ne has joined #openstack-sdks04:40
*** e0ne has quit IRC04:40
*** samP has joined #openstack-sdks04:44
*** samP has quit IRC04:45
*** honza has quit IRC04:47
*** honza has joined #openstack-sdks04:48
*** tellesnobrega has quit IRC04:49
*** honza is now known as Guest5685004:49
*** tellesnobrega has joined #openstack-sdks04:49
*** samP has joined #openstack-sdks04:52
*** e0ne has joined #openstack-sdks04:53
*** samP has quit IRC04:53
*** e0ne has quit IRC04:55
*** samP has joined #openstack-sdks04:57
*** samP has quit IRC04:58
*** samP has joined #openstack-sdks05:03
*** edmondsw has joined #openstack-sdks05:03
*** samP has quit IRC05:04
*** edmondsw has quit IRC05:08
*** JudeC has quit IRC05:09
*** samP has joined #openstack-sdks05:10
*** samP has quit IRC05:10
*** slaweq has joined #openstack-sdks05:11
*** samP has joined #openstack-sdks05:12
*** samP has quit IRC05:12
*** samP has joined #openstack-sdks05:14
*** slaweq has quit IRC05:15
*** samP has quit IRC05:15
*** Luzi has joined #openstack-sdks05:45
openstackgerritJosephine Seifert proposed openstack/python-openstackclient master: Don't sent disk_over_commit if nova api > 2.24  https://review.openstack.org/58233405:54
openstackgerritJosephine Seifert proposed openstack/python-openstackclient master: [WIP] osc-included image signing (using openstacksdk)  https://review.openstack.org/58008605:55
*** slaweq has joined #openstack-sdks06:11
*** slaweq has quit IRC06:16
*** slaweq has joined #openstack-sdks07:11
*** JudeC has joined #openstack-sdks07:12
*** slaweq has quit IRC07:12
*** slaweq has joined #openstack-sdks07:16
*** thanhnb has joined #openstack-sdks07:24
thanhnbsorry about my english.07:26
thanhnbi want help about "Connection" in openstack sdk.07:27
thanhnbi has read in docs. It said "Using an existing authenticated keystoneauth1.session.Session, such as might exist inside of an OpenStack service operational context"07:27
thanhnbHow can i use "keystoneauth1.session.Session" in this situation.07:28
*** markvoelker has quit IRC07:29
cmurphythanhnb: you can create a keystoneauth session following these docs https://docs.openstack.org/keystoneauth/latest/using-sessions.html07:33
thanhnbi has created it07:34
thanhnbbut it seem not work07:34
thanhnbthis is how i do it07:36
thanhnbam i right?07:36
cmurphythanhnb: it seems okay to me, what problem are you having?07:37
*** gildub has joined #openstack-sdks07:38
thanhnbthis is bug07:39
thanhnbhttp://paste.openstack.org/show/726499/ this is code when i has bug07:40
*** kimamisa has joined #openstack-sdks07:42
thanhnbi think the session is ok, but the connection doesn't create07:42
cmurphythanhnb: what version of openstacksdk and keystoneauth do you have?07:46
cmurphyI tried your code and it works for me so it might be a bug in one of the libraries07:47
thanhnbkeystoneauth1==3.10.0 and openstacksdk==0.17.007:49
thanhnbwhat is your version of openstacksdk and keystoneauth1?07:50
cmurphythanhnb: same versions07:53
cmurphyhmm :/07:53
thanhnblet me try another venv. :(07:56
thanhnbcmurphy- so anyway, thanks you :D07:58
cmurphythanhnb: no problem but I'm still not sure what's going on :)08:02
thanhnbi has created another venv08:02
thanhnbbut it still error @@08:03
thanhnbsame with this.08:03
thanhnbcan you send me your code @@08:03
cmurphythanhnb: i'm using your same code just with the password and user id changed08:04
cmurphyactually now i've tried it on a different system and it's giving me a completely different error, "Auth plugin requires parameters which were not given: auth_url" so i'm even more confused :)08:05
thanhnbi has openstack in centos 708:06
thanhnband run code in ubuntu18 @@08:06
thanhnbi has same bug "not given: auth_url" when i dont declared "auth_url"08:07
*** jpich has joined #openstack-sdks08:07
thanhnbmy openstack is queen08:08
*** Luzi has quit IRC08:11
thanhnbi have tried it in different node, but it still error08:13
*** gkadam has joined #openstack-sdks08:13
cmurphythanhnb: aha i reproduced your problem08:15
*** e0ne has joined #openstack-sdks08:15
thanhnbwow :D08:15
cmurphywhen I first tried it I had credentials set in OS_* environment variables08:15
cmurphyif i unset OS_PASSWORD then the problem occurs08:16
cmurphybased on that doc i think that's probably not what's intended, want to file a bug?08:17
thanhnbso i need "OS_PASSWORD" in my environment?08:17
*** shu-mutow has quit IRC08:18
thanhnbwhat is "want to file a bug" , i dont understand @@08:18
thanhnbi just newbie in openstack08:18
*** JudeC has quit IRC08:19
cmurphythanhnb: i mean to say, could you report this bug in the bug tracker? I believe you can do that here https://storyboard.openstack.org/#!/project/97208:20
*** gildub has quit IRC08:20
thanhnbyes, i think i can do this.08:20
cmurphycc mordred ^08:21
*** Luzi has joined #openstack-sdks08:26
*** ralonsoh has joined #openstack-sdks08:31
*** dtantsur|afk is now known as dtantsur08:33
*** cdent has joined #openstack-sdks08:49
*** dmellado has quit IRC09:03
*** markvoelker has joined #openstack-sdks09:33
*** stephenfin has quit IRC09:47
*** stephenfin has joined #openstack-sdks09:49
*** markvoelker has quit IRC10:04
*** thanhnb has quit IRC10:14
*** dmellado has joined #openstack-sdks10:24
*** e0ne has quit IRC10:27
*** chenyb4 has quit IRC10:31
*** e0ne has joined #openstack-sdks10:32
*** edmondsw has joined #openstack-sdks10:35
*** edmondsw has quit IRC10:40
*** e0ne has quit IRC10:40
*** cdent has quit IRC10:55
*** markvoelker has joined #openstack-sdks11:01
*** markvoelker has quit IRC11:34
*** cdent has joined #openstack-sdks11:53
*** edmondsw has joined #openstack-sdks11:57
*** e0ne has joined #openstack-sdks12:07
*** kimamisa has quit IRC12:12
*** markvoelker has joined #openstack-sdks12:12
mordredcmurphy: reading12:15
mordredoh poo. thanhnb is gone12:16
mordredcmurphy: fwiw, I think this is a doc bug - it is implying that the above is something that people should do12:20
mordredthe whole "Using an existing authenticated keystoneauth1.session.Session, such as might..." is really meant for inside of nova or something else where keystone_middleware has already provided a Session - I don't expect it to be a pattern people use in any other context12:25
*** mriedem has joined #openstack-sdks12:28
*** fabian_ has joined #openstack-sdks12:45
*** chenyb4 has joined #openstack-sdks12:51
*** cdent has quit IRC12:55
*** r-mibu has joined #openstack-sdks13:04
cmurphymordred: but in theory shouldn't that work?13:07
mordredyes - in theory it should work - so I'm sure there is a bug in there somewhere13:07
mordredbut the first bug is that doing it that way is the hardest possible way to use openstacksdk, so if someone is trying to get started and that's the way they're trying to get started, the docs are leading them astray13:08
mordred(I'm especially curious as to why you having OS_PASSWORD set in the env had any impact at all due to the way session is saved and reused in that workflow)13:09
cmurphyfair enough, it led me astray too13:09
mordredlike - that session has an Auth plugin already, so openstacksdk should not be doing _anything_ with auth information13:09
cmurphythat is what was surprising13:09
*** dave-mccowan has joined #openstack-sdks13:11
*** dave-mcc_ has joined #openstack-sdks13:20
openstackgerritEric Fried proposed openstack/os-service-types master: Switch to stestr  https://review.openstack.org/58534913:21
openstackgerritEric Fried proposed openstack/os-service-types master: Updated from OpenStack Service Type Authority  https://review.openstack.org/58506213:22
*** dave-mccowan has quit IRC13:22
*** mriedem is now known as mriedem_away13:24
*** cdent has joined #openstack-sdks13:40
openstackgerritEric Fried proposed openstack/os-service-types master: Switch to stestr  https://review.openstack.org/58534913:44
*** Luzi has quit IRC13:53
*** mriedem_away is now known as mriedem14:11
*** fabian_ has quit IRC14:13
*** mriedem1 has joined #openstack-sdks14:18
*** chenyb4 has quit IRC14:18
openstackgerritEric Fried proposed openstack/os-service-types master: Switch to stestr  https://review.openstack.org/58534914:19
*** mriedem has quit IRC14:21
*** mriedem1 is now known as mriedem14:24
*** chenyb4 has joined #openstack-sdks14:30
*** ralonsoh_ has joined #openstack-sdks14:31
*** ralonsoh has quit IRC14:33
*** samP has joined #openstack-sdks14:38
*** ralonsoh has joined #openstack-sdks14:57
*** purplerbot has quit IRC14:57
*** purplerbot has joined #openstack-sdks14:57
*** ralonsoh_ has quit IRC15:01
*** purplerbot has quit IRC15:08
*** purplerbot has joined #openstack-sdks15:08
openstackgerritArtom Lifshitz proposed openstack/python-openstackclient master: Don't sent disk_over_commit if nova api > 2.24  https://review.openstack.org/58233415:09
*** chenyb4 has quit IRC15:16
openstackgerritMonty Taylor proposed openstack/os-service-types master: Updated from OpenStack Service Type Authority  https://review.openstack.org/58506215:17
samueldmqhow many clouds have we validated shade (or the upper abstraction layer) against?15:45
samueldmqmordred: ^15:45
mordredsamueldmq: many :)15:45
samueldmqmordred: nice, I'd like to do some formal validation and report results15:46
mordredsamueldmq: all of the openstack public clouds- and many of the private (I don't have a count)15:46
samueldmqI was thinking about all those in the passport program to start15:46
samueldmqoh that's awesome ...15:46
mordredsamueldmq: sweet. you may also want to talk to melvinhillsman - the openlab folks have been working on setting up jobs to run openstacksdk tests against actual public cloud accounts15:46
rcarrillocruzyah, openlab, that15:47
* rcarrillocruz goes back to his cave15:47
*** samP has quit IRC15:47
samueldmqrcarrillocruz: o/15:47
samueldmqmordred: awesome, I met him at last summit, will talk to him15:47
openstackgerritMonty Taylor proposed openstack/os-service-types master: Allow passing in service types with _ in them  https://review.openstack.org/58541015:48
openstackgerritMonty Taylor proposed openstack/os-service-types master: Add flag for returning unofficial types  https://review.openstack.org/58541115:48
mordredsamueldmq: woot.15:48
samueldmqmordred: where is shade code in the new -sdk?15:49
mordredsamueldmq: although - for your case, it's also possible that doing the formal validation by hand would be fine - since that's different from making sure to run validation on every patch15:49
mordredsamueldmq: openstack/cloud15:49
samueldmqmordred: ++ exactly... just want to validate the tool works15:50
samueldmqalso, I don't need/want to be exhaustive or too complex15:50
mordredsamueldmq: and it's all attached to the Connection object - so as a user if you do "conn - openstack.connect(cloud='vexxhost')" - you can use that conn object same as the OpenStackCloud object from shade15:50
*** samP has joined #openstack-sdks15:50
mordredsamueldmq: ++15:50
samueldmqvalidating a core set of functionalities should be fine (whatever be a core set: vm managemnt?)15:50
mordredsamueldmq: https://review.openstack.org/#/c/582459/ is a patch you might want to look at15:50
mordredsamueldmq: it's a patch to allow running our ansible tests against a public cloud that is not devstack15:51
mordredShrews: ^^ speaking of - you might want to check out that patch too15:51
samueldmqmordred: awesome15:51
samueldmqdo all the tests normally pass?15:51
samueldmqI wouldn't be surprised otherwise... there may be some services that do not exist in some cases, for ex15:52
mordredsamueldmq: well - our ansible tests have a pretty small surface area15:52
mordredI'd love to be able to run our functional tests against a cloud - and hopefully we'll also get that working at some point15:53
samueldmqmordred: how do you see that?15:53
mordredsamueldmq: and I think it's ok for the test suite to skip some tests if they don't have a service15:53
samueldmqreimplement with ansible15:53
samueldmqor wrap the existing tests somehow to make it owrk15:53
mordrednah - it's mostly just about reworking base fixtures15:53
mordredin some clouds we may not have admin, for instance, so making a new project and testing keystone admin functions should all be skipped15:54
samueldmqto allow oyu input the cloud to talk against15:54
mordredbut we could still test that booting a vm works15:54
samueldmqwhen running tests right15:54
*** samP has quit IRC15:54
mordredthe tests already will skip if the service doesn't exist - then we have a devstack-specific test that responds to env vars and fails if an envvar asserts a service should exist but the shade tests don't find it15:55
mordredthat way we can make sure we don't accidentally fail open15:55
mordredi the gate15:55
mordredso we could probably expand on that somehow15:55
Shrewsmordred: lgtm15:56
mordredto allow manual running against public clouds to be more flexible without sacrificing gate stuff15:56
mordredShrews: woot15:56
samueldmqmordred:  how are those vars set up then?15:56
samueldmqI assume that cn be automatic by looking up the service catalog15:56
samueldmqbut would require some work15:56
mordredwell - "has_service('network')" uses the service catalog15:56
mordredbut the env var is defined in the test job15:56
mordredso the test job (which sets up the services in the first place) also sets vars that say "I believe I set up magnum"15:57
mordredthen the shade test says "the test config expects magnum, do I see it in the catalog"15:57
samueldmqah makes sense15:57
samueldmqbut none of that apply to manual run15:57
samueldmqalso, runnig against real public clouds wouldn't require any setup15:58
mordredthat's the theory :)15:58
samueldmqopenstack/ocnfig that is15:58
mordredin practice, I think there are still a good amount of code in fixtures that assumes we're running against a devstack15:58
mordredand could be make more smarter, we just haven't had the time yet15:58
samueldmq"whatever that can happen will happen" someone said15:58
samueldmqand then shade exists :-)15:58
samueldmqkk I have some info to mull over15:59
*** samP has joined #openstack-sdks15:59
samueldmqI'm happy I got an old laptop running linux again, was bored using windows all the time for company stuff16:00
Shrewssomewhere in shade i had a script that allowed us to run those tests against another (non-devstack) cloud16:00
samueldmqwhich means I am expecting to start having fun around here16:00
mordredsamueldmq: yay!16:00
samueldmqShrews: that'd be a good start16:00
mordredShrews: yeah- I remember that ...16:00
Shrewsoh, even tox support for that16:00
Shrewstox -e ansible -- -c cloudX [TAG ...]16:00
Shrewshow clever i was16:01
*** jpich has quit IRC16:01
samueldmqand that looks up on the regular /opt/openstack/clouds.yaml?16:01
samueldmqor is there a special thing that was setup for devstack16:01
Shrewsand the tags were to selectively choose which things you wanted to test (in case the cloud didn't support something)16:02
Shrewslooks like we carried that over to sdk, but i haven't tested it there16:03
samueldmqI'll take  a look to get familiar16:03
samueldmqand see what works and what doesn't16:03
*** e0ne has quit IRC16:10
*** dtantsur is now known as dtantsur|afk16:39
*** JudeC has joined #openstack-sdks16:51
*** ralonsoh has quit IRC17:02
*** slaweq has quit IRC17:16
*** e0ne has joined #openstack-sdks17:30
openstackgerritMerged openstack/openstacksdk master: Run ansible tests against specific public cloud  https://review.openstack.org/58245917:38
*** r-mibu has quit IRC17:51
*** timburke_ is now known as timburke18:14
*** e0ne has quit IRC18:14
*** slaweq has joined #openstack-sdks18:15
*** mriedem1 has joined #openstack-sdks18:16
*** mriedem has quit IRC18:16
*** gkadam has quit IRC18:17
*** bobh has joined #openstack-sdks18:19
openstackgerritMerged openstack/keystoneauth master: Add ability to filter version data by service-type  https://review.openstack.org/58502918:22
*** mriedem1 is now known as mriedem18:27
mordredShrews, dtroyer, dhellmann: feel like reviewing some os-service-types patches? (https://review.openstack.org/#/q/project:openstack/os-service-types+status:open)18:37
Shrewsi don't even know what that repo is (though i can infer from the name)   lol18:51
mordredShrews: :)18:54
*** bobh has quit IRC18:58
*** alex_xu has quit IRC19:03
*** alex_xu has joined #openstack-sdks19:07
corvusmordred: i'm having trouble using the new data= parameter to create_object:19:32
corvus  File "/home/corvus/git/openstack-infra/zuul-jobs/.tox/py35/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7525, in create_object19:32
corvus    file_size = len(data)19:32
corvusTypeError: object of type 'DeflateFilter' has no len()19:32
corvusmordred: the session.put code i was using before handled data being an iterator19:32
mordredcorvus: ah! hrm. SO ...19:33
corvuser, cloud.object_store.put which i guess is the proxy put.  which, i think, is basically going to call session.put.  you get the idea.19:34
mordredcorvus: that len() is there in service of figuring out whether the data to be uploaded needs to be uploaded as multiple segments19:34
corvusmordred: i believe httplib handles that automatically19:34
mordredwell, not for swift it doesn't19:35
corvus(briefly, if it is an iterator with no len, it gets uploaded in multiple segments)19:35
corvussorry, chunked encoding19:35
corvusare you talking about something else?19:35
mordredyah - this is about uploading to multiple swift objects if the object size is > max_objet_size19:36
mordredand then creating either a DLO or SLO object out of that19:36
mordred(dynamic large object or static large object)19:36
corvusmordred: that's something you have to decide on before you upload?19:36
mordredyup- it impacts how you upload things - and also causes sdk to spawn up a set of worker threads to handle it if it's needed19:37
corvusthat sounds fundamentally incompatible with streaming compression19:38
mordredwell ...19:39
mordredI believe each segment upload itself could be uploaded with streaming compression19:40
mordredwe should ask notmyname about it though ...19:40
mordredcorvus: the simple approach could be skip the length/segment step if data is passed as a parameter19:41
mordredcorvus: fwiw, vexxhost has 'max_file_size': 536870912019:42
corvusmordred: just assume the object isn't too large?19:42
mordredyeah. I mean, the 'standard' max_file_size is what vexxhost has -which is 5G19:42
mordredcorvus: but - the way these work is one of two ways (each resulting in a set of discreet objects being uploaded)19:43
mordredin both cases the data is uploaded in a set of 'segment' objects. in static large objects there is then a manifest object that lists the segment objects and the order in which they should be served19:44
mordredin dynamic the segment objects follow a naming scheme with an empty manifest object (so you don't have to keep track of the list and then upload the manifest object)19:45
mordredeach of the segment objects is a completely independent swift object, so I'd expect uploading to them with deflate would work19:45
mordredoh - then when you fetch the objects, as a client, you just fetch the manifest object, and swift knows that what you want to do is stream the concatenation of the segment objects19:46
mordredit's the download that I'm not sure what happens19:47
corvusthe download?19:47
mordredthe fetch19:47
corvusi'm confused, i thought you just described what happens in that case19:48
mordredyes - in the standard case - I do not know how it interacts with streaming compression19:48
mordredlargely becaues I've never tested it - I'd imagine it would DTRT19:49
mordredbut don't actually know19:49
corvusoh i see what you're saying19:49
mordredcorvus: out of curiosity - what is the iterator that has the data in it?19:50
mordredoh. the deflatefilter19:51
mordred*duh* it's in the traceback19:51
corvusyeah.  it's a thing i wrote that compresses 16k at a time and produces an unknown amount of data each chunk19:52
mordredcorvus: well - here's another question then ...19:55
mordredcorvus: does max_file_size apply to the compresesd or uncompresed value19:57
corvusexcellent question!  it's all very vague19:57
corvusi'm assuming, for the moment, that swift doesn't attempt to do any decoding.  so probably it applies to the compressed value.19:58
corvus(i'm assuming it is stored in the way i send it)19:58
corvusi have little factual basis for making that assumption19:59
mordredcorvus: for now, if you make a len method on your object that just returns 020:00
mordredif should get you past your issue20:00
mordredfile_size is only used a little later to determine if file_size < max_segment_size ...20:01
mordredso if you return 0, the code pass data on through directly20:01
openstackgerritMerged openstack/os-service-types master: Switch to stestr  https://review.openstack.org/58534920:02
openstackgerritMerged openstack/os-service-types master: Updated from OpenStack Service Type Authority  https://review.openstack.org/58506220:02
openstackgerritMonty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data  https://review.openstack.org/58553220:03
mordredcorvus: ^^ also, I think that should at least do a workaround20:03
corvusmordred: though does it get passed through to the underlying session method?  i need that not to have a len method, or it'll avoid chunked uploads.20:03
corvusmordred: i'll try your patch out in a little bit20:04
mordredcorvus: ah.20:05
mordredcorvus: oh for the love of ...20:05
mordredcorvus: we don't use file_size in the data path at all20:06
openstackgerritMonty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data  https://review.openstack.org/58553220:07
mordredcorvus: ^^ that should fix it more appropriately20:08
corvusmordred: ack.  i'll give it a spin when i finish dealing with trailing slashes :)20:09
mordredcorvus: trailing slashes are the worst20:09
mordredcorvus: they're almost as terrible as INCORRECT WHITESPACE20:09
*** notmyname has joined #openstack-sdks20:09
notmynamemordred: hello20:09
mordredyay it's notmyname !20:09
mordrednotmyname: we have several questions - I will try to ask them in some semblance of order20:10
mordrednotmyname: first of all, if you are uploading a large object and you want to upload compressed with a deflate header - does the compressed or uncompressed size count towards max_file_size?20:11
corvusspecifically, "content-encoding: deflate" is what's happening here20:12
mordredyeah. corvus is likely to say smarter words than me20:12
notmynamethat is likely to be dependent on something between the client and swift itself20:14
notmynamethere's not anything in swift that will accept compressed data and store in uncompressed20:15
notmynamehowever, if you've got some caching thing (CDN or otherwise) that understands those headers, that work can be done there20:15
notmynameswift will happily store the content-encoding header, if you send it, and return it on a read request.20:16
notmynamelol, rackspace took my name off the author byline ;-) https://blog.rackspace.com/cloud-files-cdn-compresses-at-the-edge20:17
mordredok. so - what about SLO/DLO objects and concatenation? if the segments are uploaded compressed and swift doesnt' natively do any uncompression, I'm guessing that could get weird for the read?20:17
mordrednotmyname: haha20:18
mordredand by 'get weird' I mean 'not work"20:18
notmynamelet me try something...20:19
timburkei'd expect you'd want to open the large object, stream it through a compressor, and break out segments from that compressed stream. that way when you go to download the large object, you'll get a singular large, compressed stream20:21
timburkebreaking the large object into segments first then compressing is unlikely to end well20:23
notmynameoh hi timburke20:23
corvusmakes sense20:23
corvusunfortunately, we have to know ahead of time if we're going to upload a large object or a normal one yeah?20:23
notmynameyeah, what he said. "breaking the large object into segments first then compressing is unlikely to end well". and I just confirmed I wasn't forgetting something about this against a dev box20:24
mordredthen I guess you could put content-encoding: deflate header on the manifest object - and a browser would theoretically dtrt?20:25
notmynamecorvus: you can use an SLO even if the total object size is much less than a single "normal" object limit20:25
notmynamemordred: ya20:25
mordrednotmyname: oh. well that's certainly an interesting thought ...20:26
mordrednotmyname, timburke: the overall problem we're trying to solve is what to do with the intersection of openstacksdk transparently creating large objects for you and a user of openstacksdk wanting to pass in an interable that is a compressed stream20:26
mordredit seems like one way to deal with it might be to just always create a SLO if someone passes in an interable instead of a bytes or a filename20:27
timburkecorvus: depends on how many api requests you're willing to make :-) one option would be to always upload as a large object (like notmyname said) or upload one segment's worth to the base name, then do a server-side copy to the segment location once you realize you need a large object20:27
mordredooh. that second one sounds reasonable too20:27
corvusif it's not crazy to create a SLO when not strictly necessary, maybe we could make the decision based on the size of the uncompressed data.  so if it's > max size, go ahead and SLO even if it's not strictly necessary.... we'd still only do it for "big" files :)20:27
corvusor that second one.  :)20:27
notmynameis this for log files?20:28
notmynamefor the zuul jobs?20:28
mordrednotmyname: yup20:28
corvusnotmyname: for starters (so unlikely to hit it) but container/machine images probably aren't far behind.20:28
timburkeif you've got enough memory, you could buffer the first MB or so, if it all fits do it as a normal object; otherwise fall back to SLO20:29
notmynamethen in that case, I'd optimize for simpler client write path instead of optimal read latency20:29
notmynamesince these will be frequently written and rarely read20:29
notmynametimburke has the right idea20:29
notmyname.read(1024*1024) on the input, if you get the full MB, then do a SLO. if not, write a normal object20:30
corvusmemory is actually an issue; we could end up attempting a lot (hundreds?  many many hundreds?) of these simultaneously on a 8g vm20:31
timburkeno upload pooling? i feel like you'd probably be able to saturate your i/o with tens of workers rather than hundreds... but maybe this is getting into the need to have a simple client20:36
timburkeserver-side copy (or always SLO, all the time) may work out best20:37
notmynamecorvus: mordred: so the general answer is that swift will store the bytestream you send it and also headers that may have some definition for clients (eg content-encoding). SLOs aren't special in that the segments are simply slices of the resulting range. swift doesn't do any interpretation of the contents of objects20:38
mordredtimburke: yah - server-side copy or always SLO both sound like good general options20:39
mordredthere's definitely a balancing act we're trying to do here with wanting SDK to DTRT and yet also providing enough knobs so that we can do the zuul log upload thing efficiently20:39
mordrednotmyname, timburke: thanks both of you - this has been super helpful20:41
timburkefwiw, swiftclient opts for the buffering thing when uploading from stdin -- i think we go up to 16MB (or something like that?) then start uploading 16MB segments. since its stdin, there's only one upload per-process, so we don't feel too bad about the memory20:43
*** d0ugal has quit IRC20:44
mordredtimburke: yah - the fun part of this story is that once sdk switches to "oh, you wanted an SLO" - it does so with a pool of threads (similar to swiftuploader in swiftclient)20:49
mordredof course, actually ... now that I think about it - that won't work for iterators that don't have seek anyway20:49
mordredsince it does it in parallel for files by opening multiple handles and seeking on them ... so to support SLO from an input stream we'll need to reengineer what we're doing anyway20:50
mordredcorvus: ^^20:50
corvusthis may be a limited use case.  we won't want to use the deflatefilter for, say, already compressed images.  i've only got it set up to engage for text/ types with no encoding right now.20:51
mordredcorvus: ah - cool.20:51
mordredcorvus: I'm almost starting to feel like we should add your compressiong streaming code into sdk itself so that we can put it further down the stack20:52
mordredcorvus: like, put it around the file reads after the seek in the SLO segment uploads20:52
mordredand make an option to create_object "compress=False" or something like that (just thinking out loud)20:53
corvusmordred: i think it may be pretty domain-specific; i don't think it's appropriate for everything20:53
mordredgood point. oh - and also that would be the wrong place anyway20:53
notmynameeg https://github.com/openstack/swift/blob/master/swift/common/internal_client.py#L54 ?20:53
corvus(i actually anticipate some period of us tweaking when this gets used)20:53
openstackgerritMerged openstack/os-service-types master: Allow passing in service types with _ in them  https://review.openstack.org/58541020:54
corvusnotmyname: why in the world isn't that in the standard library? :)20:54
corvusnotmyname: i wrote one of those too.  i may improve it now :)20:55
mordredcorvus: it seems like the 'don't run len on data' patch from above should be the only thing you'd need for the easier case - and that we can probably wait until later to deal with SLO and streamed input20:58
corvusmordred: your patch works.  though using that method as opposed to the proxy results in a HEAD request to the container before each PUT21:13
mordredcorvus: yeah. we should really cache that container status21:19
corvusmordred: that's probably going to be an extra 1500 requests for, say, a devstack job, yeah?21:20
corvuswhat's it for?21:20
mordredcorvus: create_object will create the container for you if it doesn't exist21:20
mordredhead is checking container existence21:20
corvusmordred: i want to set some things on the container when it's created; i assumed it'd be best for me to handle that explicitly before doing the upload21:22
mordredcorvus: yes - it's totally best for you to do that21:22
mordredcorvus: is this multiple executions or a single program with a single long-lived session?21:23
corvusmordred: a single program with 30 threads uploading in parallel21:23
mordredcorvus: hrm. there is a container cache already - it seems like it should only make one HEAD21:24
corvusmordred: my test isn't big enough to re-use a thread.  maybe they're all racing to get the first head21:24
mordredprobably so. we could make the container cache more threadsafe though21:24
corvusmordred: i'm not yet performing my existence check -- would that prime the cache?21:25
mordredsimilar to how we do for servers21:25
mordredwill do it21:25
corvuscool, then we may not need to do anything to resolve this.  i'll plumb that code in now.21:25
corvusmordred: yep that took care of it.  sorry for the false alarm :)21:28
*** gildub has joined #openstack-sdks21:28
*** slaweq has quit IRC21:30
*** edmondsw has quit IRC21:34
*** gildub has quit IRC21:37
*** notmyname has left #openstack-sdks21:43
*** JudeC_ has joined #openstack-sdks21:44
*** JudeC has quit IRC21:44
openstackgerritMerged openstack/python-openstackclient master: Fix error with image show when image name is None  https://review.openstack.org/52946421:45
*** cdent has quit IRC21:45
*** cdent has joined #openstack-sdks22:06
*** slaweq has joined #openstack-sdks22:11
*** cdent has quit IRC22:14
*** slaweq has quit IRC22:16
openstackgerritDean Troyer proposed openstack/python-openstackclient master: Support --community in openstack image list  https://review.openstack.org/56515222:48
*** slaweq has joined #openstack-sdks22:50
*** slaweq has quit IRC22:54
mordreddtroyer: https://review.openstack.org/#/c/582334 <-- I left a novel of a comment there - I'm guessing we should probably just land it in that form since it's definitely an error it's fixing and leave making it more better to next cycle ...23:10
mordreddtroyer: but whatever you think23:11
*** slaweq has joined #openstack-sdks23:11
*** slaweq has quit IRC23:16
corvusmordred: you said i could pass a dict to the connection constructor... how's that go again?23:17
corvusjust openstack.connect(**args) ?23:18
corvushrm, that gets me: keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url23:18
dtroyermordred: agreed, there are a number of microversion-affected changes to compute outstanding, some merged already, all of them are likely in this situation, this gets things working to some extent, we'll clean it up when we SDK-enable things and that may influence the order we do that work in...23:19
corvusmordred: oh apparently that's because profile is ignored, so i need to add auth_url in there23:26
corvusmordred: reading https://docs.openstack.org/openstacksdk/latest/user/transition_from_profile.html it's not clear how to accomplish what i want (construct a connection from a dict, using profile=vexxhost and not specifying auth_url)23:28
*** gildub has joined #openstack-sdks23:50
*** yolanda_ has joined #openstack-sdks23:56
*** yolanda has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!