Wednesday, 2018-05-16

*** germs_ has quit IRC00:48
*** gyee has quit IRC01:38
spsuryaHello all01:46
spsuryais there any automated tool to install SAIO ?01:47
spsuryarather than following SAIO deployment guide01:47
*** gyankum has joined #openstack-swift01:55
*** bkopilov has quit IRC02:00
notmynamespsurya: are you wanting to build a developer einvironment or for a testing target?02:06
spsuryanotmyname:  for POC  and testing02:07
spsuryabut quick one02:07
spsuryaI know SAIO is more about dev purpose02:07
notmynamewhat do you mean by POC? are you testing that an app works when talking to swift or are you testing a deployment to see if it will work at your scale?02:07
spsuryabefore that i have a query02:10
notmynameyou could be wanting to contribute code to swift. you could be wanting to practice with swift from a sysadmin perspective. you could be wanting to test an app to see if it works against swift. depending on which of these three categories you fall in, I can point you in different directions02:10
spsuryacan i want to run swift existing account,container and object  into container02:12
notmynameI'm sorry, but I don't understand what you mean02:13
spsuryai want to migrate swift services into container02:13
spsuryai.e operational swift into container02:13
spsuryathats is my goal02:14
spsuryabut to reach there i am doing small tests of swift02:14
spsuryanotmyname: hope you get my point02:15
notmynamenot completely, yet. by container you mean docker containers? lxc?02:15
spsuryanotmyname: docker container02:15
spsuryanotmyname: lemme explain02:16
spsuryasuppose you have installed swift on two node and each one running the application account, contqainer and object and proxy too02:17
notmynameok02:17
spsuryaas any operator add new node to swift cluster then after rebalancing new node would be part of that cluster02:18
notmynameright02:18
spsuryanow rather than adding node to swift cluster i want to add container in swift cluster02:18
spsuryaand now that container would be serving all the purpose of swift services02:19
spsuryaso container would be running account, container and object service with old data of swift02:20
spsuryathis is what i want to do at the moment02:20
spsuryabut before that also wanted to install SAIO for dev just to few things02:21
spsuryaHope you got my point ^^02:21
spsuryanotmyname:02:21
notmynamewhat have you found or tried so far?02:22
spsuryanotmyname: i want to find the configuration point where we configure Swift to add new node02:24
spsuryacan you show me the point in swift02:24
notmynamethat doesn't have anything to do with docker...02:25
notmynamenew nodes are added by adding drives to a ring02:25
notmynamenormally via `swift-ring-builder`02:25
spsuryanotmyname: yes, we add drive to ring02:25
spsuryaso wanted to add swift container docker/volume  to swift-ring file02:26
spsuryawe can check if more configuration or support required for thisd02:26
notmynamedevices in a ring are identified by IP, port, and mount point. (as an operator, you also give it a zone and region based on your cluster layout)02:27
spsuryaso notmyname i am testing all these point02:27
notmynamethat doesn't have anything to do with how you choose to package the code (docker or otherwise)02:27
spsuryayes in docker by some tooling02:28
spsuryajust trying to do this02:29
notmynameI'm not sure exactly what you're wanting, but I'm sure you're getting frustrated with me not understanding. here's a project that runs some swift processes in a docker container: https://github.com/bouncestorage/docker-swift02:29
spsuryanotmyname:02:29
notmynameI know of zero people who run swift in production in this way02:29
notmynamethat project is only one way to do things. and it's not designed to do *any* clustering at all02:29
notmynamehow you package and deploy the linux processes does't matter. use docker. use distro packages. build your own from source. for any way, you'll end up with a set of hard drives across all your machines that are each identified by an (IP, port, mountpoint). add that info to your ring files, deploy your rings, and the running swift processes will take care of the rest02:31
notmynamethat's not to say there aren't any hard problems with docker-izing swift. that's still a thing to do. you can look at the project above to help you figure that out.02:33
spsuryanotmyname: thanks for info02:34
*** gkadam has joined #openstack-swift02:36
*** klamath has quit IRC02:36
spsuryanotmyname: will you be at Summit ?02:38
notmynameI will, but not the whole week. I'll be there tuesday afternoon through thursday02:39
spsuryanotmyname: nice02:45
*** psachin has joined #openstack-swift02:50
spsuryahttps://www.irccloud.com/pastebin/55liTRcJ/02:57
spsuryanotmyname:02:57
spsuryafailing on $$$$ cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd -03:03
*** gkadam has quit IRC03:07
*** gyankum has quit IRC03:20
*** bkopilov has joined #openstack-swift03:23
*** links has joined #openstack-swift03:41
*** klamath has joined #openstack-swift03:51
*** klamath has quit IRC04:07
*** gyankum has joined #openstack-swift04:10
*** cshastri has joined #openstack-swift04:59
kota_good afternoon05:36
*** armaan has joined #openstack-swift05:47
*** psachin has quit IRC06:03
*** armaan has quit IRC06:10
*** armaan has joined #openstack-swift06:10
*** psachin has joined #openstack-swift06:11
*** silor has joined #openstack-swift06:19
*** klamath has joined #openstack-swift06:39
*** klamath has quit IRC06:44
*** ccamacho has joined #openstack-swift07:08
*** rcernin has quit IRC07:10
*** geaaru has joined #openstack-swift07:17
*** tesseract has joined #openstack-swift07:17
*** mikecmpbll has joined #openstack-swift07:54
*** armaan has quit IRC08:06
*** armaan has joined #openstack-swift08:07
*** armaan has quit IRC08:11
*** armaan has joined #openstack-swift08:11
*** zaitcev has quit IRC08:12
acolesgood morning08:17
*** cbartz has joined #openstack-swift08:22
*** kei_yama has quit IRC08:44
*** gkadam has joined #openstack-swift08:53
*** links has quit IRC09:07
*** jistr has quit IRC09:09
*** jistr has joined #openstack-swift09:12
*** links has joined #openstack-swift09:23
yuxin_o/09:28
yuxin_good afternoon09:28
yuxin_I am writing unit tests for swift3, and I need to patch swift3.request.get_container_info. But there are two places calling get_container_info in swift3/requet.py with different args (with[1] and without[2] swift_source kwarg).09:28
yuxin_[1]https://github.com/openstack/swift3/blob/master/swift3/request.py#L110309:29
yuxin_[2]https://github.com/openstack/swift3/blob/master/swift3/request.py#L125409:29
yuxin_When it patched with `lambda x, y: {}`, it raised an error said `lambda() got an unexpected keyword argument 'swift_source'`. can anyone help me out?09:29
acolesyuxin_: try `lambda x, y, **kwargs: {}`, or if you don't care about the args in the lambda, `lambda *args, **kwargs: {}`09:40
yuxin_acoles: thanks, acloes! it works.09:43
*** silor1 has joined #openstack-swift09:58
*** silor has quit IRC10:00
*** silor1 is now known as silor10:00
*** bkopilov has quit IRC10:05
kota_yuxin_: note that swift3 is now a part of swift upstream as "s3api". if you're thinking of using the newest master, you can consider to fix the Swift's one.10:20
kota_yuxin_: around https://github.com/openstack/swift/blob/master/swift/common/middleware/s3api/s3request.py10:20
* kota_ has been tied up to fix my presentation slides for OpenStack Summit in the next week to get approval from my boss :/10:21
kota_i hope i can work on reviewing deep container since tomorrow, it might be... too late???10:22
*** corvus has quit IRC10:29
*** corvus has joined #openstack-swift10:30
*** notmyname has quit IRC10:33
*** notmyname has joined #openstack-swift10:34
*** ChanServ sets mode: +v notmyname10:34
acoleskota_: the first few patches in the chain have had a lot of review but it would be great if you get chance to look at the later ones10:41
*** klamath has joined #openstack-swift10:41
kota_acoles: ok, thanks for the info!10:43
*** links has quit IRC10:44
*** klamath has quit IRC10:46
*** links has joined #openstack-swift10:57
remix_tjjust for curiosity... how does internal swift communication happens? how does proxy communicates with account/container/object server? Using http queries?11:07
yuxin_kota_: cool.11:08
yuxin_By the way, I am working on swift3 object versioning and wonder if anyone still focusing on this feature. I've got some ideas on this and would like to get my works submitted.11:08
acolesremix_tj: yes, http is used between internal services11:10
remix_tjso i expect to see queries to the object server requesting to do something on the object?11:10
acolesremix_tj: yes, proxy sends http requests to object server(s) that host the particular object11:12
remix_tji'm trying to design a WORM middleware for trying to replace retention features that EMC Centera provides (file cannot be updated or deleted before the expiration of a given retention time, SEC Rule 17a-4)11:14
remix_tjIt's just for playing with some python code11:14
remix_tji think the best option is to write a filter that denies some operations11:16
*** armaan has quit IRC11:26
*** armaan has joined #openstack-swift11:27
openstackgerritNguyen Hai proposed openstack/python-swiftclient master: Change python-swiftclient webpage to openstackdocstheme  https://review.openstack.org/55343311:37
*** gkadam has quit IRC11:43
*** gkadam has joined #openstack-swift11:43
*** vinsh_ has joined #openstack-swift12:32
*** vinsh has quit IRC12:32
*** armaan has quit IRC12:41
*** klamath has joined #openstack-swift12:42
*** armaan has joined #openstack-swift12:42
*** klamath has quit IRC12:46
*** armaan has quit IRC12:50
*** armaan has joined #openstack-swift12:50
openstackgerritNguyen Hai proposed openstack/python-swiftclient master: Switch from oslosphinx to openstackdocstheme  https://review.openstack.org/55343312:57
*** mikecmpbll has quit IRC13:30
*** cshastri has quit IRC13:38
*** klamath has joined #openstack-swift13:38
*** vinsh_ has quit IRC13:38
*** vinsh has joined #openstack-swift13:39
*** mikecmpbll has joined #openstack-swift13:44
*** bkopilov has joined #openstack-swift13:47
*** mvk has quit IRC13:52
openstackgerritTovin Seven proposed openstack/swift master: OSprofiler in OpenStack Swift  https://review.openstack.org/46831613:57
*** cbartz has quit IRC13:58
*** mikecmpb_ has joined #openstack-swift14:01
*** mikecmpbll has quit IRC14:02
tdasilvayuxin_: great to hear your interest on swift3, but as kota_ mentioned, I'd make sure to send any patches to s3api and yeah, object versioning is difentely welcome14:09
*** gyankum has quit IRC14:09
tdasilvaremix_tj: WORM functionality is one of those requests that always come up, the real challenge with WORM in swift is eventual consistency, preventing overwrites is very difficult14:10
*** zaitcev has joined #openstack-swift14:11
*** ChanServ sets mode: +v zaitcev14:11
tdasilvaremix_tj: and thanks for the PRs on ansible-saio, will look at those today.14:12
remix_tjtdasilva: i'm trying to think like centera does. In centera once you wrote an object and set the retention, you're denied to do any edit/delete operations. I understand the issue of eventual consistency and i'll try to take in account, but i'll first start with a filter14:13
remix_tjwe've several applications requiring to store data for an indefinite time ensuring that no changes will happen to those data14:13
remix_tjup to now only some propretary solutions does this and in several cases it's done only with a flag on the data container saying that nothing can be deleted or changed before X time14:15
remix_tjanyway it's an excercise :-)14:16
tdasilvaremix_tj: one way to achieve is with object versioning, that way even if an object got overwritten you were able to maintain an old copy14:17
tdasilvabut i realize that not really WORM functionality like centera does14:18
remix_tjno, because versioning means that at time t0 i'll have the good content, after the update with a request i'll get the updated content that is not what i require14:19
remix_tj*requested14:19
remix_tji require to have the original content served at every time.14:20
remix_tji have to check on enterprise storages what happens on retained objects if i try to update their content.14:20
DHEso you need eventually consistent with the OLDEST copy served?14:21
remix_tjDHE: indeed i require to have only one version of data14:22
remix_tjin centera if i write an object it gets written with new identifier, so is not possible to overwrite the existing content. That's due to the nature of that kind of storage. But i've seen that EMC implemented the same retention on S3 buckets, so i'll try to explore the behaviour and then test14:25
*** germs has joined #openstack-swift14:27
*** germs has quit IRC14:27
*** germs has joined #openstack-swift14:27
*** mvk has joined #openstack-swift14:29
tdasilvaremix_tj: ack...please do keep us in the loop14:30
remix_tjsure14:30
remix_tjtdasilva: i'll start with the simplest thing which is filtering delete requests, when i'll move on more complex things :-)14:32
*** germs has quit IRC14:32
remix_tjis there any blueprint about it on launchpad?14:33
notmynamepreventing deletes isnt' too hard. preventing overwrites is very very difficult14:33
tdasilvaremix_tj: i believe the centera approach could be replicated. i.e., give a UID to an object (probably the object url itself), but then keep that list somewhere that is strong consistent14:34
tdasilvawas just having this conversation with notmyname yesterday...14:34
remix_tjtdasilva: on centera if you write an object you obtain an ID. If you loose the ID there's only one way to recover data: obtain the list of all the stored data and look into the metadata to see if matches with what you're looking at14:35
remix_tjmetadata of each stored object14:36
notmynamefortunately, swift doesn't work that way ;-)14:37
remix_tjyeah, fortunately :-)14:38
notmynamereminder: the september PTG tickets early-bird pricing ends tomorrow. register now, if you haven't already14:43
notmynamehttps://www.eventbrite.com/e/project-teams-gathering-denver-2018-tickets-4529670366014:43
*** armaan has quit IRC14:47
remix_tji'm "very happy" i can't be at berlin summit this year. I think i'll get ATC discount code, but my wife should give birth to my first son/daughter during the summit, so  i'll skip and wait for another opportunity in my zone :-)14:47
*** cbartz has joined #openstack-swift14:54
*** links has quit IRC14:58
*** ukaynar has joined #openstack-swift15:00
tdasilvaremix_tj: congrats! that is a good excuse indeed :)15:08
remix_tj:-)15:09
*** cshastri has joined #openstack-swift15:30
*** gyankum has joined #openstack-swift15:39
claygnotmyname: I always assumed a storage policy with a different cleanup hashdir could just prefer first timestamp - done.16:00
*** itlinux has joined #openstack-swift16:05
timburkegood morning16:11
notmynameclayg: interesting16:24
*** germs has joined #openstack-swift16:28
*** germs has quit IRC16:28
*** germs has joined #openstack-swift16:28
*** gyee has joined #openstack-swift16:29
*** germs has quit IRC16:32
*** itlinux has quit IRC16:36
*** ccamacho has quit IRC16:37
*** gkadam has quit IRC16:46
*** links has joined #openstack-swift16:53
*** germs has joined #openstack-swift17:04
*** germs has quit IRC17:04
*** germs has joined #openstack-swift17:05
*** germs has quit IRC17:05
*** germs has joined #openstack-swift17:05
*** germs has quit IRC17:05
*** germs has joined #openstack-swift17:06
*** germs has quit IRC17:06
*** SkyRocknRoll has joined #openstack-swift17:07
*** germs has joined #openstack-swift17:07
*** itlinux has joined #openstack-swift17:12
*** psachin has quit IRC17:14
*** links has quit IRC17:16
*** SkyRocknRoll has quit IRC17:20
*** mikecmpb_ has quit IRC17:23
*** tesseract has quit IRC17:27
*** mvk has quit IRC17:27
*** SkyRocknRoll has joined #openstack-swift17:33
*** SkyRocknRoll has quit IRC17:35
*** cshastri has quit IRC17:38
*** NM has joined #openstack-swift17:40
*** mikecmpbll has joined #openstack-swift17:46
*** cbartz has quit IRC17:53
openstackgerritTim Burke proposed openstack/python-swiftclient master: Back out some version bumps  https://review.openstack.org/56891417:54
*** NM has quit IRC18:00
*** NM has joined #openstack-swift18:01
*** NM has quit IRC18:01
*** NM has joined #openstack-swift18:03
*** NM has quit IRC18:05
*** geaaru has quit IRC18:13
openstackgerritTim Burke proposed openstack/python-swiftclient master: Back out some version bumps  https://review.openstack.org/56891418:15
openstackgerritTim Burke proposed openstack/python-swiftclient master: Back out some version bumps  https://review.openstack.org/56891418:16
*** NM has joined #openstack-swift18:16
*** eandersson has joined #openstack-swift18:17
eanderssonIs this the recommended way to provide region_name for Swift? https://review.openstack.org/#/c/568921/1/zaqar/storage/swift/driver.py18:18
patchbotpatch 568921 - zaqar - [WIP] Adding Swift backend Region support18:18
notmynameeandersson: that doesn't look right18:19
eanderssonDo you have an example? I was looking at this https://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L156118:19
notmynamehang on. maybe it is. checking18:19
notmynameyes. that's the way to pass a region name to keystone for auth18:21
eanderssoncool thanks notmyname18:21
notmynameI see some references to os_region_name, too. TBH, I'm not entirely sure ont he difference18:21
eanderssonI might follow up with an update to the Swift examples after this :P18:22
notmynamecool, thanks!18:22
*** gyankum has quit IRC18:24
DHEswift regions and openstack regions don't align. openstack regions will provide different URLs to different services depending on regions. hence openstack directs you to a local proxy server, but that's different from storage regions18:27
openstackgerritTim Burke proposed openstack/python-swiftclient master: Back out some version bumps  https://review.openstack.org/56891418:29
eanderssonYea - in this case the issue was that it was using the first keystone catalog endpoint.18:30
eanderssonWhich lead it to connect to a Swift backend in a different region.18:30
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: Add support for sharding in ContainerBroker  https://review.openstack.org/56574418:47
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: Add shard range support to container server  https://review.openstack.org/56574518:47
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: Enable proxy to build listings from shards  https://review.openstack.org/56574618:47
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: Redirect object updates to shard containers  https://review.openstack.org/56574718:47
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: Add sharder daemon, manage_shard_ranges tool and probe tests  https://review.openstack.org/56574818:47
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: Add container sharding documentation  https://review.openstack.org/56574918:47
openstackgerritAlistair Coles proposed openstack/swift feature/deep-review: SQUASH: handle delimiter listings in proxy  https://review.openstack.org/56893018:47
acolestimburke: sorry I got bogged down with some tests today and didn't get through all your review comments today so that's just what I have...feel free to push new patchsets if you want, or I'll crack on tomorrow.18:49
*** mvk has joined #openstack-swift19:22
*** NM has quit IRC19:51
*** NM has joined #openstack-swift19:54
*** silor has quit IRC19:58
*** NM1 has joined #openstack-swift20:03
*** NM has quit IRC20:03
timburkedependencies are the worst. when i did https://github.com/openstack/python-swiftclient/commit/ff505d7 i made a liar of our impressively-low requests>=1.1 requirement, 'cause the exception came in with https://github.com/requests/requests/commit/811ee4e (so, requests 2.4.0)20:19
timburkealso, http://logs.openstack.org/14/568914/4/check/openstack-tox-lower-constraints/bca39dc/job-output.txt.gz#_2018-05-16_18_34_43_901554 is some pretty *horrible* test-failure output20:21
*** ukaynar has quit IRC20:28
notmynameswift team meeting in 10 minutes20:50
claygwooooooo meetings!20:50
notmynameI didn't get re-banned in -meeting, so we'll have it in #openstack-meeting ;-)20:50
notmynamethis week we'll talk about feature/deep-review and specifically when to merge it20:51
*** m_kazuhiro has joined #openstack-swift20:52
*** NM1 has quit IRC20:53
*** rledisez1 has joined #openstack-swift20:55
*** rledisez1 has quit IRC20:57
kota_good morning20:58
*** alecuyer has quit IRC20:58
*** rledisez has quit IRC20:58
*** rledisez has joined #openstack-swift20:58
notmynamemeeting time21:00
claygwooooo!21:02
*** itlinux has quit IRC21:13
*** NM has joined #openstack-swift21:23
zaitcevacoles: is there a comment or a function with evident code that explains how params['marker'] += '\x00' works in feature/deep?21:25
zaitcevoh, wait. Meeting21:25
timburkeis this about making end_marker inclusive? tacking on a null byte gets you the very next possible string21:29
acoleszaitcev: timburke I'll need to reload context some more but IIRC it's to do with shard range upper being inclusive but end marker not being inclusive21:33
*** d0ugal_ has joined #openstack-swift21:39
*** d0ugal has quit IRC21:41
*** itlinux has joined #openstack-swift21:51
*** NM has quit IRC22:11
*** portante has quit IRC22:15
*** ndk- has quit IRC22:16
*** rcernin has joined #openstack-swift22:25
*** rledisez has quit IRC22:26
*** alecuyer has joined #openstack-swift22:26
*** rledisez has joined #openstack-swift22:27
*** m_kazuhiro has quit IRC22:27
*** itlinux has quit IRC22:33
*** portante has joined #openstack-swift22:35
*** ndk_ has joined #openstack-swift22:36
*** d0ugal__ has joined #openstack-swift22:40
*** d0ugal_ has quit IRC22:41
claygnotmyname: epic meeting bruh22:44
eanderssonbtw the region code is not working as intended22:49
eanderssonhttps://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L64722:49
eanderssonThat line should be passing on region_name from os_options22:50
eanderssonHad to do it like this https://review.openstack.org/#/c/568921/3/zaqar/storage/swift/driver.py22:56
patchbotpatch 568921 - zaqar - Adding Swift backend Region support22:56
tdasilvasorry i missed missed, just caught up reading scroll back! go container sharding! go acoles !22:58
openstackgerritErik Olof Gunnar Andersson proposed openstack/python-swiftclient master: Make swiftclient respect region_name when using sessions  https://review.openstack.org/56897923:04
eandersson^ should fix the region issue23:07
notmynamethanks23:09
eanderssonNo more using the endpoint for China when trying to access Swift from the US :D23:10
*** ndk_ has quit IRC23:17
*** portante has quit IRC23:18
*** ndk_ has joined #openstack-swift23:20
*** portante has joined #openstack-swift23:21
*** kei_yama has joined #openstack-swift23:21
*** germs has quit IRC23:33
*** germs has joined #openstack-swift23:35
*** germs has quit IRC23:35
*** germs has joined #openstack-swift23:35

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!