Thursday, 2017-03-02

*** dja has joined #openstack-swift00:00
notmynametimburke: not having looked at this in a while, it does re-raise the question in my mind: why do we have *any* aws auth stuff in tempauth in the first place?00:06
notmyname"hysterical raisons" I'm sure00:07
timburkenotmyname: that's my assumption. i suppose it demonstrates how other middleware authors may want to implement their middleware?00:08
notmynamelol, right. "example code". not at all because swift3 used to be in the repo, and when we pulled it out we didn't cleanly extract the authy bits from the auth system we don't care much about00:09
*** _JZ_ has quit IRC00:11
*** dja has quit IRC00:12
claygacoles: so in ssync, self.df_mgr.yield_hashes is quite happen to have be given a list of suffixes which contain only empty hash dirs00:35
*** dja has joined #openstack-swift00:35
claygit's not cleanup_ondisk_files (or any of the methods that sheperd around bags of filenames or  ondisk_info) that detects and reaps empty hashdirs - it's _hash_suffix_dir00:36
*** sams-gleb has joined #openstack-swift00:52
*** sams-gleb has quit IRC00:56
acolesclayg: so it might be interesting to consider whether this code could actually move into cleanup_ondisk_files?? https://github.com/openstack/swift/blob/9be1d8ba28f9702e8e0e5024fd12f367a18506c4/swift/obj/diskfile.py#L983-L98800:57
*** catintheroof has joined #openstack-swift01:04
*** jamielennox is now known as jamielennox|away01:16
tdasilvahttps://medium.com/@jim_dowling/reflections-on-s3s-architectural-flaws-71f14c05a5fa#.5y4ap4jup01:19
tdasilvainteresting to start reading something that you might think might be an unbiased review only to then find out at the end that it's all a setup to introduce my project01:19
*** catintheroof has quit IRC02:01
*** JimCheung has joined #openstack-swift02:11
claygacoles: *gasp* very interesting indeed!02:13
*** klrmn has quit IRC02:13
claygtdasilva: i started to look at that the other day and didn't make it too far02:14
kota_morning again at office02:40
kota_it seems to sleep too much02:40
*** tonanhngo has joined #openstack-swift02:42
*** tonanhngo has quit IRC02:46
claygkota_: I think this is g2g - the py35 experimental check passes -> https://review.openstack.org/#/c/43877002:52
patchbotpatch 438770 - pyeclib - Don't let nose play with our PYTHONPATH.02:52
*** sams-gleb has joined #openstack-swift02:54
*** _JZ_ has joined #openstack-swift02:58
*** sams-gleb has quit IRC02:59
*** dmorita_ has quit IRC03:03
*** links has joined #openstack-swift03:08
kota_clayg: alright, it looks like you and timburke had an answer to reproduce it yeah?03:16
kota_will look at03:17
*** JimCheung has quit IRC03:23
*** JimCheung has joined #openstack-swift03:24
*** JimCheung has quit IRC03:25
*** JimCheung has joined #openstack-swift03:25
*** JimCheung has quit IRC03:30
*** klrmn has joined #openstack-swift03:50
*** rcernin has quit IRC03:50
*** klrmn has quit IRC03:52
*** dmorita has joined #openstack-swift04:08
*** dmorita has quit IRC04:10
*** dmorita has joined #openstack-swift04:10
*** dmorita_ has joined #openstack-swift04:11
*** dmorita has quit IRC04:11
*** dmorita has joined #openstack-swift04:15
*** dmorita_ has quit IRC04:15
*** dmorita_ has joined #openstack-swift04:17
*** _JZ_ has quit IRC04:19
*** dmorita has quit IRC04:19
*** dmorita_ has quit IRC04:21
*** dmorita has joined #openstack-swift04:24
*** dmorita has quit IRC04:27
*** psachin has joined #openstack-swift04:27
*** dmorita has joined #openstack-swift04:27
kota_ok, i got the same error with the way to reproduce timburke reported04:29
*** mmotiani_ has joined #openstack-swift04:31
*** dmorita has quit IRC04:31
*** dmorita has joined #openstack-swift04:32
*** Jeffrey4l has joined #openstack-swift04:32
*** dmorita has quit IRC04:36
*** dmorita has joined #openstack-swift04:36
*** dmorita_ has joined #openstack-swift04:37
*** dmorita has quit IRC04:37
*** dmorita_ has quit IRC04:42
*** klrmn has joined #openstack-swift04:44
claygkota_: timburke is awesome04:46
claygtdasilva: do you like diffs, follow-ups, or push-overs?04:46
openstackgerritClay Gerrard proposed openstack/swift master: refactor some common code from crypto  https://review.openstack.org/44002804:50
*** klrmn has quit IRC04:53
*** sams-gleb has joined #openstack-swift04:57
*** dmorita has joined #openstack-swift04:59
*** dmorita_ has joined #openstack-swift05:03
*** sams-gleb has quit IRC05:03
*** dmorita has quit IRC05:04
*** dmorita_ has quit IRC05:07
*** dmorita has joined #openstack-swift05:13
*** dmorita has quit IRC05:13
*** dmorita has joined #openstack-swift05:14
*** dmorita_ has joined #openstack-swift05:15
*** dmorita has quit IRC05:15
*** dmorita_ has quit IRC05:18
*** adriant_ has quit IRC05:20
*** dja has quit IRC05:28
*** mmotiani_ has quit IRC05:54
*** dja has joined #openstack-swift05:56
openstackgerritMatthew Oliver proposed openstack/swift master: Add container sharding to Swift containers  https://review.openstack.org/42390606:00
*** dja has quit IRC06:07
*** psachin_ has joined #openstack-swift06:18
*** psachin_ has quit IRC06:19
*** dja has joined #openstack-swift06:26
*** JimCheung has joined #openstack-swift06:59
*** sams-gleb has joined #openstack-swift07:01
*** McMurlock1 has joined #openstack-swift07:03
*** ChubYann has quit IRC07:04
*** sams-gleb has quit IRC07:05
*** dmorita has joined #openstack-swift07:09
*** dmorita has quit IRC07:14
*** rcernin has joined #openstack-swift07:18
*** tesseract has joined #openstack-swift07:18
*** JimCheung has quit IRC07:32
*** zhugaoxiao has joined #openstack-swift07:34
*** hseipp has joined #openstack-swift07:43
*** winggundamth has joined #openstack-swift08:10
*** pcaruana has joined #openstack-swift08:13
*** sams-gleb has joined #openstack-swift08:16
*** bob_cheesey has joined #openstack-swift08:18
*** geaaru has joined #openstack-swift08:24
*** jordanP has joined #openstack-swift08:46
*** hseipp has quit IRC08:52
*** dja has quit IRC09:02
*** SkyRocknRoll has joined #openstack-swift09:11
*** SkyRocknRoll has quit IRC09:11
*** dmorita has joined #openstack-swift09:12
*** amoralej|off is now known as amoralej09:13
*** cbartz has joined #openstack-swift09:13
*** dmorita has quit IRC09:17
*** oshritf has joined #openstack-swift09:19
*** dja has joined #openstack-swift09:23
*** Jeffrey4l has quit IRC09:25
*** Jeffrey4l has joined #openstack-swift09:25
*** dja has quit IRC09:29
*** Jeffrey4l has quit IRC09:31
*** cshastri has joined #openstack-swift09:34
*** dmorita has joined #openstack-swift09:39
*** kei_yama has quit IRC09:39
*** bkopilov has joined #openstack-swift09:42
*** dmorita_ has joined #openstack-swift09:42
*** dmorita__ has joined #openstack-swift09:43
*** dmorita_ has quit IRC09:43
*** Jeffrey4l has joined #openstack-swift09:43
*** dmorita has quit IRC09:44
*** dmorita__ has quit IRC09:47
*** dmorita has joined #openstack-swift09:48
*** oshritf has quit IRC09:49
openstackgerritChristopher Bartz proposed openstack/python-swiftclient master: ISO 8601 timestamps for tempurl  https://review.openstack.org/42337709:52
*** dmorita has quit IRC09:52
openstackgerritRico Lin proposed openstack/swift master: [Fix gate]Update test requirement  https://review.openstack.org/44027409:55
openstackgerritRico Lin proposed openstack/python-swiftclient master: [Fix gate]Update test requirement  https://review.openstack.org/44027709:56
*** jordanP has quit IRC09:57
openstackgerritRico Lin proposed openstack/swift-bench master: [Fix gate]Update test requirement  https://review.openstack.org/44028210:01
*** oshritf has joined #openstack-swift10:06
*** bob_cheesey has quit IRC10:25
*** sams-gle_ has joined #openstack-swift10:27
*** sams-gleb has quit IRC10:27
*** links has quit IRC10:31
openstackgerritMerged openstack/pyeclib master: Don't let nose play with our PYTHONPATH.  https://review.openstack.org/43877010:33
*** bob_cheesey has joined #openstack-swift10:46
*** links has joined #openstack-swift10:47
*** mvk has quit IRC10:57
*** gcb has quit IRC10:57
*** bob_cheesey has quit IRC11:13
*** oshritf has quit IRC11:13
*** oshritf has joined #openstack-swift11:14
*** sams-gle_ has quit IRC11:17
*** bob_cheesey has joined #openstack-swift11:17
*** sams-gleb has joined #openstack-swift11:17
*** bob_cheesey has quit IRC11:18
*** oshritf has quit IRC11:20
*** sams-gleb has quit IRC11:22
*** klrmn has joined #openstack-swift11:22
*** klrmn1 has joined #openstack-swift11:25
*** klrmn has quit IRC11:26
*** mvk has joined #openstack-swift11:28
*** links has quit IRC11:51
*** klrmn1 has quit IRC11:55
kota_oops? patchbot is absent?11:58
kota_patch 44046611:59
patchbothttps://review.openstack.org/#/c/440466/ - swift - Draft: TestObjController refactoring11:59
kota_hello patch bot but maybe something is in trouble from gerrit notification11:59
kota_acoles: patch 440466 is for the homework from Atlanta PTG. to refactor the structure of TestObjController and ECTestMixin.12:01
patchbothttps://review.openstack.org/#/c/440466/ - swift - Draft: TestObjController refactoring12:01
*** sams-gleb has joined #openstack-swift12:02
kota_acoles: it's still WIP because I have notes to understand the test classes relations but if you could quick check the way I'd try to go, it'll super helpful because it will be large diff anyway.12:03
*** links has joined #openstack-swift12:04
*** dja has joined #openstack-swift12:07
*** winggundamth has quit IRC12:09
*** bkopilov has quit IRC12:10
*** gkadam has joined #openstack-swift12:13
*** oshritf has joined #openstack-swift12:26
*** oshritf has quit IRC12:32
*** oshritf has joined #openstack-swift12:33
*** oshritf has quit IRC12:38
*** oshritf has joined #openstack-swift12:38
*** gkadam has quit IRC12:46
*** catintheroof has joined #openstack-swift12:56
*** oshritf has quit IRC13:04
*** cshastri has quit IRC13:07
*** jlwhite has quit IRC13:15
*** jlwhite has joined #openstack-swift13:15
*** oshritf has joined #openstack-swift13:17
*** dja has quit IRC13:29
*** klamath has joined #openstack-swift13:31
*** klamath has quit IRC13:32
*** klamath has joined #openstack-swift13:32
*** chlong_ has joined #openstack-swift13:40
jrichlimathiasb: thanks for setting up the devstack!13:45
*** Jeffrey4l has quit IRC13:49
mathiasbjrichli: yw!13:52
*** mvk has quit IRC13:52
*** oshritf has quit IRC13:59
*** jith has quit IRC14:07
*** links has quit IRC14:12
*** dmorita has joined #openstack-swift14:23
*** dmorita has quit IRC14:23
*** dmorita has joined #openstack-swift14:24
*** Jeffrey4l has joined #openstack-swift14:29
*** Jeffrey4l has quit IRC14:35
*** _JZ_ has joined #openstack-swift14:44
*** _JZ_ has quit IRC14:44
*** gabor_antal_km has joined #openstack-swift14:50
*** ujjain has quit IRC14:50
*** gabor_antal_ has quit IRC14:51
*** sams-gleb has quit IRC14:53
*** mvk has joined #openstack-swift14:56
jrichlioh - and for those who will be using devstack for testing the root secret in Barbican, I suggest that you do not reboot your devstack vm.15:00
jrichliunless you have already worked with devstack enough to know how to start keystone and barbican to be exactly how stack.sh sets it up.15:01
jrichlior unless you are ok with re-provisioning some things afterwards15:01
*** dmorita_ has joined #openstack-swift15:02
*** dmorita has quit IRC15:04
*** dmorita_ has quit IRC15:07
*** _JZ_ has joined #openstack-swift15:09
*** dmorita has joined #openstack-swift15:12
*** dmorita has quit IRC15:16
*** sams-gleb has joined #openstack-swift15:17
*** hseipp has joined #openstack-swift15:26
*** bkopilov has joined #openstack-swift15:41
claygjrichli: there is a screenrc file that defines how services are started15:49
claygjrichli: vagrant-swift-devstack has a thing so you can "ctrl-a \" kill then when you start screen all the services come up15:50
patchbotError: No closing quotation15:50
jrichliclayg: I dabbled with screen, but gave up and found a way to restart barbican api and keystone (just apache2 restart).  but now I get SSL errors.15:50
jrichlii dont know if a cert was lost, or i have installed some new OpenSSL version that will cause me to change config, ... still looking.15:51
jrichlii did make a running snapshot before i rebooted, so i can just go back.  but i am still curious to find out how to recover from a devstack reboot (one with ks + bb + swift)15:53
*** rcernin has quit IRC15:59
*** chlong_ has quit IRC16:01
*** chlong__ has joined #openstack-swift16:01
clayggood thinking on the snapshot!16:02
*** janonymous has quit IRC16:04
*** dmorita has joined #openstack-swift16:06
*** dmorita_ has joined #openstack-swift16:09
claygjrichli: does yours say SSLCertificateFile: file '/opt/stack/data/devstack-cert.pem' does not exist or is empty16:09
*** dmorita_ has quit IRC16:10
jrichlino, I get "SSLError: SSL exception connecting to https://127.0.0.1:35357: bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')],)"16:10
jrichlibut nothing exists in the /opt/stack/data directory.16:10
*** dmorita_ has joined #openstack-swift16:10
*** dmorita has quit IRC16:10
*** klrmn has joined #openstack-swift16:11
jrichliI don't even know if barbican was trying to use keystone via https before when it worked.  it could be that something changed that behavior - and it should just be using http16:11
jrichliI started barbican with /opt/stack/barbican/bin/barbican-api16:12
jrichlibut that is where the error comes from.  barbican calling out to keystone.  i can try to change the config there.  but it is still strange that this worked before.16:12
*** psachin has quit IRC16:12
*** dmorita has joined #openstack-swift16:13
*** dmorita_ has quit IRC16:15
*** dmorita has quit IRC16:17
*** dmorita has joined #openstack-swift16:18
*** dmorita has quit IRC16:22
jrichliBTW: the devstack vm I am speaking of was something I had setup last week - so this is not from using the devstack mathiasb has pointed us to16:24
*** dmorita has joined #openstack-swift16:29
*** dmorita_ has joined #openstack-swift16:31
*** dmorita has quit IRC16:34
*** SkyRocknRoll has joined #openstack-swift16:34
*** dmorita_ has quit IRC16:36
timburkegood morning16:39
*** chsc has joined #openstack-swift16:39
*** chsc has joined #openstack-swift16:39
*** pcaruana has quit IRC16:42
*** SkyRocknRoll has quit IRC16:42
*** dmorita has joined #openstack-swift16:45
*** dmorita_ has joined #openstack-swift16:46
*** dmorita has quit IRC16:46
*** dmorita has joined #openstack-swift16:50
*** dmorita_ has quit IRC16:51
*** dmorita has quit IRC16:55
*** JimCheung has joined #openstack-swift16:55
openstackgerritTim Burke proposed openstack/swift master: Fix port ranges in install guide  https://review.openstack.org/44069416:57
*** ChubYann has joined #openstack-swift16:57
*** cbartz has left #openstack-swift17:04
*** dmorita has joined #openstack-swift17:07
*** dmorita has quit IRC17:08
*** dmorita has joined #openstack-swift17:08
*** dmorita has quit IRC17:09
*** dmorita has joined #openstack-swift17:09
*** dmorita_ has joined #openstack-swift17:10
*** dmorita has quit IRC17:13
*** dmorita_ has quit IRC17:15
*** geaaru has quit IRC17:21
jrichliFYI: I created an etherpad at https://etherpad.openstack.org/p/swift-kms-keymaster-setup17:29
jrichlithat includes a link to mathiasb's script and has my  notes from setting up things during the PTG for this testing17:30
*** bkopilov has quit IRC17:31
notmynamegood morning17:32
notmynamejrichli: cool17:32
notmynamejrichli: is that linked to anywhere? like on the ideas wiki page?17:32
*** JimCheung has quit IRC17:33
acoleskota_: ack, thank17:33
notmynameFYI the API working group has proposed a set of guidelines for API compatibility. among other things, it defines a lot of commonly-used terms (including "compatibility"). this is a proposal we need to pay attention to17:43
notmynamehttps://review.openstack.org/#/c/421846/4/guidelines/api_compatibility.rst,unified17:43
patchbotpatch 421846 - api-wg - Refactor and re-validate api change guidelines17:43
*** dmorita has joined #openstack-swift17:46
*** dmorita has quit IRC17:50
*** tesseract has quit IRC17:52
*** hseipp has quit IRC17:54
*** dmorita has joined #openstack-swift18:01
*** McMurlock1 has quit IRC18:23
jrichlinotmyname: I added a link to it inside a comment on the patch, but i will take a look at the ideas page for what exists for keymaster18:39
jrichliand thanks for the fyi on api compatibility18:41
*** amoralej is now known as amoralej|off18:44
jrichlinotmyname: just added a link to the patch and the etherpad to the existing ideas page for keymaster work18:52
notmynamethanks18:52
claygtimburke: everytime something reminds me of patch 274840 I freak out until I remember patch 31189919:14
patchbothttps://review.openstack.org/#/c/274840/ - swift - change default ports for servers (MERGED)19:14
patchbothttps://review.openstack.org/#/c/311899/ - swift - Fix probe tests from commit cf48e75 (MERGED)19:14
notmynameFYI - S3 post-mortem https://aws.amazon.com/message/41926/19:14
claygrledisez: do you already have the patch that fixes the ssync semaphore thing so you can limit to ~4 connections per disk or whatever?19:21
*** glange_ has joined #openstack-swift19:34
*** ChanServ sets mode: +v glange_19:34
glange_https://developer.openstack.org/api-ref/object-storage/?expanded=get-object-content-and-metadata-detail <- in the Range section of object gets there are several possible formats for that header specified19:35
glange_but the code only seems to accept _content_range_pattern = re.compile(r'^bytes (\d+)-(\d+)/(\d+)$')19:36
glange_am I seeing that right?  or am I missing something19:36
timburkethat pattern looks more like what you'd expect in a response, as a Content-Range header19:37
timburke(Content-Range probably ought to be documented in the api ref...)19:38
torgomaticdocument it by reference to RFC 7233 section 4.219:39
glange_is that documentation out of date? the one I linked?19:43
glange_was it changed just to follow the RFC?19:43
glange_or did it always work that way?19:43
openstackgerritJim Cheung proposed openstack/pyeclib master: Add Phazr.IO libphazr backend support  https://review.openstack.org/44078119:44
rledisezclayg: i uploaded this during Barcelona summit: https://review.openstack.org/#/c/390781/19:45
patchbotpatch 390781 - swift - WIP: Replace replication_one_per_device by custom ...19:45
rledisezclayg: it would need some cleanup and test, but i think this is what you’re looking for19:45
*** tdasilva has quit IRC19:46
torgomaticglange_: The docs for the Range header are correct, but redundant. Swift does what the RFC says to do for Range, bugs notwithstanding.19:48
rledisezclayg: i wanted to do like rsync, lockf’ing range of file instead of having multiple files, but it didn’t work, i don’t remember why, i think it’s because a lockf() is valid by process, not thread or file descriptor19:52
claygrledisez: I had assumed we could leverage the semaphore stuff?19:53
claygrledisez: have a locks[device].aquire() sorta thing?19:53
*** tdasilva has joined #openstack-swift19:56
*** jamielennox|away is now known as jamielennox20:02
rledisezclayg: if i remember well, semaphore is initialized at the very start of the object server (with the « config » object). at that point we don’t have yet the list of devices. i think this is why i didn’t choose this way20:06
rledisezclayg: it’s happening here: https://github.com/openstack/swift/blob/master/swift/obj/server.py#L1180-L120120:09
rledisezclayg: called from wsgi.run_wsgi()20:09
rledisezclayg: well, it should be do-able as this is also the place where server_per_port is used20:10
claygrledisez: you could lazy load the sems anway - just create them under a global lock20:16
claygrledisez: you gunna change it - or you want me to spin something20:16
claygrledisez: i'm obliviuous to timezones - for all I know it's beer thirty in your neck of the woods20:16
claygoh.. maybe you can't lazy create them - you *have* to make them before fork20:18
claygthat... should have been obvious - my bad20:18
rledisezexactly :/ and if they are pre-spawned, what about ring update with added/removed devices20:19
claygfuuuuu20:19
rledisez(beer time was few hours ago, for a week day it’s more like tea ;))20:19
claygrledisez: side quesiton - are you using servers per port for your replication servers?20:22
rledisezclayg: yes -> https://review.openstack.org/#/c/337861/20:24
patchbotpatch 337861 - swift - Permit to bind object-server on replication_port20:24
claygrledisez: right20:25
openstackgerritClay Gerrard proposed openstack/swift master: Permit to bind object-server on replication_port  https://review.openstack.org/33786120:30
*** oshritf has joined #openstack-swift20:37
openstackgerritThiago da Silva proposed openstack/swift master: refactor some common code from crypto  https://review.openstack.org/43905820:43
*** oshritf has quit IRC20:43
*** adriant has joined #openstack-swift20:59
mattoliverauMorning21:07
*** garyj has joined #openstack-swift22:04
*** garyj has quit IRC22:10
*** dja has joined #openstack-swift22:12
*** sams-gleb has quit IRC22:40
*** catintheroof has quit IRC22:40
claygtdasilva: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh23:00
claygso there's a "debian-openstack" mirror that nodepool adds when it's preparing nodes23:00
claygso maybe getting a centos/epel/fedora-openstack mirror for rpm-packaging is a thing that isn't years away?23:01
*** dja has quit IRC23:04
*** tdasilva has quit IRC23:05
*** dmorita_ has joined #openstack-swift23:23
*** dmorita has quit IRC23:23
*** tdasilva has joined #openstack-swift23:23
*** kei_yama has joined #openstack-swift23:34
*** chsc has quit IRC23:36
*** jamielennox is now known as jamielennox|away23:37
*** klamath has quit IRC23:37
*** dja has joined #openstack-swift23:37
*** dja has quit IRC23:41
*** _JZ_ has quit IRC23:44
kota_good morning!23:48
notmynamehello kota_23:49
kota_notmyname: o/23:49
notmynamekota_: I just clicked +2+A ont he tempauth patch to master23:49
kota_notmyname: thanks a lot!23:49
notmynameI'd like to see that land before doing the ocata one. does that make sense?23:50
notmynamemaybe it's fine23:50
notmynamekota_: anyway, we spend a *lot* of time talking about this today23:50
kota_with timburke?23:51
notmynameyes. and clayg23:51
kota_can i get the summary?23:51
timburkethe interactions are terrible, but we can't come up with a clean intra-middleware api in a day23:51
notmynamethe basic summary of my current thoughts is (1) I hate that there's so much knowledge of s3 inside of auth systems (2) we're changing the interface anyway (3) can we change it to something we'll be happy to support longer term without worrying about breaking changes23:51
notmynamehowever, we have no idea what that interface would look like23:51
notmynametimburke mocked up a small change, but even it's not great (ie would need more), so we're looking at incremental improvements overall23:52
kota_i see23:52
notmynameand I don't want to rush something just because it might be better. seems like a great way to make the wrong call23:52
kota_+123:53
notmynameso... it's code, let's make it better where we can today and keep making it better as we go, based on what our users need23:53
notmynameI don't want to hold up something that's known-good and fixes a bad bug in the hope that we'll designe the perfect api in an afternoon23:53
notmynameso that's why I landed it23:53
kota_thanks, on the (1) of the summary, I'd support that but I also have no idea because I know, we could want to one more complex auth for somewhere (i mean signature v4 of AWS)23:55
notmynameyep :-)23:55
kota_we did it in the keystone side but tempauth still has not supported...23:55
notmynameit's like you were sitting in the room with us today :-)23:55
kota_difficult problem for good maintainancibility :/23:56
kota_but anyway, thanks again to land the patch. :D23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!