| openstackgerrit | Merged openstack/python-swiftclient master: Remove unnecessary calls to parse_header_string(). https://review.openstack.org/585515 | 00:25 |
|---|---|---|
| notmyname | swiftclient release tag request made: https://review.openstack.org/#/c/585575/ | 01:05 |
| patchbot | patch 585575 - releases - python-swiftclient 3.6.0 | 01:05 |
| openstackgerrit | Nguyen Hai proposed openstack/swift master: add lower-constraints job https://review.openstack.org/556255 | 01:16 |
| *** links has joined #openstack-swift | 01:30 | |
| timburke | yay! https://review.openstack.org/#/c/585568/ shows all clear! | 01:39 |
| patchbot | patch 585568 - swift - Lower the required version of ipaddress | 01:39 |
| timburke | probetests are still good | 01:40 |
| *** gyee has quit IRC | 01:55 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add support for multiple root encryption secrets https://review.openstack.org/577874 | 02:04 |
| kota_ | morning | 02:13 |
| kota_ | tdasilva: could you have a time to look at https://review.openstack.org/#/c/447129/ ? | 02:13 |
| patchbot | patch 447129 - swift - Configure diskfile per storage policy | 02:13 |
| *** m_kazuhiro has joined #openstack-swift | 02:13 | |
| kota_ | i briefly looked at the patch yesterday and i'm realizing i'd like to hear your opinion, | 02:14 |
| kota_ | that patch attempt to involve the diskfile extension for losf but IIRC you already maintains custom diskfile on swift-on-file. | 02:15 |
| kota_ | if that patch helps your swift-on-file project, that would be nice to have but I'm not sure the technic (e.g. configuration format) would be nicer for swift-on-file way. | 02:16 |
| timburke | yay! thanks for looking at p 578075, mattoliverau! now i need to propose a follow-on to p 577874 to make the kmip keymaster multi-key too... | 02:30 |
| patchbot | https://review.openstack.org/#/c/578075/ - swift - Add keymaster to fetch root secret from KMIP service | 02:30 |
| patchbot | https://review.openstack.org/#/c/577874/ - swift - Add support for multiple root encryption secrets | 02:30 |
| *** StevenK has quit IRC | 02:35 | |
| *** StevenK has joined #openstack-swift | 02:35 | |
| *** seongsoocho has joined #openstack-swift | 02:52 | |
| *** kei_yama has quit IRC | 03:07 | |
| *** armaan has joined #openstack-swift | 03:13 | |
| openstackgerrit | Merged openstack/swift master: Add keymaster to fetch root secret from KMIP service https://review.openstack.org/578075 | 03:49 |
| *** psachin` has joined #openstack-swift | 04:00 | |
| *** psachin`` has joined #openstack-swift | 04:06 | |
| *** psachin` has quit IRC | 04:07 | |
| *** armaan has quit IRC | 04:19 | |
| *** armaan has joined #openstack-swift | 04:20 | |
| *** armaan has quit IRC | 04:56 | |
| *** armaan has joined #openstack-swift | 04:56 | |
| *** armaan has quit IRC | 05:17 | |
| *** armaan has joined #openstack-swift | 05:18 | |
| *** armaan has quit IRC | 05:32 | |
| *** armaan has joined #openstack-swift | 05:32 | |
| *** hoonetorg has quit IRC | 06:01 | |
| *** armaan has quit IRC | 06:04 | |
| *** armaan has joined #openstack-swift | 06:05 | |
| *** ccamacho has joined #openstack-swift | 06:09 | |
| *** cshastri has joined #openstack-swift | 06:14 | |
| *** hoonetorg has joined #openstack-swift | 06:17 | |
| *** armaan has quit IRC | 06:22 | |
| *** armaan has joined #openstack-swift | 06:22 | |
| *** psachin`` has quit IRC | 06:24 | |
| *** pcaruana has joined #openstack-swift | 06:33 | |
| *** rcernin has quit IRC | 06:59 | |
| *** armaan has quit IRC | 07:09 | |
| *** armaan has joined #openstack-swift | 07:10 | |
| *** armaan has quit IRC | 07:14 | |
| *** gkadam has joined #openstack-swift | 07:16 | |
| *** tesseract has joined #openstack-swift | 07:16 | |
| *** DHE has quit IRC | 07:24 | |
| *** DHE has joined #openstack-swift | 07:25 | |
| *** geaaru has joined #openstack-swift | 07:26 | |
| *** geaaru_ has joined #openstack-swift | 07:27 | |
| *** geaaru_ has quit IRC | 07:40 | |
| seongsoocho | Hi. Is there a way to change the storage policy to existing container? | 07:44 |
| mattoliverau | seongsoocho: not that the moment. There is work on policy migration but it hasn't landed yet. There are some cowboy ways you can try and do it, involving changing the policy manaully in the container (editing the sqlite dbs) and waiting for the reconciler to work trhough them. | 07:47 |
| seongsoocho | mattoliverau: oh I got it. To changing the policy manually is not dangerous? I think it might be rollback by replicator. | 07:50 |
| mattoliverau | oh I'm sure it could be considered dangerous, which is why policy migration is worked on. | 08:01 |
| *** mikecmpbll has joined #openstack-swift | 08:01 | |
| kota_ | it's hard problem to change the policy in the working real cluster, I'm sure. | 08:03 |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Configure diskfile per storage policy https://review.openstack.org/447129 | 08:03 |
| kota_ | it's not just a db change but also it requires data translation if you want to change it from replication to something like EC. | 08:04 |
| *** armaan has joined #openstack-swift | 08:05 | |
| seongsoocho | kota_: Yes that's right. When i change db, the existed data is a problem. Because I will make new storage policy with new ring. | 08:06 |
| mattoliverau | seongsoocho: yeah, it's better to start with a new policy and a new container. Then if required migrate things, or just start using the new container and refer back to the old one when needed. You could also use symlinks for the older objects with you didn't need to or want to move them. | 08:14 |
| mattoliverau | ie, space isn't the issue | 08:14 |
| *** stephen_m has joined #openstack-swift | 08:20 | |
| *** rcernin has joined #openstack-swift | 08:29 | |
| *** stephen_m has quit IRC | 08:37 | |
| *** stephen_m has joined #openstack-swift | 08:38 | |
| *** armaan has quit IRC | 08:46 | |
| *** armaan has joined #openstack-swift | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** armaan has quit IRC | 08:51 | |
| *** armaan has joined #openstack-swift | 09:11 | |
| *** notmyname has quit IRC | 09:16 | |
| *** notmyname has joined #openstack-swift | 09:17 | |
| *** ChanServ sets mode: +v notmyname | 09:17 | |
| *** armaan has quit IRC | 09:26 | |
| *** armaan has joined #openstack-swift | 09:26 | |
| *** armaan has quit IRC | 09:30 | |
| *** d0ugal has joined #openstack-swift | 09:36 | |
| *** d0ugal has quit IRC | 09:36 | |
| *** d0ugal has joined #openstack-swift | 09:36 | |
| *** m_kazuhiro has quit IRC | 09:37 | |
| *** lifeless has quit IRC | 09:37 | |
| *** openstackgerrit has joined #openstack-swift | 09:48 | |
| openstackgerrit | Alexander Bolshakov proposed openstack/python-swiftclient master: Fix update when PUT on containers not allowed https://review.openstack.org/585325 | 09:48 |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Configure diskfile per storage policy https://review.openstack.org/447129 | 10:08 |
| *** d0ugal has quit IRC | 10:21 | |
| *** kukacz_ has quit IRC | 10:23 | |
| *** kukacz_ has joined #openstack-swift | 10:23 | |
| *** lifeless has joined #openstack-swift | 10:23 | |
| *** armaan has joined #openstack-swift | 10:45 | |
| *** armaan has quit IRC | 10:55 | |
| *** armaan has joined #openstack-swift | 10:56 | |
| *** armaan has quit IRC | 10:56 | |
| *** armaan has joined #openstack-swift | 11:07 | |
| *** rcernin has quit IRC | 11:21 | |
| *** armaan has quit IRC | 11:27 | |
| *** armaan has joined #openstack-swift | 11:27 | |
| *** armaan has quit IRC | 11:29 | |
| *** armaan has joined #openstack-swift | 11:30 | |
| *** ccamacho has quit IRC | 11:38 | |
| *** ccamacho has joined #openstack-swift | 11:44 | |
| *** armaan has quit IRC | 12:03 | |
| *** armaan has joined #openstack-swift | 12:04 | |
| *** armaan has quit IRC | 12:06 | |
| *** armaan has joined #openstack-swift | 12:07 | |
| *** armaan has quit IRC | 12:11 | |
| *** armaan has joined #openstack-swift | 13:18 | |
| *** armaan has quit IRC | 13:25 | |
| *** d0ugal has joined #openstack-swift | 13:28 | |
| *** d0ugal has quit IRC | 13:45 | |
| *** armaan has joined #openstack-swift | 14:17 | |
| *** armaan_ has joined #openstack-swift | 14:19 | |
| *** armaan has quit IRC | 14:21 | |
| *** stephen_m has quit IRC | 14:29 | |
| *** armaan_ has quit IRC | 14:31 | |
| *** armaan has joined #openstack-swift | 14:32 | |
| *** armaan has quit IRC | 14:32 | |
| *** links has quit IRC | 14:33 | |
| *** stephen_m has joined #openstack-swift | 14:35 | |
| *** armaan has joined #openstack-swift | 14:37 | |
| *** armaan has quit IRC | 14:42 | |
| *** pcaruana has quit IRC | 14:52 | |
| *** armaan has joined #openstack-swift | 15:02 | |
| *** armaan has quit IRC | 15:14 | |
| *** armaan has joined #openstack-swift | 15:14 | |
| *** armaan has quit IRC | 15:14 | |
| *** cshastri has quit IRC | 15:16 | |
| *** gyee has joined #openstack-swift | 15:29 | |
| *** armaan has joined #openstack-swift | 15:29 | |
| *** kei-ichi has quit IRC | 15:45 | |
| *** kei-ichi has joined #openstack-swift | 15:46 | |
| *** itlinux has joined #openstack-swift | 15:49 | |
| timburke | good morning | 15:50 |
| notmyname | good morning | 15:51 |
| *** gkadam has quit IRC | 15:52 | |
| *** armaan has quit IRC | 15:58 | |
| *** armaan has joined #openstack-swift | 15:58 | |
| *** armaan has quit IRC | 15:59 | |
| *** armaan has joined #openstack-swift | 16:00 | |
| DHE | is 135 hard drives on a single object server too much? (somewhat serious question... it's a machine with 3x 45-drive enclosures attached) | 16:00 |
| timburke | i think the largest i've heard about previously was 90... 135 seems like a lot... but what do i know? | 16:03 |
| notmyname | DHE: it's not what I'd choose ;-) | 16:03 |
| DHE | 90 in a chassis with both server+disks is the most I've seen in that form factor. This is more retrofitting an existing machine into an object storage server. It's a 1U server + 3 external enclosures | 16:04 |
| notmyname | up to a point, it's a matter of drive:core ratios | 16:05 |
| DHE | dual socket 8-core HT intels | 16:05 |
| notmyname | normally the highest ratio I've seen is 4:1 | 16:05 |
| DHE | well that's ~8:1, or 4.2:1 if you consider hyperthreads to be cores | 16:06 |
| timburke | i'd really worry about not having enough resources per drive. 135 spindles gets you a lot of disk IO, so you'll almost certainly have a bottleneck at the NIC. xfs inode cache for all those drives is going to put a lot of strain on your memory, too. and yeah, gotta worry about having enough cycles for all those processes | 16:08 |
| *** ccamacho has quit IRC | 16:12 | |
| DHE | and it would probably do double duty as a proxy... ugh... | 16:13 |
| DHE | maybe I'll get some 1U server for each enclosure then.. | 16:14 |
| timburke | how many boxes in the cluster? was it just going to be the one otherwise? | 16:15 |
| *** armaan has quit IRC | 16:16 | |
| DHE | I'm still in the planning and business case phase. I have my eyes on this machine, two more with 90 drives, and two more with ~30 drives. Most will be located in various different cities as a sort of CDN | 16:16 |
| *** armaan has joined #openstack-swift | 16:16 | |
| DHE | these machines already exist and will be (slowly) repurposed | 16:16 |
| *** mikecmpbll has quit IRC | 16:33 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-swiftclient master: Update reno for stable/rocky https://review.openstack.org/585823 | 16:51 |
| *** tesseract has quit IRC | 16:53 | |
| zaitcev | Guys | 17:02 |
| zaitcev | Our object auditor does not check for dark data, does it? | 17:02 |
| zaitcev | I mean, it does not do any get_info() things versus container and account. | 17:02 |
| timburke | nope, don't believe so | 17:02 |
| zaitcev | It does not even know where your memcacheds are... | 17:03 |
| zaitcev | Just making sure, because often I read the code and miss the obvious. | 17:03 |
| zaitcev | So, here's a problem | 17:03 |
| timburke | might be an interesting application for https://review.openstack.org/#/c/212824/ ... | 17:04 |
| patchbot | patch 212824 - swift - Let developers/operators add watchers to object audit | 17:04 |
| zaitcev | I know someone who has a cluster, 4 nodes with 12 6TB drives. So, about 90TB, replication is 3. | 17:04 |
| zaitcev | They put it in a corner and left it rot until it started falling apart. | 17:05 |
| zaitcev | I found that a while back a few drives overflowed. When that happens, XFS can sometimes detect internal inconsistency and then it panics. When it does so, it marks the mount as suspect and all I/O to that volume returns EIO. | 17:06 |
| zaitcev | The idea is to let the administrator unmount and remount it, so the kernel does fsck. | 17:06 |
| zaitcev | It's a kernel 3.x, which is pretty ancient stuff that we shipped in RHEL 7. | 17:07 |
| zaitcev | The bad part is, their drives stayed locked like this for a very long time, and remember that they were overflowed. Basically, completely frozen. When they tried to delete some objects, nothing happened to them. | 17:08 |
| zaitcev | You see where this is going | 17:08 |
| zaitcev | Once they rebooted - months since the overflow and Oops, possibly, the old data all came back | 17:09 |
| zaitcev | Well, at least I'm concerned that it did | 17:09 |
| zaitcev | And, I'm afraid it came back as dark data, because all the tombstones and container entries were cleared out long time ago at the working part of the cluster. | 17:10 |
| zaitcev | I'm thinking about hacking the auditor to find how much dark data they have now. | 17:11 |
| timburke | yeah, that seems likely. if even one replica accepted the delete, the container would've gotten updated and that update would get replicated separately... | 17:11 |
| timburke | i feel like clayg may have hacked something together to look for dark data before... | 17:11 |
| zaitcev | Yeah | 17:12 |
| zaitcev | In my friend's case it's a well founded concern, I think. | 17:12 |
| zaitcev | I had some on my testbed in the lab, but that might be caused by some of my PUT+POST and other test patches not working right. | 17:13 |
| clayg | ugh, swift-integrity, it's about as elegant as one might imagine - would take some work to get it out on a license | 17:13 |
| *** bharath1234 has joined #openstack-swift | 17:14 | |
| *** itlinux has quit IRC | 17:47 | |
| *** itlinux has joined #openstack-swift | 17:51 | |
| *** juggsmcb has joined #openstack-swift | 17:58 | |
| *** juggsmcb has quit IRC | 18:01 | |
| *** geaaru has quit IRC | 18:06 | |
| *** rcernin has joined #openstack-swift | 18:09 | |
| *** kazsh has quit IRC | 18:52 | |
| *** kazsh has joined #openstack-swift | 18:52 | |
| *** rcernin has quit IRC | 18:53 | |
| *** kota_ has quit IRC | 18:54 | |
| *** ktsuyuzaki has joined #openstack-swift | 18:54 | |
| *** ChanServ sets mode: +v ktsuyuzaki | 18:54 | |
| *** rcernin has joined #openstack-swift | 19:10 | |
| *** d0ugal has joined #openstack-swift | 19:11 | |
| *** rcernin has quit IRC | 19:16 | |
| *** bharath1234 has quit IRC | 19:19 | |
| *** d0ugal has quit IRC | 19:46 | |
| tdasilva | is there are meeting today? | 19:56 |
| notmyname | tdasilva: ya, but I think it should be brief | 19:57 |
| tdasilva | ack | 19:59 |
| openstackgerrit | John Dickinson proposed openstack/swift master: added docker test target tools https://review.openstack.org/577467 | 20:20 |
| timburke | oh yeah, we should totally go land https://review.openstack.org/#/c/585568/ -- probe tests pass again! | 20:23 |
| patchbot | patch 585568 - swift - Lower the required version of ipaddress | 20:23 |
| notmyname | nice! | 20:23 |
| notmyname | also, we should make probe tests voting | 20:23 |
| timburke | yeah, we definitely seem to be getting pretty good signal from them | 20:24 |
| ktsuyuzaki | morning | 20:52 |
| notmyname | hello ktsuyuzaki | 20:53 |
| ktsuyuzaki | oh, my nick was changed... | 20:53 |
| notmyname | I see that | 20:53 |
| ktsuyuzaki | hello notmyname | 20:53 |
| notmyname | you lose your password or soemthing? | 20:53 |
| *** ktsuyuzaki is now known as kota_ | 20:54 | |
| kota_ | no? | 20:54 |
| notmyname | lol | 20:54 |
| notmyname | ok :-) | 20:54 |
| notmyname | oh, I was just wondering why you chnged your nick. I'm fine either way :-) | 20:54 |
| kota_ | I may have to check my irc bouncer (or irc client) configuration | 20:55 |
| notmyname | kota_: you were on vacation, right? I hope it was relazing | 20:56 |
| notmyname | *relaxing | 20:56 |
| kota_ | notmyname: yes, in the last whole week. it was completely fine to me. | 20:57 |
| *** m_kazuhiro has joined #openstack-swift | 20:58 | |
| notmyname | swift team meeting time in #openstack-meeting | 20:59 |
| *** openstackgerrit has quit IRC | 21:04 | |
| *** mrjk has quit IRC | 21:05 | |
| *** mrjk has joined #openstack-swift | 21:06 | |
| *** m_kazuhiro has quit IRC | 21:27 | |
| *** itlinux has quit IRC | 21:31 | |
| *** itlinux has joined #openstack-swift | 21:36 | |
| tdasilva | gotta love it...so...to install python-pip in centos7, there's a dependency on python-ipaddress which as timburke mentioned centos ships 1.0.16, but then to install python-openstackclient, that has a dependency on 1.0.17 and everthing fails | 21:44 |
| timburke | whomp whomp | 21:45 |
| timburke | the crazy thing (to me) is that *we installed 1.0.22*: http://logs.openstack.org/74/577874/4/check/swift-probetests-centos-7/8f8a67c/job-output.txt.gz#_2018-07-25_02_10_02_450548 | 21:47 |
| tdasilva | yep! really crazy! | 21:47 |
| timburke | but then we somehow still got stuck with the system package | 21:47 |
| timburke | it was stuff like that that made me avoid adding a dep on ipaddress in https://review.openstack.org/#/c/581906/ ... but swift already had ipaddress as a transitive dep via cryptography | 21:51 |
| patchbot | patch 581906 - python-swiftclient - Add more validation for ip_range args | 21:51 |
| tdasilva | timburke: i think i figured it out for my ansible script. with centos I guess we are not supposed to use the epel repo anymore and should just use the cloud sig repo now | 21:52 |
| tdasilva | timburke: current one being: centos-release-openstack-queens | 21:53 |
| tdasilva | i don't think we need this anymore: https://github.com/openstack/swift/blob/master/tools/playbooks/saio_single_node_setup/install_dependencies.yaml#L18,L42 | 21:55 |
| tdasilva | need to test... | 21:55 |
| timburke | oh good! i didn't exactly like doing https://github.com/openstack/swift/commit/37dafa0 | 21:58 |
| *** openstackgerrit has joined #openstack-swift | 22:08 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: Remove contentdir hack https://review.openstack.org/585894 | 22:08 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: make probe tests voting in the gate https://review.openstack.org/585900 | 22:19 |
| *** eandersson has quit IRC | 22:22 | |
| *** eandersson has joined #openstack-swift | 22:24 | |
| *** itlinux has quit IRC | 22:25 | |
| *** mtreinish has quit IRC | 22:33 | |
| *** mtreinish has joined #openstack-swift | 22:36 | |
| *** ukaynar_ has joined #openstack-swift | 23:00 | |
| *** stephen_m has quit IRC | 23:06 | |
| *** mikecmpbll has joined #openstack-swift | 23:07 | |
| *** kei_yama has joined #openstack-swift | 23:08 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Remove some unnecessary SkipTests https://review.openstack.org/585910 | 23:15 |
| *** itlinux has joined #openstack-swift | 23:44 | |
| mattoliverau | morning... again | 23:47 |
| *** ukaynar_ has quit IRC | 23:54 | |
| *** ukaynar has joined #openstack-swift | 23:55 | |
| notmyname | once upon a time, we partially implemented containers as having an internal uuid with an external human-readable name. this allowed for renaming containers without copying all the data. we later abandoned the idea after we couldn't figure out some consistency issues | 23:56 |
| *** ukaynar has quit IRC | 23:56 | |
| *** ukaynar has joined #openstack-swift | 23:56 | |
| notmyname | however, given than we've done crazy stuff since then, like fast POST, it makes me wonder if such a thing would be possible, given enough time to think about it | 23:56 |
| notmyname | certainly it's waaay down on a list of important stuff to do next, but still... I wonder if it's possible | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!