Tuesday, 2017-08-22

kota_good morning00:07
kota_nice to see encryption with barbican landed!00:08
mattoliveraukota_: morning00:08
kota_mattoliverau: morning00:09
kota_notmyname: ack for https://review.openstack.org/#/c/495805/, it looks like not a big patch (I'm just opening the URL though) so I think I can do it today.00:09
patchbotpatch 495805 - swift - Drop reconstructor stats when worker has no devices00:09
*** baojg has quit IRC00:16
*** baojg has joined #openstack-swift00:17
*** baojg has quit IRC00:21
*** catintheroof has quit IRC00:25
*** kiennt has joined #openstack-swift00:36
*** shuyingya has joined #openstack-swift00:43
*** shuyingya has quit IRC00:45
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements  https://review.openstack.org/8873600:48
*** mat128 has joined #openstack-swift00:51
*** cshastri has joined #openstack-swift00:59
*** klrmn has quit IRC01:12
*** shuyingya has joined #openstack-swift01:16
*** bkopilov has quit IRC02:13
*** shuyingya has quit IRC02:14
*** shuyingya has joined #openstack-swift02:14
*** baojg has joined #openstack-swift02:23
*** mvk_ has quit IRC02:25
*** baojg has quit IRC02:28
*** catintheroof has joined #openstack-swift02:30
*** shuyingya has quit IRC02:31
*** mvk_ has joined #openstack-swift02:36
*** catintheroof has quit IRC02:57
*** itlinux has joined #openstack-swift03:04
*** itlinux has quit IRC03:21
*** Tgrv has joined #openstack-swift03:21
*** bkopilov has joined #openstack-swift03:31
*** gkadam has joined #openstack-swift03:33
*** itlinux has joined #openstack-swift03:35
*** itlinux has quit IRC03:36
*** links has joined #openstack-swift03:51
*** mat128 has quit IRC03:53
*** itlinux has joined #openstack-swift04:16
*** itlinux has quit IRC04:16
*** itlinux has joined #openstack-swift04:49
*** itlinux has quit IRC05:17
*** tovin07_ has joined #openstack-swift05:18
*** shuyingya has joined #openstack-swift05:19
*** psachin has joined #openstack-swift05:24
*** mkrcmari__ has joined #openstack-swift05:28
*** links has quit IRC05:30
*** mvk_ has quit IRC05:31
*** links has joined #openstack-swift05:33
*** hoonetorg has quit IRC05:34
*** hoonetorg has joined #openstack-swift05:49
*** kiennt has quit IRC05:58
*** kiennt has joined #openstack-swift06:09
*** hoonetorg has quit IRC06:14
*** Tgrv has quit IRC06:15
*** alenavolk has joined #openstack-swift06:20
*** alenavolk has quit IRC06:20
*** hoonetorg has joined #openstack-swift06:21
*** kiennt has quit IRC06:22
mathiasbtdasilva: thanks for your comprehensive testing and +2!06:29
*** ChubYann has quit IRC06:35
*** rcernin has joined #openstack-swift06:45
*** pcaruana has joined #openstack-swift06:45
*** hseipp has joined #openstack-swift06:47
*** jistr is now known as jistr|trng06:53
*** kiennt has joined #openstack-swift06:59
*** xrb has joined #openstack-swift07:04
*** xrb has quit IRC07:04
*** xrb has joined #openstack-swift07:05
*** cshastri has quit IRC07:19
*** tesseract has joined #openstack-swift07:20
*** cshastri has joined #openstack-swift07:21
*** shuyingya has quit IRC07:25
*** shuyingya has joined #openstack-swift07:27
*** psachin has quit IRC07:34
-openstackstatus- NOTICE: Gerrit is going to be restarted due to slow performance07:37
*** oshritf has joined #openstack-swift07:37
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata  https://review.openstack.org/49613707:38
*** geaaru has joined #openstack-swift07:41
-openstackstatus- NOTICE: Gerrit has been restarted successfully07:42
*** skudlik has joined #openstack-swift07:57
*** skudlik has left #openstack-swift07:58
acolesmathiasb: congratulations! glad your patch has finally merged08:03
acolesgood morning08:04
mathiasbgood morning08:04
mathiasbacoles: thanks for your feedback and reviews too!08:04
acolesmathiasb: you're welcome - I'm only sorry I wasn't able to join in the final push08:05
*** kei_yama has quit IRC08:13
*** kei_yama has joined #openstack-swift08:14
openstackgerritTim Burke proposed openstack/swift master: Socket errors don't warrant tracebacks when talking to memcached  https://review.openstack.org/49615008:26
*** mvk_ has joined #openstack-swift08:39
*** cbartz has joined #openstack-swift08:41
*** mkrcmari__ has quit IRC08:43
kota_acoles: morning08:45
kota_and I'm on https://review.openstack.org/#/c/495805/08:45
patchbotpatch 495805 - swift - Drop reconstructor stats when worker has no devices08:45
kota_fyi08:45
*** mkrcmari__ has joined #openstack-swift08:54
*** mvk_ has quit IRC08:57
*** mahatic has joined #openstack-swift08:59
*** mvk has joined #openstack-swift09:03
*** mkrcmari__ has quit IRC09:06
*** mahatic has quit IRC09:07
*** mahatic has joined #openstack-swift09:08
acoleskota_: thanks09:22
kota_acoles: just pushing my comments :-)09:27
*** baojg has joined #openstack-swift09:30
*** baojg has quit IRC09:34
acoleskota_: looking...09:34
kota_perhaps, my comments may be overlook rather than just the no device situation...09:35
*** cshastri has quit IRC09:46
acoleskota_: I'm replying on gerrit so others can read too09:50
kota_ok, thanks acoles09:50
*** kei_yama has quit IRC09:50
kota_I'll circle back to the patch tomorrow morning again09:53
acoleskota_: posted reply09:54
kota_so soon!09:54
acoleskota_: have a good evening09:54
acoleskota_: I'm not sure if I perfectly understood your concern so let me know if I did not answer your question09:55
acoleszaitcev: ditto ^^09:55
*** tovin07_ has quit IRC10:00
mahaticgood evening10:01
mahaticI'm back from a vacation and fixed up my bouncer issues10:01
kota_acoles: quick reading for your worth comment, it would make sense to me that "parent process aggregates the per_disk stats and workers updates per_disk info only (basically)"10:05
acolesmahatic: welcome back!10:05
*** kiennt has quit IRC10:05
mahaticacoles: hello! thanks10:06
acoleskota_: does it make sense to you to have the special case (no devices) when the worker updates the root?10:06
acoleskota_: or are you arguing that should also be done by the parent aggregate_recon_update?10:06
kota_but, I'd like to have time for a while (tonight), mainly, I'm wondering how can i finalize the reconstruction_time/last in the root node in recon_update with final_recon_dump when run_once case...10:07
kota_it's definately "with devices" case though10:08
acoleskota_: sure. the case of updating the root when using run_once in worker is an issue, but separate IMHO :)10:08
kota_with devices, self.all_local_devices will be a list contains some devices10:09
kota_i see10:09
kota_tentatively, i pulled out my -1 from the patch but anyway, I'd like to dig the behavior (probably it's from my lack of knowledge on multiple reconstructor process) more to give my +210:12
kota_if zaitcev gets it and feels to add his +2, I'm ok with it.10:13
kota_while I'm asleep10:13
*** silor has joined #openstack-swift10:39
*** bkopilov has quit IRC10:56
*** dancn has quit IRC11:00
*** htruta has quit IRC11:00
*** neonpastor has quit IRC11:00
*** vint_bra has quit IRC11:00
*** StevenK has quit IRC11:00
*** htruta has joined #openstack-swift11:01
*** geaaru has quit IRC11:01
*** Dinesh_Bhor has quit IRC11:01
*** dja_ has quit IRC11:01
*** nikivi has quit IRC11:01
*** etienneme has quit IRC11:01
*** StevenK has joined #openstack-swift11:02
*** vint_bra has joined #openstack-swift11:04
*** Dinesh_Bhor has joined #openstack-swift11:05
*** nikivi has joined #openstack-swift11:05
*** neonpastor has joined #openstack-swift11:05
*** etienneme has joined #openstack-swift11:05
*** geaaru has joined #openstack-swift11:06
*** dja has joined #openstack-swift11:06
*** dancn has joined #openstack-swift11:53
*** mat128 has joined #openstack-swift12:07
*** SkyRocknRoll has quit IRC12:29
*** oshritf has quit IRC12:31
*** oshritf has joined #openstack-swift12:32
*** SkyRocknRoll has joined #openstack-swift12:32
*** psachin has joined #openstack-swift12:35
*** gkadam has quit IRC12:36
*** xrb has quit IRC12:37
*** shuyingya has quit IRC12:37
*** xrb has joined #openstack-swift12:37
*** xrb has quit IRC12:38
*** shuyingya has joined #openstack-swift12:38
*** sanchitmalhotra has quit IRC12:38
*** psachin has quit IRC12:39
*** psachin has joined #openstack-swift12:39
*** shuyingya has quit IRC12:42
*** bkopilov has joined #openstack-swift12:49
*** baojg has joined #openstack-swift12:56
*** zhurong has joined #openstack-swift12:56
*** oshritf has quit IRC13:32
zaitcevkota_: thanks for the explanations13:36
zaitcevre. 49500513:37
*** shuyingya has joined #openstack-swift13:38
*** oshritf has joined #openstack-swift13:41
*** lucasxu has joined #openstack-swift13:54
*** SkyRocknRoll has quit IRC13:58
*** zhurong has quit IRC14:19
*** links has quit IRC14:20
*** gyee has joined #openstack-swift15:11
*** SkyRocknRoll has joined #openstack-swift15:15
*** rcernin has quit IRC15:15
*** tovin07 has joined #openstack-swift15:22
*** oshritf has quit IRC15:27
*** jarbod has joined #openstack-swift15:28
*** cbartz has quit IRC15:28
notmynamegood morning15:37
*** tesseract has quit IRC16:04
openstackgerritMerged openstack/swift master: Make test assertion more holistic  https://review.openstack.org/49538916:10
*** catintheroof has joined #openstack-swift16:10
openstackgerritMerged openstack/swift master: Fix more X-Delete-At timing issues  https://review.openstack.org/49475616:15
*** silor has quit IRC16:15
*** tovin07 has quit IRC16:16
*** silor has joined #openstack-swift16:26
*** hseipp has quit IRC16:26
*** pcaruana has quit IRC16:37
*** chsc has joined #openstack-swift16:38
*** chetna has joined #openstack-swift16:44
*** srb has joined #openstack-swift16:44
*** mvk_ has joined #openstack-swift16:45
*** srb has left #openstack-swift16:45
*** srbgupta has joined #openstack-swift16:46
*** shuyingya has quit IRC16:47
*** mvk has quit IRC16:48
srbguptaHi, I am trying to validate MD5 of a Static large object in Swift with the original object on disk. ?multipart-manifest=get" is returning different Etag value when compared to MD5 of the object. Any idea how I can compare these?16:49
timburkesrbgupta: it'll be a little complicated... since swift never sees all of the bytes for the object all at once, it only knows the MD5s of the pieces. you can do the ?multipart-manifest=get, then walk through the segments yourself -- read "bytes" from the source file, take the MD5, and compare that against the "hash"16:58
timburkepython-swiftclient does something like that around https://github.com/openstack/python-swiftclient/blob/3.4.0/swiftclient/service.py#L1175-L118717:00
srbguptaIf I get all MD5s of the pieces/segments, does it relate to the original object in any way?17:08
srbguptabecause if I do a HEAD with ?multipart-manifest=get , as per documentation, it is sum of all the MD5s of the segments17:09
srbguptabut this is not equating with MD5 of original file on disk.17:10
notmynamesrbgupta: I'm not sure which file you're referring to, exactly17:12
notmynamethe etag returned when reading the large object (*not* the manifest) will be equal to the md5 of the concatenation of the etags of each segment. the etag for each segment is the md5 of the contents of that segment17:12
notmynameso what timburke said about getting the manifest to find the etags for the component segments is the way to validate the large object download. and yes it's complicated17:13
srbgupta@notmyname by file, I meant the original large file which I have on the disk to compare Etag with.17:18
srbguptathanks guys for help :)17:20
*** SkyRocknRoll has quit IRC17:21
notmynamesrbgupta: ah, right. the md5 of that original large file that you have locally won't match the etag of the downloaded large object from swift17:23
notmynamesrbgupta: you could split the local file in the same way it is split in the swift cluster and do the same md5 dance and that should match17:23
srbguptaWe used swift client to upload with --use-slo and segment-size17:24
notmynameyeah, so if you split the local file by the same segment size, it should match17:24
*** geaaru has quit IRC17:24
srbguptayeah that would match, but the data is huge like 45TB, and it will be a big overhead to do this for every file.17:26
notmynameyep17:26
notmynameso .. here's what I'd recommend17:26
notmynameif you upload the large object successfully, then swift has validated that the data is correctly saved17:26
notmynamethen HEAD the large object and get the etag. store that someplace and used that saved value to compare against future downloads17:27
notmynamethere's 2 things I don't like about that solution, though17:27
srbguptaI passed the Etag in header, it works well for small objects but for large objects it returns 422 as it must be comparing against segments.17:27
notmynamefirst, you can't really send the etag with the upload to validate the local data is the same as what's saved (swift will validate that is saved what it got from the network)17:28
notmynamesecond, you have to store something locally and keep track of that state17:28
srbguptaBoth of your statements are correct17:29
notmynameyeah, that's what I don't like about what I suggested. however, there's not really a better/different way to validate etags for large objects17:30
srbgupta:(17:30
notmynameif I had a checksum/hash algorithm that would allow for combined checksums to equal the checksum of the combined contents, then I could make a better feature for you17:32
srbguptayeah.. Was looking for something like that.17:33
*** klrmn has joined #openstack-swift17:33
notmynameyeah, me too :-)17:33
srbguptaOk.. I will post if I will get some better solution for this.17:33
notmynameour of curiosity, what sort of data are you storing in swift?17:34
srbguptaThese are the log.gz files, but are very important for future reference17:34
*** baojg has quit IRC17:37
notmynametorgomatic: any better ideas for srbgupta?17:37
*** baojg has joined #openstack-swift17:38
notmynamejoeljwright: how's work on https://review.openstack.org/#/c/365371/ going?17:49
patchbotpatch 365371 - swift - Add Preamble and Postamble to SLO and SegmentedIte...17:49
notmynamekota_: mahatic: mattoliverau: acoles: I don't see any agenda updates for the 0700 meeting this week, nor do I know who's chairing17:52
torgomaticnotmyname: I'd suggest sticking the real checksum in object metadata and comparing that17:52
notmynamewhich makes me think we should have the 0700 meeting17:52
notmynametorgomatic: that is a very simple and very good idea :-)17:52
notmynamesrbgupta: does that make sense?17:52
torgomaticLarge-object ETags aren't really useful; they have some semantic meaning, but it's a real pain in the rear to use them effectively. Basically they meet the HTTP requirements for "ETag", but nothing beyond that.17:52
acolesnotmyname: did you mean 'should'?17:53
acolesnotmyname: I'm probably the only one on your list that is awake17:53
torgomaticPlus, if you do your own checksum storage, you can use whatever checksum(s) you want and you're not bound by what Swift has support for. Depending on what you're doing, this could be handy.17:53
notmynamewhich makes me think we *shouldn't* have the 0700 meeting17:53
notmynameacoles: yeah, I'm trusting (1) you know what I mean ;-) and (2) bouncer playback17:54
notmynameI can't count on (1), but maybe tech won't fail me :-)17:54
acoleslol17:54
acolesnotmyname: I'm just reading log of the last 0700. great job mattoliverau17:55
acolesnotmyname: mattoliverau volunteered himself to chair next meeting http://eavesdrop.openstack.org/meetings/swift/2017/swift.2017-08-09-07.01.log.html#l-5817:56
notmynameoh, I see that :-)17:57
acolesI'll try to get online in time to keep him company IF it happens17:58
*** srbgupta has quit IRC18:05
*** itlinux has joined #openstack-swift18:06
*** mvk_ has quit IRC18:18
*** rmcall has joined #openstack-swift18:21
*** itlinux has quit IRC18:26
*** itlinux has joined #openstack-swift18:42
*** ChubYann has joined #openstack-swift18:49
timburkethinking more about checksums and validations -- i suppose what people ultimately want is a way to verify "all the bytes sent by one client are properly received on a subsequent download" -- a separate checksum in metadata *could* help, but (1) we'll need something like p 336323 before we can *really* trust it and (2) you won't know that your data's bad until someone tries to access it19:05
patchbothttps://review.openstack.org/#/c/336323/ - swift - Add checksum to object extended attributes19:05
timburkebecause ultimately, ^^^ requires three different "validations". there's "swift stored all the bytes the client sent correctly", "over time, the stored representation in swift is still correct", and "when a client downloads this, they can verify that it matches what swift sent"19:05
timburkethat middle one's important!19:05
*** itlinux has quit IRC19:21
timburkep 212824 might be able to help with it -- maybe put a bunch of (SLO, segment data) pairs in a queue with a bunch of workers that head the segments and quarantine (maybe? maybe that's too heavy handed?) the SLO if there's a failure19:25
patchbothttps://review.openstack.org/#/c/212824/ - swift - Let developers/operators add watchers to object audit19:25
timburkebasically, notmyname's instinct was spot-on -- torgomatic will save us all ;-)19:27
timburkethis did lead me to noticing that zlib has a crc32_combine function, though, so that's cool https://github.com/madler/zlib/blob/v1.2.11/crc32.c#L42819:30
*** srbgupta has joined #openstack-swift19:32
*** silor1 has joined #openstack-swift19:33
*** silor has quit IRC19:35
*** silor1 is now known as silor19:35
*** itlinux has joined #openstack-swift19:47
notmynametimburke: but in your 3 different validations, the first and third can be grouped and the second treated separately19:54
timburkenotmyname: kinda. but sending x-object-meta-original-sha512 or some such is only useful for the third one; swift just sees meta to store during the PUT19:56
notmynameyes (today). I'm only stating that we don't have to solve all 3 for every object at the same time. we maybe could support something that would do 1 and 3 but leave 2 for later19:57
notmynamebut I still don't have any idea how to solve 1 in the large object case19:57
notmynamethat is, in the case of objects that are manifests19:58
openstackgerritClay Gerrard proposed openstack/swift master: Add tests for memcache logging change  https://review.openstack.org/49639220:04
*** silor has quit IRC20:07
torgomaticman, what happened to patch 336323? it needs a rebase, but I don't recognize half the stuff in there, so rebasing gets confusing to me :|20:07
patchbothttps://review.openstack.org/#/c/336323/ - swift - Add checksum to object extended attributes20:07
*** baojg has quit IRC20:09
timburkemain thing i remember about it is that it prompted us to have the split between gate-swift-python27-ubuntu-xenial and gate-swift-tox-xfs-tmp-py27-ubuntu-xenial20:09
notmynameyeah, and saio should have and xfs tmpdir too20:09
notmynamebut all that is working now20:09
*** baojg has joined #openstack-swift20:09
*** cschwede_ has quit IRC20:27
*** rmcall has quit IRC20:30
*** rmcall has joined #openstack-swift20:31
*** lucasxu has quit IRC20:31
*** psachin has quit IRC20:34
*** baojg has quit IRC20:38
*** baojg has joined #openstack-swift20:38
*** rcernin has joined #openstack-swift20:50
*** itlinux has quit IRC20:59
claygtimburke: i don't care what the tests say - just that they blow up when I break the change - have at it21:00
claygFWIW the debug logger doesn't include tracebacks - just the message that goes with the traceback21:01
claygthe memcache vs memcached was me trying to validate it broke and getting bored/careless21:01
claygalso $%^& virtualbox networking21:01
*** itlinux has joined #openstack-swift21:06
tdasilvaheh, was just looking at those patches21:06
tdasilvaguess will wait for timburke21:06
*** mat128 has quit IRC21:15
*** mat128 has joined #openstack-swift21:16
*** mat128 has quit IRC21:17
timburke`FAILED (SKIP=14, failures=61)` oh man, what a yak! it's gonna be great though. looking forward to being able to disambiguate between logging.error and logging.exception in tests :D21:28
claygdon't do it21:29
claygdon't do it21:29
claygdon't do it21:29
timburkeclayg: hey, you wanted testts... coulda just +A'ed :P21:29
claygok21:30
claygtry to keep orthogonal changes separate - adding infrastructure to test logging in the tests for the module where you changed logging; great - reworking a bunch of unrelated test infrastructure can hopefully be on a different non-blocking review/priority cycle21:32
openstackgerritTim Burke proposed openstack/swift master: Socket errors don't warrant tracebacks when talking to memcached  https://review.openstack.org/49615021:35
*** catintheroof has quit IRC22:01
*** itlinux has quit IRC22:03
*** baojg has quit IRC22:11
*** baojg has joined #openstack-swift22:12
*** baojg has quit IRC22:20
*** baojg has joined #openstack-swift22:20
*** baojg has quit IRC22:25
*** baojg has joined #openstack-swift22:25
notmynameFYI this is my response to the openstack marketing people about "what is new in pike": https://gist.github.com/notmyname/5fa8ca9cf0857ec2f9bdedc44c348e6822:26
*** baojg has quit IRC22:30
*** baojg has joined #openstack-swift22:30
*** rcernin has quit IRC22:32
openstackgerritTim Burke proposed openstack/swift master: Replicator timeouts don't deserve tracebacks  https://review.openstack.org/49643322:48
notmynametimburke: traceback are for replicators that don't time out?22:49
timburkegd right they are22:49
*** itlinux has joined #openstack-swift22:54
*** srbgupta has quit IRC22:58
*** baojg has quit IRC23:05
mattoliveraumorning23:06
mattoliverauacoles: thanks, please come it was lonely last time :P23:06
*** baojg has joined #openstack-swift23:06
*** vint_bra has quit IRC23:18
*** mat128 has joined #openstack-swift23:19
*** baojg has quit IRC23:22
notmynamemattoliverau: so are you gonna chair? is there a need for a meeting? nothings been added to the agenda23:22
notmynameI'd propose skipping this one, and then there is one more 0700 meeting right before the ptg23:22
*** chetna has quit IRC23:23
*** baojg has joined #openstack-swift23:23
mattoliverauI'm fine with that.. I added info to the agenda but nothing more interesting then letting people know things23:23
*** mat128 has quit IRC23:24
notmynamethere's only one thing that'd I'd like some follow-up on, and that's joeljwright's patch 36537123:24
patchbothttps://review.openstack.org/#/c/365371/ - swift - Add Preamble and Postamble to SLO and SegmentedIte...23:24
notmynameand aside from that, the only reason to have an 0700 or 2100 meeting is for info update. so I'd propose skipping23:25
notmyname(but I'm waiting to hear from joeljwright )23:25
mattoliveraukk, I'll wait until and see if mahatic and kota_ turn up and see if there ok with skipping it then.. But yeah I'm fine with it. I just gathered info from emails over the last week, so really everyone should have already read em ;)23:27
*** kei_yama has joined #openstack-swift23:27
notmynameyep. sounds good. thanks23:27
mattoliverautdasilva: I've got version 1 of statsd/collectd/influxdb/grafana playbook working (well should work, I'll test it somemore today), working for setting up the statsd server in swift-ansible. Once I test it again, I'll PR over to your repo.23:31
mattoliveraunotmyname: the playbook probably work great for runway as well :) ^^23:31
mattoliverauI guess my next step is to see if I can even run runway on OpenSuse, and if not port it. The nodes can happily be whatever, so ubuntu. Just the running host.23:34
*** chsc has quit IRC23:38
*** baojg has quit IRC23:40
notmynamemattoliverau: cool!23:40
*** baojg has joined #openstack-swift23:41
*** baojg has quit IRC23:45
*** baojg has joined #openstack-swift23:46
*** itlinux has quit IRC23:47
*** baojg has quit IRC23:50
*** baojg has joined #openstack-swift23:52
*** gyee has quit IRC23:53
*** gyee has joined #openstack-swift23:57
*** baojg has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!