| *** haomaiwa_ has quit IRC | 00:01 | |
| *** haomaiwang has joined #openstack-swift | 00:01 | |
| *** chlong has quit IRC | 00:11 | |
| *** garthb has quit IRC | 00:18 | |
| *** gyee has quit IRC | 00:20 | |
| *** rickyrem has joined #openstack-swift | 00:29 | |
| *** briancline has quit IRC | 00:38 | |
| *** briancline has joined #openstack-swift | 00:45 | |
| *** StraubTW has quit IRC | 00:47 | |
| *** sheel has quit IRC | 00:47 | |
| *** klamath has quit IRC | 00:52 | |
| *** haomaiwang has quit IRC | 01:01 | |
| *** haomaiwang has joined #openstack-swift | 01:01 | |
| *** lyrrad has quit IRC | 01:10 | |
| *** sgundur has joined #openstack-swift | 01:11 | |
| *** sgundur has left #openstack-swift | 01:11 | |
| *** Jeffrey4l has joined #openstack-swift | 01:15 | |
| *** klrmn1 has quit IRC | 01:15 | |
| *** dmorita has quit IRC | 01:21 | |
| *** rickyrem has quit IRC | 01:33 | |
| *** rickyrem has joined #openstack-swift | 01:34 | |
| *** chlong has joined #openstack-swift | 01:36 | |
| *** dmorita has joined #openstack-swift | 01:37 | |
| *** dmorita has quit IRC | 01:41 | |
| *** haomaiwang has quit IRC | 01:44 | |
| *** baojg has joined #openstack-swift | 01:46 | |
| *** Jeffrey4l has quit IRC | 01:58 | |
| *** mtreinish has quit IRC | 01:59 | |
| *** mtreinish has joined #openstack-swift | 02:01 | |
| *** panda has quit IRC | 02:05 | |
| *** panda has joined #openstack-swift | 02:06 | |
| *** dmorita has joined #openstack-swift | 02:23 | |
| *** klrmn has joined #openstack-swift | 02:27 | |
| *** dmorita has quit IRC | 02:28 | |
| *** mingdang1 has joined #openstack-swift | 02:37 | |
| *** StraubTW has joined #openstack-swift | 02:48 | |
| mahatic | good morning | 02:52 |
|---|---|---|
| *** StraubTW has quit IRC | 02:52 | |
| notmyname | good evening | 02:53 |
| *** dmorita has joined #openstack-swift | 03:10 | |
| tdasilva | good night | 03:13 |
| notmyname | I'm reminded of the truman show. http://alexpeak.com/art/films/tts/excerpts/greetings.jpg | 03:14 |
| *** dmorita has quit IRC | 03:14 | |
| *** links has joined #openstack-swift | 03:27 | |
| *** haomaiwang has joined #openstack-swift | 03:29 | |
| *** haomaiwang has quit IRC | 04:01 | |
| *** haomaiwang has joined #openstack-swift | 04:01 | |
| *** klrmn has quit IRC | 04:05 | |
| *** Zyric has quit IRC | 04:08 | |
| *** dmorita has joined #openstack-swift | 04:11 | |
| *** dmorita has quit IRC | 04:16 | |
| *** sheel has joined #openstack-swift | 04:18 | |
| *** Zyric has joined #openstack-swift | 04:22 | |
| *** treaki__ has joined #openstack-swift | 04:23 | |
| *** treaki_ has quit IRC | 04:25 | |
| *** takashi has joined #openstack-swift | 04:26 | |
| *** chlong has quit IRC | 04:56 | |
| *** rickyrem has quit IRC | 04:57 | |
| *** tanee has quit IRC | 04:58 | |
| *** haomaiwang has quit IRC | 05:01 | |
| *** haomaiwang has joined #openstack-swift | 05:01 | |
| *** tanee has joined #openstack-swift | 05:03 | |
| *** dmorita has joined #openstack-swift | 05:08 | |
| *** chlong has joined #openstack-swift | 05:12 | |
| *** dmorita has quit IRC | 05:12 | |
| *** hosanai has joined #openstack-swift | 05:15 | |
| *** ChanServ sets mode: +v hosanai | 05:15 | |
| *** baojg has quit IRC | 05:19 | |
| *** hosanai has quit IRC | 05:33 | |
| *** trifon has joined #openstack-swift | 05:42 | |
| *** asettle has quit IRC | 05:44 | |
| *** asettle has joined #openstack-swift | 05:44 | |
| *** asettle has quit IRC | 05:45 | |
| *** pcaruana has quit IRC | 05:48 | |
| *** takashi has quit IRC | 05:51 | |
| *** haomaiwang has quit IRC | 06:01 | |
| *** haomaiwang has joined #openstack-swift | 06:01 | |
| *** panda has quit IRC | 06:05 | |
| *** panda has joined #openstack-swift | 06:06 | |
| *** chlong has quit IRC | 06:08 | |
| *** chlong has joined #openstack-swift | 06:22 | |
| *** trifon has quit IRC | 06:22 | |
| *** rcernin has joined #openstack-swift | 06:23 | |
| *** sheel has quit IRC | 06:27 | |
| *** baojg has joined #openstack-swift | 06:27 | |
| *** StraubTW has joined #openstack-swift | 06:27 | |
| *** sheel has joined #openstack-swift | 06:29 | |
| *** StraubTW has quit IRC | 06:32 | |
| *** trifon has joined #openstack-swift | 06:35 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift: WIP: Swift Automated Tiering https://review.openstack.org/287057 | 06:36 |
| *** ChubYann has quit IRC | 06:39 | |
| *** baojg has quit IRC | 06:42 | |
| *** esker has quit IRC | 06:46 | |
| *** baojg has joined #openstack-swift | 06:47 | |
| *** haomaiwang has quit IRC | 07:01 | |
| *** haomaiwa_ has joined #openstack-swift | 07:01 | |
| *** silor has joined #openstack-swift | 07:02 | |
| *** lifeless has quit IRC | 07:06 | |
| *** baojg has quit IRC | 07:06 | |
| *** baojg has joined #openstack-swift | 07:08 | |
| *** silor1 has joined #openstack-swift | 07:08 | |
| *** silor has quit IRC | 07:11 | |
| *** silor1 is now known as silor | 07:11 | |
| *** pcaruana has joined #openstack-swift | 07:32 | |
| *** daemontool has joined #openstack-swift | 07:33 | |
| *** wanghua has joined #openstack-swift | 07:38 | |
| *** tesseract has joined #openstack-swift | 07:41 | |
| *** tesseract is now known as Guest27982 | 07:42 | |
| *** geaaru has joined #openstack-swift | 07:52 | |
| *** Trixboxer has quit IRC | 07:54 | |
| *** Trixboxer has joined #openstack-swift | 07:55 | |
| *** haomaiwa_ has quit IRC | 08:01 | |
| *** haomaiwang has joined #openstack-swift | 08:01 | |
| openstackgerrit | Merged openstack/swift: 2.7.0 authors and changelog updates https://review.openstack.org/296175 | 08:01 |
| *** Zyric has quit IRC | 08:08 | |
| *** rledisez has joined #openstack-swift | 08:09 | |
| *** mmcardle has joined #openstack-swift | 08:09 | |
| *** dmorita has joined #openstack-swift | 08:10 | |
| *** mmcardle1 has joined #openstack-swift | 08:11 | |
| *** mmcardle has quit IRC | 08:11 | |
| *** Zyric has joined #openstack-swift | 08:12 | |
| *** dmorita has quit IRC | 08:14 | |
| openstackgerrit | David Liu proposed openstack/swift: Handle tempurl Content-Disposition header missing from HEAD https://review.openstack.org/296936 | 08:17 |
| openstackgerrit | Marek Kaleta proposed openstack/python-swiftclient: Add copy object method https://review.openstack.org/280200 | 08:31 |
| *** jmccarthy has quit IRC | 08:32 | |
| *** cbartz has joined #openstack-swift | 08:33 | |
| *** jmccarthy has joined #openstack-swift | 08:35 | |
| cschwede | acoles: Good Morning! so, i’m looking into this rolling upgrade thing | 08:36 |
| cschwede | and i’m wondering about https://governance.openstack.org/reference/tags/assert_supports-rolling-upgrade.html#tag-assert-supports-rolling-upgrade | 08:36 |
| *** acoles_ is now known as acoles | 08:37 | |
| *** sheel has quit IRC | 08:37 | |
| cschwede | does that mean a rolling upgrade within a cloud, ie for example upgrade nova at time t0, but no other service, then at t1 keystone, and so on? | 08:37 |
| cschwede | or upgrade within a service, ie swift-proxy, then swift-object server, and so on | 08:38 |
| acoles | cschwede: good morning, i just got here | 08:38 |
| *** chlong has quit IRC | 08:38 | |
| acoles | cschwede: tbh, i am not sure, but what i took away from last night's discussion was at least a proxy server running version x and a/c/o servers running version y | 08:38 |
| cschwede | acoles: yes, so that would match with patch 64074, that added rolling upgrade tests for nova a while ago | 08:40 |
| patchbot | cschwede: https://review.openstack.org/#/c/64074/ - openstack-dev/grenade - Add rolling upgrade test for nova-compute (MERGED) | 08:40 |
| cschwede | so only the api is updated, but not nova-compute then | 08:41 |
| cschwede | and i think that’s how to do this in swift as well | 08:42 |
| acoles | cschwede: what i was planning to try, as an experiment, was running swift services inside virtual envs, and then rolling back the version in one venv | 08:43 |
| cschwede | so the two code env are setup by devstack (old branch and new branch), so we don’t need to take care of that (using a virtualenv or similar) - iiuc | 08:43 |
| *** Trixboxer has quit IRC | 08:43 | |
| *** Trixboxer has joined #openstack-swift | 08:44 | |
| acoles | cschwede: so you think grenade already has some infrastructure to manage multiple versions? how is it deploying the services, in separate vms? | 08:48 |
| cschwede | acoles: yes, it has this; and it’s using virtualenvs. see patch 158376 | 08:49 |
| patchbot | cschwede: https://review.openstack.org/#/c/158376/ - openstack-dev/grenade - Prepare for Services in virtual env (MERGED) | 08:49 |
| cschwede | especially https://review.openstack.org/#/c/158376/5/upgrade-swift | 08:50 |
| patchbot | cschwede: patch 158376 - openstack-dev/grenade - Prepare for Services in virtual env (MERGED) | 08:50 |
| *** Jeffrey4l has joined #openstack-swift | 08:50 | |
| *** early has quit IRC | 08:50 | |
| cschwede | so i’m wondering, with this already in place, it shouldn’t be too hard to only update the proxy for example | 08:50 |
| *** jordanP has joined #openstack-swift | 08:52 | |
| *** zaitcev has quit IRC | 08:54 | |
| *** joeljwright has joined #openstack-swift | 08:56 | |
| *** ChanServ sets mode: +v joeljwright | 08:56 | |
| *** zaitcev has joined #openstack-swift | 08:57 | |
| *** ChanServ sets mode: +v zaitcev | 08:57 | |
| acoles | cschwede: OIC. so the sequence of events is install and start all services at version x, then upgrade to version y, then re-start only a subset of services. i can see how that can work in a single venv. | 09:00 |
| cschwede | acoles: yes, i think that’s the plan | 09:00 |
| *** haomaiwang has quit IRC | 09:01 | |
| *** haomaiwang has joined #openstack-swift | 09:01 | |
| acoles | cschwede: i was trying to figure out how to simultaneously start different services on different versions, which is why i was thinking multiple venvs. | 09:01 |
| *** stantonnet has quit IRC | 09:02 | |
| acoles | cschwede: have you setup a grenade test environment locally? | 09:03 |
| cschwede | so, instead of stopping all swift services in https://github.com/openstack-dev/grenade/blob/master/projects/30_swift/shutdown.sh, we could only stop the proxy maybe? and that get’s restarted after the upgrade | 09:03 |
| cschwede | acoles: no, at least not yet... | 09:03 |
| acoles | hehe. i gave up on devstack a while ago :/ | 09:03 |
| *** stantonnet has joined #openstack-swift | 09:04 | |
| acoles | cschwede: yes, i think you have a good idea there. looks like we'd need to break out, or parameterise, stop_swift to stop only specific services | 09:05 |
| cschwede | ok, i’ll try to setup a local grenade vm to test this - i’ll get back to you later | 09:06 |
| cschwede | acoles: ^^ | 09:06 |
| acoles | cschwede: thanks for doing the research! i am about to commute to office, and may try the same i.e. setup grenade, in case i can be of nay help later | 09:07 |
| acoles | s/nay/any/ | 09:07 |
| *** acoles is now known as acoles_ | 09:15 | |
| *** kei_yama has quit IRC | 09:38 | |
| *** joeljwright1 has joined #openstack-swift | 09:48 | |
| *** cbartz has left #openstack-swift | 09:51 | |
| *** joeljwright has quit IRC | 09:52 | |
| *** early has joined #openstack-swift | 09:53 | |
| *** baojg has quit IRC | 09:55 | |
| *** _JZ_ has quit IRC | 09:56 | |
| *** haomaiwang has quit IRC | 10:01 | |
| *** haomaiwang has joined #openstack-swift | 10:01 | |
| *** Jeffrey4l has quit IRC | 10:06 | |
| cschwede | so, i think we need three patches to test rolling upgrades: | 10:08 |
| cschwede | 1. devstack: a method to stop only the swift proxy server | 10:08 |
| cschwede | 2. grenade: adding a flag, that will only stop the proxy using the method from devstack | 10:09 |
| cschwede | 3. project-config: adding another job(?) to run grenade with a flag for the rolling upgrade | 10:10 |
| cschwede | i’m not sure if this is the right way to do this, but it somehow makes sense to me | 10:10 |
| cschwede | i will upload patch 1+2 later, currently running grenade on my vm to test this | 10:10 |
| patchbot | cschwede: https://review.openstack.org/#/c/1/ - openstack-infra/system-config - Add puppet module for ssh that installs an sshd_co... (MERGED) | 10:10 |
| cschwede | ? | 10:11 |
| cschwede | acoles_: notmyname: ^^ | 10:11 |
| cschwede | at least that’s something we can discuss with TC, and iterate from that | 10:11 |
| cschwede | first patch: 296991 | 10:15 |
| cschwede | patch 296991 | 10:15 |
| patchbot | cschwede: https://review.openstack.org/#/c/296991/ - openstack-dev/devstack - WIP: Add stop_swift_proxy method | 10:15 |
| *** asettle has joined #openstack-swift | 10:16 | |
| *** Jeffrey4l has joined #openstack-swift | 10:18 | |
| *** acoles_ is now known as acoles | 10:18 | |
| acoles | cschwede: ack, sounds good | 10:19 |
| *** early has quit IRC | 10:21 | |
| *** asettle has quit IRC | 10:24 | |
| cschwede | next patch 296997 | 10:24 |
| patchbot | cschwede: https://review.openstack.org/#/c/296997/ - openstack-dev/grenade - WIP: Add option to test Swift rolling upgrade test | 10:24 |
| *** joeljwright has joined #openstack-swift | 10:28 | |
| *** ChanServ sets mode: +v joeljwright | 10:28 | |
| *** StraubTW has joined #openstack-swift | 10:29 | |
| *** early has joined #openstack-swift | 10:30 | |
| *** joeljwright1 has quit IRC | 10:31 | |
| *** StraubTW has quit IRC | 10:34 | |
| *** Jeffrey4l has quit IRC | 10:35 | |
| cschwede | and, finally patch 297005 | 10:38 |
| patchbot | cschwede: https://review.openstack.org/#/c/297005/ - openstack-infra/project-config - WIP: Add grenade job for Swift rolling upgrade tests | 10:38 |
| cschwede | ^^ these patches are all WIP, and open for discussion. as i said, not sure it’s the current right way to do it, so let’s discuss this with infra & TC | 10:39 |
| *** daemontool has quit IRC | 10:46 | |
| *** km has quit IRC | 10:52 | |
| *** eranrom has joined #openstack-swift | 10:54 | |
| eranrom | acoles,: Hi, A question on container sync with fast post: To my surprise the following seems to work: | 10:57 |
| eranrom | 1. put object in source and have it synced to dest. | 10:57 |
| eranrom | 2. update the data on source at t1 | 10:57 |
| eranrom | 3. update metadata and content type on dest at t2>t1 | 10:58 |
| eranrom | 4. sync. | 10:58 |
| eranrom | Apparently the data got replicated to the other side. | 10:58 |
| eranrom | which IMO is the right behavior, although, I did not think it would work given the data ts1 being smaller then ts2.... | 10:59 |
| eranrom | If this is in fact the case, and considering the other probe tests in place, we can say container sync works with fast-post. yay! | 11:00 |
| *** haomaiwang has quit IRC | 11:00 | |
| *** haomaiwang has joined #openstack-swift | 11:04 | |
| *** lifeless has joined #openstack-swift | 11:07 | |
| acoles | eranrom: actually i would not have expected that outcome. unfortunately my dev environment has broken this morning so i cannot reproduce right now. but at step3 the dest becomes newer than the source, so I would not expect a sync at step 4. | 11:17 |
| acoles | eranrom: are you up to date with master, i.e. do you have https://review.openstack.org/#/c/270961/ | 11:18 |
| patchbot | acoles: patch 270961 - swift - Container-Sync to perform HEAD before PUT object o... (MERGED) | 11:18 |
| acoles | eranrom: i can see how your scenario worked *prior* to that patch^^ because then the PUT would be accepted | 11:19 |
| openstackgerrit | Eran Rom proposed openstack/swift: Add process level concurrency to container sync https://review.openstack.org/210099 | 11:19 |
| acoles | eranrom: but the behavior on master now should be the same as with post-as-copy | 11:19 |
| eranrom | acoles,: right. False alarm :-( indeed with patch 270961) the scenarion fails. | 11:23 |
| patchbot | eranrom: https://review.openstack.org/#/c/270961/ - swift - Container-Sync to perform HEAD before PUT object o... (MERGED) | 11:23 |
| acoles | eranrom: phew, you made me sweat there for a moment ;) | 11:24 |
| acoles | eranrom: however, your scenario is one that imho it would be good to have work i.e. when source has data at t1 and dest has data at t0 and metadata at t2, we'd like to end up with dest having data at t1 and metadata at t2 | 11:25 |
| acoles | eranrom: that is 'further work' that i am aware needs doing, but i do think that at this point the behavior is equitable with post-as-copy | 11:25 |
| acoles | eranrom: "equitable" is wrong word, i mean equivalent | 11:26 |
| eranrom | acole: I was just looking up "equitable" :) | 11:26 |
| acoles | eranrom: sorry! we need to improve container sync further to achieve what ssync does which is to treat sync of data and metadata separately w.r.t. their timestamps | 11:27 |
| *** mvk_ has quit IRC | 11:28 | |
| eranrom | acoles: two points that comes into mind: | 11:28 |
| acoles | that way we can also save sending content bytes to dest when only metadata has changed on source | 11:28 |
| eranrom | 1. I am not positive that we really want to end up with data at t1 and metadata at t2 - the way I see it in object stores data and metadata should be viewed as a whole, and if the data got updated in one cluster it is does not necessarily make sense to retain the 'new' metadata | 11:30 |
| eranrom | clearly the content type might not make sense anymore.... | 11:31 |
| eranrom | 2. Personally, I think that the current behavior is an acceptable limitation for working with fast-post with container sync. | 11:31 |
| eranrom | That is to say: stating the limitation we may want to encourage working with fast-post together with container sync | 11:32 |
| acoles | eranrom: that's an interesting point. within a cluster the goal is always to make all replicas of an object have consistent state. you are arguing that the semantic of container sync might be different, that we sync newest "composite" state. | 11:38 |
| acoles | eranrom: it depends on use case for sync i think, and whether it is a bi-directional sync of two active clusters or one-directional backup-type use case | 11:40 |
| eranrom | acoles,: I think that I am arguing that the data ts, ct ts and md ts are not symmetric. That is: a newer data timestamp should cause older ct,md to 'vanish'. | 11:41 |
| *** jmb__ has joined #openstack-swift | 11:42 | |
| eranrom | acoles: well, not sure that it has to do with symmetry... Its just that new data implies also new ct and metadata | 11:42 |
| acoles | yes, that should always be true | 11:43 |
| eranrom | acoles: ok, so is this is the way that a consistent state is achieved in a single cluster - right? | 11:44 |
| acoles | eranrom: within a cluster your statement is true: i.e. " a newer data timestamp should cause older ct,md to 'vanish'" | 11:45 |
| acoles | and its also true for sync | 11:45 |
| eranrom | acoles,: ok, so I probably missed something along the way. | 11:45 |
| eranrom | acoles: The scenario I tried above, was trying to test that this is also the case for container sync: | 11:46 |
| eranrom | source has newer data, target has newer metadata but older data | 11:46 |
| acoles | the case i was considering was, for sync, source has t1.data, dest has t0.data and t2.meta - what would you like the outcome of a sync to be? | 11:46 |
| acoles | :) right, that^^ | 11:47 |
| acoles | at the moment the outcome is no change, which is the same outcome as if the same events had occurred with post-as-copy | 11:48 |
| eranrom | acoles,: dest shoudl be, IMO, t1.data along with source metadata - as t2.meta refers to an 'obsolete' data | 11:48 |
| acoles | eranrom: ok, so that ^^ is what i think you observed before updating to have patch 270961, but I don't think you'd observe that with HEAD of master | 11:50 |
| patchbot | acoles: https://review.openstack.org/#/c/270961/ - swift - Container-Sync to perform HEAD before PUT object o... (MERGED) | 11:50 |
| eranrom | acoles: Let me test this again. | 11:50 |
| acoles | eranrom: let me try to fix my system.. :/ | 11:51 |
| *** sheel has joined #openstack-swift | 11:58 | |
| *** haomaiwang has quit IRC | 12:01 | |
| *** haomaiwang has joined #openstack-swift | 12:01 | |
| eranrom | acoles: Alright, tested again with 270961 and the behavior is what I (we? - not sure anymore:-) thought - the new data is not propegated. | 12:05 |
| eranrom | acoles,: or in other words: the metadata timestamp 'wins' | 12:05 |
| *** daemontool has joined #openstack-swift | 12:07 | |
| eranrom | acoles: back to the discussion: I guess you were right with the 'depends on use case' remark. If one is doing a unidirectional sync then a metadata update to the destination is most likely a metadata update that reflects the local data, and hence it needs to 'vanish' as noew data comes in from the source | 12:07 |
| eranrom | the local data == the local data at the destination | 12:08 |
| eranrom | acoles,: I am not sure I have an opinion on what is the right behavior for bi-directional sync though :-) | 12:09 |
| acoles | eranrom: ok, so I am satisfied that what you now see is what I would expect, and that it is the same behavior as if you were using post-as-copy | 12:10 |
| acoles | with post-as-copy, when you make a post to the destination object at time t2, it's data timestamp is effectively advance to t2. then source data at t1 would never be sync'd to the dest. | 12:11 |
| acoles | eranrom: and i also understand the alternative behaviour you are describing, and I am not sure what I think of that proposal. Need time to consider it some more. | 12:12 |
| *** kairat has joined #openstack-swift | 12:13 | |
| kairat | Hi guys | 12:13 |
| kairat | When i got 499 error when creating object on object server | 12:13 |
| kairat | what does it mean | 12:13 |
| kairat | who break the connection | 12:13 |
| acoles | eranrom: away for a while | 12:14 |
| kairat | client who requested object create (me) | 12:14 |
| kairat | or swift proxy server | 12:14 |
| kairat | Could you please help me with some info, I am trying to debug a bug between glance and swift | 12:15 |
| *** daemontool has quit IRC | 12:15 | |
| *** mvk_ has joined #openstack-swift | 12:35 | |
| *** StraubTW has joined #openstack-swift | 12:46 | |
| eranrom | acoles,: Sure, Agree this requires more thought. Thanks | 12:48 |
| *** haomaiwang has quit IRC | 13:01 | |
| *** haomaiwang has joined #openstack-swift | 13:01 | |
| *** haomaiwang has quit IRC | 13:07 | |
| *** NM has joined #openstack-swift | 13:08 | |
| *** NM has quit IRC | 13:09 | |
| *** mingdang1 has quit IRC | 13:23 | |
| *** jmb__ has quit IRC | 13:27 | |
| *** links has quit IRC | 13:28 | |
| jrichli | kairat: this means the client disconnected before the server expected it to. there are a few reasons for that. one thing to look at is the request content length to be sure its correct. | 13:36 |
| openstackgerrit | oshritf proposed openstack/swift: Per container stat. report https://review.openstack.org/281814 | 13:38 |
| kairat | jrichli, who can be the client | 13:42 |
| kairat | proxy-server? | 13:42 |
| kairat | or glance itself | 13:43 |
| jrichli | whatever sent the http request that is getting the 499 | 13:43 |
| jrichli | we you able to get some swift logs? | 13:43 |
| kairat | yep | 13:44 |
| kairat | just a sec | 13:44 |
| jrichli | the 499 should be logged - and then you can see which request generated that response | 13:44 |
| cschwede | acoles: notmyname: ok, so i think the proposed patches for the rolling upgrade tests should work fine, at least they are for me. i added an process list output as a comment on patch 296997 | 13:45 |
| patchbot | cschwede: https://review.openstack.org/#/c/296997/ - openstack-dev/grenade - WIP: Add option to test Swift rolling upgrade test | 13:45 |
| kairat | jrichli, http://paste.openstack.org/show/491711/ | 13:45 |
| kairat | that's object-server.log | 13:46 |
| acoles | cschwede: ack, thanks. | 13:46 |
| tdasilva | acoles, cschwede: not trying to make too much fuss about it, but wouldn't you typically do the oposite of first upgrading storage services and then proxy? | 13:47 |
| tdasilva | just wondering if it should be the other way around | 13:48 |
| acoles | cschwede: i haven't got far setting up grenade ...vm problems | 13:48 |
| cschwede | tdasilva: well, we could change that, but before i spent even more time on this i want to get some feedback from infra and the TC if this is the way to go. we can still improve this | 13:49 |
| jrichli | kairat: is the content-length being specified on the put? does it match the length of the data being sent? | 13:51 |
| *** daemontool has joined #openstack-swift | 13:52 | |
| *** sgundur has joined #openstack-swift | 13:58 | |
| *** panda has quit IRC | 14:05 | |
| *** panda has joined #openstack-swift | 14:05 | |
| kairat | it is hard to check | 14:06 |
| kairat | ah | 14:06 |
| kairat | we are using chunked requests | 14:07 |
| kairat | so content-length is absent at all | 14:07 |
| kairat | jrichli, ^ | 14:07 |
| *** asettle has joined #openstack-swift | 14:12 | |
| *** jmb__ has joined #openstack-swift | 14:13 | |
| *** asettle has quit IRC | 14:16 | |
| jrichli | kairat: ok. the server was expecting more data for some reason, and it either detected a disconnect or may have timed out waiting for more data. to find out why, i'd have to debug to see where the problem is happening. | 14:16 |
| jrichli | maybe there is somebody else here who has some suggestions | 14:17 |
| kairat | jrichli, thanks a lot | 14:17 |
| kairat | I also asked to enable debug on CI env where we got this failure | 14:17 |
| kairat | *debug on object-server and container-server | 14:18 |
| jrichli | kairat: there are some places in the proxy/controller/obj that swift will raise that error, so you may as well request debug there too. although, i dont really see debug logging that would help in that file. | 14:21 |
| jrichli | (debugging on proxy, that is) | 14:21 |
| jrichli | oh, nm. we saw that it was in the obj srv | 14:22 |
| kairat | yeah | 14:23 |
| *** vint_bra has joined #openstack-swift | 14:24 | |
| *** ametts has joined #openstack-swift | 14:33 | |
| *** esker has joined #openstack-swift | 14:36 | |
| *** esker has quit IRC | 14:36 | |
| *** esker has joined #openstack-swift | 14:37 | |
| *** haomaiwang has joined #openstack-swift | 14:56 | |
| hurricanerix | jrichli: is there a blueprint/spec/overview of all the encryption stuff? | 14:57 |
| *** kairat has left #openstack-swift | 14:58 | |
| jrichli | hurricanerix: spec: http://specs.openstack.org/openstack/swift-specs/specs/in_progress/at_rest_encryption.html | 15:00 |
| jrichli | hurricanerix: status: https://etherpad.openstack.org/p/swift-hackathon-feb-2016-encryption | 15:00 |
| *** haomaiwang has quit IRC | 15:01 | |
| *** haomaiwang has joined #openstack-swift | 15:01 | |
| jrichli | hurricanerix: I have been wanting to start the docs - just have a lot to do right now. i think that might be more of what you are looking for. if you have questions, let me know. | 15:01 |
| *** links has joined #openstack-swift | 15:02 | |
| *** trifon has quit IRC | 15:03 | |
| hurricanerix | jrichli: Thanks! | 15:03 |
| *** eranrom has quit IRC | 15:05 | |
| *** StraubTW_ has joined #openstack-swift | 15:08 | |
| *** StraubTW has quit IRC | 15:10 | |
| *** acorwin_ has joined #openstack-swift | 15:12 | |
| *** lifeless has quit IRC | 15:13 | |
| *** david-lyle has quit IRC | 15:13 | |
| *** fbo has quit IRC | 15:13 | |
| *** acorwin has quit IRC | 15:13 | |
| *** dmsimard has quit IRC | 15:13 | |
| *** kota_ has quit IRC | 15:13 | |
| notmyname | good morning | 15:14 |
| notmyname | cschwede: great work on those patches. I just read over the scrollback, and I've got them open | 15:14 |
| notmyname | cschwede: have you raised anything with -infra yet? | 15:16 |
| *** Guest27982 has quit IRC | 15:17 | |
| *** rcernin has quit IRC | 15:17 | |
| *** david-lyle has joined #openstack-swift | 15:17 | |
| *** links has quit IRC | 15:18 | |
| cschwede | notmyname: thx! not yet, i first wanted to ensure we’re on the same line with this. if you agree with the overall approach, we should get in touch with -infra (and probably leave a comment on the TC patch, that there is something in progress) | 15:23 |
| notmyname | cschwede: I think I am, but TBH I'm still fuzzy on how those yaml files set it up. eg I don't actually see/know when swift is actually started. or when the upgrade happens and what versions are used | 15:25 |
| *** dmsimard has joined #openstack-swift | 15:25 | |
| *** wasmum has quit IRC | 15:29 | |
| *** lifeless has joined #openstack-swift | 15:29 | |
| *** fbo has joined #openstack-swift | 15:29 | |
| *** kota_ has joined #openstack-swift | 15:29 | |
| *** wolfe.freenode.net sets mode: +v kota_ | 15:29 | |
| *** Guest27982 has joined #openstack-swift | 15:29 | |
| *** rcernin has joined #openstack-swift | 15:29 | |
| *** lifeless has quit IRC | 15:30 | |
| *** lifeless has joined #openstack-swift | 15:30 | |
| notmyname | cschwede: I can start coordinating with -infra and the TC today to get these looked at (and maybe landed) | 15:33 |
| cschwede | notmyname: that would be very welcome, i’m currently in a meeting | 15:33 |
| *** asettle has joined #openstack-swift | 15:34 | |
| cschwede | notmyname: i can get back to you later if you like to discuss this approach | 15:34 |
| notmyname | cschwede: sure | 15:34 |
| *** asettle has quit IRC | 15:38 | |
| *** garthb has joined #openstack-swift | 15:44 | |
| *** wasmum has joined #openstack-swift | 15:48 | |
| *** clarkb has joined #openstack-swift | 15:49 | |
| clarkb | notmyname: o/ | 15:49 |
| notmyname | clarkb: thanks :-) | 15:49 |
| notmyname | clarkb: background is that we're setting up a rolling upgrade test | 15:49 |
| notmyname | cschwede (in germany) worked on it today. he's in a meeting now, but I'm taking the handoff this morning | 15:50 |
| notmyname | we've got patch 296991 patch 296997 and patch 297005 | 15:50 |
| patchbot | notmyname: https://review.openstack.org/#/c/296991/ - openstack-dev/devstack - WIP: Add stop_swift_proxy method | 15:50 |
| patchbot | notmyname: https://review.openstack.org/#/c/296997/ - openstack-dev/grenade - WIP: Add option to test Swift rolling upgrade test | 15:50 |
| patchbot | notmyname: https://review.openstack.org/#/c/297005/ - openstack-infra/project-config - WIP: Add grenade job for Swift rolling upgrade tests | 15:50 |
| notmyname | cschwede has tested these locally (see the comments), but we'd like an -infra eye on the setup/etc to see if this actually is the right way to do stuff | 15:51 |
| notmyname | and, ideally, if things look good, to land them ASAP | 15:51 |
| clarkb | let me see | 15:51 |
| clayg | are grenade tests always "rolling" | 15:54 |
| clayg | do the storage servers eventually get upgraded in the test? | 15:55 |
| notmyname | clayg: no, I'm told that it's more of a cold upgrade currently. ie stop everything, upgrade everything, start it, run tests | 15:55 |
| clarkb | clayg: no grenades default behavior is bring up services, run some basic functionality testing, stop all services, upgrade everything, start services and run more tests | 15:55 |
| clayg | k, yeah that's how i understood it - so i was questioning the naming here -> https://review.openstack.org/#/c/297005/2/jenkins/jobs/devstack-gate.yaml | 15:56 |
| patchbot | clayg: patch 297005 - openstack-infra/project-config - WIP: Add grenade job for Swift rolling upgrade tests | 15:56 |
| clarkb | the way these rolling tests are set up for swift appears to be only upgrade the proxy server | 15:56 |
| notmyname | clarkb: right. that's the point. to test a scenario where only some of the services are upgraded | 15:56 |
| notmyname | ie what would be partway through a rolling upgrade | 15:56 |
| notmyname | clayg: if we have this test + the tests we already have that test everything upgraded, then we cover the bases | 15:57 |
| clarkb | clayg: I think the terminology used elsewhere for this type of upgrade is 'partial' | 15:57 |
| clarkb | n-cpu-partial but I would have to double check | 15:57 |
| notmyname | well, specifically the reason for working on this (and rushing on it) is because the TC is wanting to strip swift of it's "supports rolling upgrades" tag | 15:58 |
| notmyname | and the requirement listed there is to test a partially upgraded cluster | 15:58 |
| clarkb | the devstack and grenade things look fine to me but I am not core there so you'll probably want someone else to actually confirm that. There is one small issue with the job config I will comment in a few once I figure out how the other grenade jobs handle it | 15:59 |
| notmyname | clarkb: thanks for looking. who's core on devstack and grenade that I can ping about it? | 16:00 |
| clarkb | sdague, mtreinish, dtroyer off hte top of my head | 16:00 |
| *** haomaiwang has quit IRC | 16:01 | |
| notmyname | thanks | 16:01 |
| *** haomaiwang has joined #openstack-swift | 16:01 | |
| clarkb | export DO_NOT_UPGRADE_SERVICES=[n-cpu] is apparently how the nova cpu tests work | 16:01 |
| clarkb | they may ask that you use that in the grenade change | 16:02 |
| notmyname | clarkb: FWIW, ideally, I'd like to have this all squared away by next tuesday. that's the timeframe I'm working under | 16:06 |
| notmyname | clarkb: meaning, landed and running on patches in the gate | 16:06 |
| acoles | cschwede: i am getting closer to having a grenade vm setup. once i have, is it straightforward to point the grenade script at your patches rather than it pulling master branches of devstack? | 16:07 |
| clarkb | ya I think its a fairly simple set of updates because this is a mostly solved problem | 16:07 |
| clarkb | its just slapping this into that framework and gerrit just dumped my comments because I need to log in again aparenlty | 16:07 |
| notmyname | clayg: on patch 257502, you had reviewed but not approved, waiting on follow-up stuff. acoles pushed another patch set and added +2. if you're good with the current one and it gets in the gate this morning, it can be in the mitaka release | 16:08 |
| patchbot | notmyname: https://review.openstack.org/#/c/257502/ - swift - Fix full_listing in internal_client | 16:08 |
| clarkb | notmyname: ok commented on https://review.openstack.org/#/c/297005/ hopefully that is enough detail? | 16:09 |
| patchbot | clarkb: patch 297005 - openstack-infra/project-config - WIP: Add grenade job for Swift rolling upgrade tests | 16:09 |
| notmyname | clarkb: yeah, gerrit does that a lot :-) | 16:09 |
| clarkb | apparenlty if you go anonymous your comment isn't deleted, you can copy it, then login, and paste it back | 16:10 |
| clarkb | woo gerrit | 16:10 |
| notmyname | clarkb: thanks. I'm still trying to understand how it actually is working. where is it actually upgrading code and running the tests? how is it picking the versions of code to run (yes, new code will be HEAD of the branch, but what's the starting point?) | 16:11 |
| *** gyee has joined #openstack-swift | 16:12 | |
| clarkb | notmyname: devstack-gate picks the versions of code to test just below where I linked in my comment https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n287 which is why that case statement falls through | 16:12 |
| clarkb | notmyname: as for what is actually upgrading code and running the tests that is all in grenade | 16:12 |
| clarkb | basicall it is a list of if I am testing a change for $branch then the old branch is foo | 16:13 |
| notmyname | ah ok | 16:13 |
| clarkb | we just aproved the change for liberty -> mitaka and mitaka -> master so that will show up today | 16:13 |
| notmyname | clarkb: the envvar is name BRANCh, but is it really just a git refspec? eg could one specify GRENADE_OLD_BRANCH as a tag? | 16:14 |
| clarkb | notmyname: not the way that is current configured | 16:15 |
| notmyname | ok | 16:15 |
| clarkb | because it upgrades an entire cloud together you need a common ref | 16:15 |
| clarkb | tags unfortunately are not common across the cloud projcets | 16:15 |
| *** mvk_ has quit IRC | 16:15 | |
| notmyname | ah, right. these tests are running tests on other projects too | 16:16 |
| notmyname | there was some talk around just pulling the oldest tag we could find (or even just an old commit SHA) to test rolling upgrades from there. just to show that rolling upgrades work from eg austin release to current master ;-) | 16:17 |
| cschwede | acoles: i don’t think so, because you need the swift_proxy_stop on the liberty branch as well. i simply added that function to both devstack branches manually | 16:19 |
| cschwede | and that reminds me that we probably need a backport to liberty… ah well | 16:19 |
| notmyname | clarkb: so updating the gate wrap script will take a fourth patch, right? or is that the same repo as patch 296991? | 16:20 |
| patchbot | notmyname: https://review.openstack.org/#/c/296991/ - openstack-dev/devstack - WIP: Add stop_swift_proxy method | 16:20 |
| clarkb | notmyname: correct it is a fourth patch | 16:20 |
| notmyname | cschwede: did clarkb's comments make sense? | 16:20 |
| cschwede | clarkb: thanks for your comment, will update the patch | 16:20 |
| cschwede | notmyname: yes, sounds good to me | 16:20 |
| notmyname | cschwede: great. and let's ping the devstacka nd grenade people as soon as those are updated | 16:21 |
| cschwede | i’m not sure when i will be able to update this, will try to work on it tonight | 16:21 |
| cschwede | notmyname: yes, absolutely. i also remove the wip flags then | 16:21 |
| *** eranrom has joined #openstack-swift | 16:21 | |
| *** trifon has joined #openstack-swift | 16:21 | |
| notmyname | cschwede: ok. if you arent' getting to it until later tonight, I can take a stab at updating it (I'll probably need to figure out how to set up a test env for it) | 16:22 |
| *** haomaiwang has quit IRC | 16:22 | |
| *** haomaiwang has joined #openstack-swift | 16:22 | |
| *** haomaiwang has quit IRC | 16:23 | |
| clarkb | for a test env you may be able to take the reproduce.sh script in the logs of other grenade runs and tweak the parameters | 16:23 |
| clarkb | then run that on an ubuntu trusty VM | 16:23 |
| *** haomaiwa_ has joined #openstack-swift | 16:23 | |
| clarkb | in theory that will just work | 16:23 |
| *** haomaiwa_ has quit IRC | 16:24 | |
| clarkb | but I doubt anyone has tried it against grenade recently | 16:24 |
| *** haomaiwang has joined #openstack-swift | 16:25 | |
| *** haomaiwang has quit IRC | 16:26 | |
| *** haomaiwa_ has joined #openstack-swift | 16:26 | |
| *** haomaiwa_ has quit IRC | 16:27 | |
| *** haomaiwang has joined #openstack-swift | 16:27 | |
| *** haomaiwang has quit IRC | 16:28 | |
| *** jordanP has quit IRC | 16:28 | |
| *** haomaiwa_ has joined #openstack-swift | 16:28 | |
| *** haomaiwa_ has quit IRC | 16:29 | |
| *** haomaiwa_ has joined #openstack-swift | 16:29 | |
| *** haomaiwa_ has quit IRC | 16:30 | |
| *** 7F1AAK9L3 has joined #openstack-swift | 16:30 | |
| *** 7F1AAK9L3 has quit IRC | 16:31 | |
| *** haomaiwa_ has joined #openstack-swift | 16:31 | |
| *** haomaiwa_ has quit IRC | 16:32 | |
| *** haomaiwang has joined #openstack-swift | 16:32 | |
| *** haomaiwang has quit IRC | 16:33 | |
| *** chsc has joined #openstack-swift | 16:33 | |
| *** haomaiwa_ has joined #openstack-swift | 16:33 | |
| *** haomaiwa_ has quit IRC | 16:34 | |
| *** haomaiwang has joined #openstack-swift | 16:34 | |
| *** haomaiwang has quit IRC | 16:35 | |
| *** nadeem has joined #openstack-swift | 16:35 | |
| *** haomaiwang has joined #openstack-swift | 16:35 | |
| *** haomaiwang has quit IRC | 16:36 | |
| *** haomaiwang has joined #openstack-swift | 16:36 | |
| *** dmorita has joined #openstack-swift | 16:36 | |
| *** haomaiwang has quit IRC | 16:37 | |
| *** lyrrad has joined #openstack-swift | 16:37 | |
| *** 14WAAKC1C has joined #openstack-swift | 16:37 | |
| *** 14WAAKC1C has quit IRC | 16:38 | |
| *** haomaiwa_ has joined #openstack-swift | 16:38 | |
| *** haomaiwa_ has quit IRC | 16:39 | |
| *** haomaiwang has joined #openstack-swift | 16:39 | |
| *** haomaiwang has quit IRC | 16:40 | |
| *** haomaiwang has joined #openstack-swift | 16:40 | |
| *** haomaiwang has quit IRC | 16:41 | |
| *** haomaiwa_ has joined #openstack-swift | 16:41 | |
| *** haomaiwa_ has quit IRC | 16:42 | |
| timburke | good morning | 16:42 |
| *** haomaiwa_ has joined #openstack-swift | 16:42 | |
| *** haomaiwa_ has quit IRC | 16:43 | |
| *** haomaiwa_ has joined #openstack-swift | 16:43 | |
| *** haomaiwa_ has quit IRC | 16:43 | |
| notmyname | timburke: hello. read the scrollback for interesting developments on the rolling upgrade test. cschwede has done a bunch of good work | 16:44 |
| clayg | oh man, gerrit is getting under my skin today.... "Working" | 16:44 |
| timburke | yep, i saw. thanks cschwede! | 16:44 |
| clayg | ... | 16:44 |
| *** StraubTW_ has quit IRC | 16:44 | |
| *** geaaru has quit IRC | 16:46 | |
| *** trifon has quit IRC | 16:48 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Fix up get_account_info and get_container_info https://review.openstack.org/280978 | 16:52 |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Make info caching work across subrequests https://review.openstack.org/280977 | 16:52 |
| *** [1]eranrom has joined #openstack-swift | 16:53 | |
| *** eranrom has quit IRC | 16:56 | |
| *** [1]eranrom is now known as eranrom | 16:56 | |
| *** esker has quit IRC | 16:56 | |
| *** wasmum has quit IRC | 16:56 | |
| *** openstackgerrit has quit IRC | 17:01 | |
| *** openstackgerrit has joined #openstack-swift | 17:02 | |
| notmyname | clayg: yeah, I'm seeing some big slowdowns in gerrit too | 17:05 |
| *** trifon has joined #openstack-swift | 17:05 | |
| timburke | still? they just restarted it. i thought it was supposed to be better :( | 17:05 |
| notmyname | oh, did they just restart it? might be better now | 17:06 |
| timburke | no, it's still pretty slow... | 17:06 |
| notmyname | yeah | 17:06 |
| notmyname | seems that gerrit needs to be restarted every other day recently | 17:06 |
| notmyname | cschwede: acoles: timburke: I talked to ttx about the rolling upgrade tag. based on the current work being close to landing, we'll at least have some support in the TC for not dropping the tag | 17:07 |
| notmyname | but likely not universal | 17:07 |
| notmyname | and I'll need to give an update on the TC resolution before the end of this week | 17:07 |
| clarkb | it should be happy now | 17:10 |
| clarkb | at least I hae been able to review several changes iwthout it going nuts | 17:10 |
| *** StraubTW has joined #openstack-swift | 17:11 | |
| *** wasmum has joined #openstack-swift | 17:12 | |
| *** StraubTW has quit IRC | 17:12 | |
| clayg | torgomatic: i've been trying to approve that patch for 10mins! how'd you get gerrit to load! | 17:12 |
| timburke | anybody else see this when running unit tests lately? "proxy ERROR: Exception fetching fragments for '/a/ec-discon/test': ChunkWriteTimeout (60s)" | 17:16 |
| clayg | well it *was* a discon test? | 17:17 |
| clayg | timburke: anyway it reminds me of a really *old* error we *used* to get - but I think we (like literally you and I) managed to get it fixed? | 17:18 |
| timburke | yeah...i just don't like having such messages leak out. makes me think something's up | 17:18 |
| clayg | timburke: I have this vague memory of like having to add a sleep before nosetests exits in order to make sure it was fixed? | 17:18 |
| timburke | clayg: yeah, that's part of why it weirds me out. like, i've seen this before, and it seemed to be masking a problem last time i saw it, so... | 17:18 |
| clayg | do you have like old branch checked out that's maybe not fully up to speed rebased? | 17:19 |
| clayg | timburke: iirc the problem last time ... oh yeah maybe it was a bug - something kota ended up fixing seperatly for the change we had to make in tests ... still - SEEMS LIKE WE'VE DONE THIS ALREADY! :) | 17:19 |
| clayg | timburke: anyway, no i'm not seeing it on master - is it all the time or just some of the time for you? | 17:20 |
| timburke | clayg: nope; very recent branch. only missing the 2.7.0 authors/changelog commit. seems to be every time i run the full suite | 17:21 |
| timburke | maybe my VM's just slower than yours :P | 17:21 |
| timburke | definitely the get test, rather than the put | 17:21 |
| *** asettle has joined #openstack-swift | 17:22 | |
| *** trifon has quit IRC | 17:23 | |
| *** gyee has quit IRC | 17:23 | |
| clayg | timburke: ok, i added my "sleep before you exit" hack to nose.core - we'll see if i can spot it | 17:24 |
| clayg | timburke: you think you know the specific test that's generating the request? like the path is unique or something? you say "the get test" - which one is that? | 17:24 |
| timburke | clayg: test_ec_client_disconnect, not test_ec_client_put_disconnect | 17:25 |
| *** klrmn has joined #openstack-swift | 17:26 | |
| timburke | (i added -get and -put to my container path so i could differentiate) | 17:26 |
| notmyname | clayg: thanks on patch 257502. when patch 296800 lands, I think we're good for a release | 17:26 |
| patchbot | notmyname: https://review.openstack.org/#/c/257502/ - swift - Fix full_listing in internal_client | 17:26 |
| patchbot | notmyname: https://review.openstack.org/#/c/296800/ - swift - Check marker params in SimpleClient full listing r... | 17:26 |
| *** rledisez has quit IRC | 17:26 | |
| *** asettle has quit IRC | 17:27 | |
| clayg | timburke: oic | 17:27 |
| clayg | timburke: maybe the change you're on breaks it :P | 17:28 |
| clayg | why is the *GET* test seening a chunkwritetimeout? Does the test fail? | 17:29 |
| acoles | timburke: i see it | 17:29 |
| clayg | acoles: wat!? how am I missing it then? | 17:29 |
| timburke | clayg: timeout writing to client | 17:30 |
| acoles | did i not file a bug?? i remember raising it here but no one bit | 17:30 |
| clayg | lol | 17:30 |
| clayg | timburke: ok, well that sorta makes sense - i mean the client *did* disconnect | 17:31 |
| acoles | iirc it was a deliberate client disconnect that leaves thread writing to chunk to the queue in proxy waiting to write...until the tiemout pops | 17:31 |
| timburke | yeah. not *entirely* sure it's a problem. it just felt suspicious | 17:31 |
| clayg | but i just see -> proxy INFO: 127.0.0.1 127.0.0.1 24/Mar/2016/17/26/39 GET /v1/a/ec-discon/test HTTP/1.0 499 - - t - 8192 - tx3410690f3583472b8682c-0056f4234f - 0.0335 - - 1458840399.807980061 1458840399.841509104 3 | 17:31 |
| clayg | proxy WARNING: Client disconnected on read (txn: tx3410690f3583472b8682c-0056f4234f) (client_ip: 127.0.0.1) | 17:31 |
| acoles | timburke: that was my conclusion, it was noise rather than a bug | 17:32 |
| clayg | eventlet==0.18.1 | 17:32 |
| clayg | ??? | 17:32 |
| acoles | still annoying | 17:32 |
| clayg | timburke: acoles: you see the message on *stderr* on a normal `.unittests` run? like mixed in with the ..... | 17:33 |
| clayg | i'm on "5d00ce9 2.7.0 authors and changelog updates | 17:33 |
| timburke | clayg: yeah, immediately after the "OK" line at the end. at least, i assume it's stderr | 17:33 |
| clayg | timburke: ok well i'm definately *not* seeing that - so what's your swift sha and eventlet version? | 17:34 |
| clayg | i can't think of anything else relevant that might be different between my config and your | 17:35 |
| *** rcernin has quit IRC | 17:36 | |
| acoles | i'm just running tests to confirm | 17:37 |
| acoles | hehe, my mention was right at top of my scrollback... | 17:37 |
| acoles | "hmmm, the yield in https://github.com/openstack/swift/blob/902cb8f8d74feabc4f402a687b311875935d5221/swift/proxy/controllers/base.py#L825-L825 will timeout after 60s, way after the test has completed. not sure how to prevent/silence that" | 17:37 |
| *** wasmum has quit IRC | 17:38 | |
| acoles | clayg: yeah i see it with tox -e py27, eventlet=0.18.4 | 17:38 |
| acoles | commit 4be370 | 17:39 |
| timburke | i'm exactly where acoles is; same sha, same eventlet | 17:40 |
| clayg | yay! "proxy ERROR: Exception fetching fragments for '/a/ec-discon/test': ChunkWriteTimeout (60s)" | 17:40 |
| clayg | yeah i didn't see it with eventlet 0.18.1 | 17:40 |
| *** wasmum has joined #openstack-swift | 17:40 | |
| *** pcaruana has quit IRC | 17:40 | |
| acoles | yay! i hated the thought of clayg missing out | 17:41 |
| clayg | let me take out my sleep hack - i'm guessing eventlet added some new hawtness to pop timeouts on running coro's during shutdown or something cute - and by that time who's ya boy nose has already stopped his capture stderr jazz | 17:41 |
| clayg | hrm... nope nothing that fancy - i acctually have to sleep before letting proxy exit to see it | 17:42 |
| *** joeljwright has quit IRC | 17:43 | |
| clayg | well at least I think it's innocuous | 17:44 |
| *** trifon has joined #openstack-swift | 17:46 | |
| *** wasmum has quit IRC | 17:51 | |
| timburke | is there any reason not to pass on ChunkWriteTimeouts around https://github.com/openstack/swift/blob/4be370/swift/proxy/controllers/obj.py#L1440, like we do for GreenletExits? | 17:52 |
| clayg | timburke: acoles: I'm looking at https://github.com/openstack/swift/blob/5d00ce9e3a1f5e32ae91c78b6bdfd953658ab984/swift/proxy/controllers/obj.py#L1445 | 17:52 |
| *** wasmum has joined #openstack-swift | 17:53 | |
| timburke | yeah, that guy! | 17:53 |
| *** mmcardle1 has quit IRC | 17:54 | |
| *** ChubYann has joined #openstack-swift | 17:54 | |
| clayg | where is the WriteTimeout being raised from exactly? | 17:54 |
| notmyname | cschwede: oh, I just saw you propose a patch update. you're working on it? | 17:55 |
| clayg | it's kind of annoying that the blanket exception handler is trying to log a traceback for us and our logger is being helpful and coverting it to a oneliner :\ | 17:55 |
| cschwede | notmyname: yes | 17:55 |
| notmyname | ok | 17:55 |
| cschwede | nearly done, updating my patch chain | 17:56 |
| cschwede | notmyname: or did i miss something, are you working on it too? | 17:56 |
| notmyname | cschwede: no, it's better if you do it. I had started looking, but I've been hunting for the right repos to download to patch | 17:56 |
| cschwede | notmyname: sorry, should have pinged you. my day is a ittle bit crazy, my mind feels like there is an error in the floating point unit now :P | 17:58 |
| notmyname | no worries :-) | 17:58 |
| timburke | clayg: roundabouts https://github.com/openstack/swift/blob/902cb8f8d74feabc4f402a687b311875935d5221/swift/proxy/controllers/base.py#L897-L899 | 17:58 |
| timburke | (or a more up-to-date equivalent) | 17:58 |
| clayg | timburke: this is what I got -> https://gist.github.com/clayg/23e6a477f7370efe3fd1 | 18:01 |
| clayg | timburke: but yeah that doesn't really tell me where it came from - thanks stack switching | 18:01 |
| cschwede | notmyname: do you want the new job experimantal or voting on the gate? | 18:01 |
| cschwede | we could start experimental, and change to voting later on | 18:02 |
| cschwede | that would be my proposal | 18:02 |
| notmyname | cschwede: yes, start with experimental. but very very quickly move to gating | 18:03 |
| *** esker has joined #openstack-swift | 18:04 | |
| *** panda has quit IRC | 18:04 | |
| *** panda has joined #openstack-swift | 18:05 | |
| cschwede | notmyname: ok, done; the last patch is 297005, and the other three patches are linked using the depends-on tags. hopefully that’s good for now | 18:06 |
| notmyname | cschwede: got it. and I added myself to them all so I can follow them | 18:07 |
| cschwede | notmyname: answering your earlier comment, i totally think this can and should be improved to do a full rolling upgrade, so if you talk to the TC, that’s in my mind and i will continue working on this once this has landed | 18:07 |
| clayg | timburke: idk more than just silencing the error it'd be nice if we could close those sockets and shutdown those coro's as soon as the front-end notices the pipe is torndown | 18:07 |
| cschwede | but i won’t be able to work on any follow up patches this week | 18:07 |
| notmyname | cschwede: what do you mean by full rolling upgrade? what did you have in mind? | 18:07 |
| *** resker has joined #openstack-swift | 18:08 | |
| cschwede | notmyname: well, right now it’s just the proxy that is upgraded, ie all other services are running on liberty, and proxy on mitaka (or head). but i can think to add more steps inbetween, so another step for account-servers, container-servers, object-servers | 18:09 |
| notmyname | cschwede: yeah, that sounds good | 18:09 |
| cschwede | notmyname: that are just ideas for now, i will have a closer look how other projects handle this. and of course i’m open to all suggestions from the TC, comments welcome! | 18:09 |
| notmyname | cschwede: thank you for working on this :-) | 18:09 |
| cschwede | you’re welcome :) | 18:10 |
| *** esker has quit IRC | 18:11 | |
| clayg | notmyname: so the works on http://governance.openstack.org/reference/tags/assert_supports-rolling-upgrade.html seem pretty clear that this tag means it's tested in the gate - i'm not sure on the timeline tho - did they grandfather the tag with the assumption we'd setup a job during this release cycle - or did the merge "yeah this is a gated assert tag now, you're out" in the past couple of weeks and now we're fire drilled? | 18:13 |
| timburke | clayg: notmyname claimed support in patch 246558 | 18:14 |
| patchbot | timburke: https://review.openstack.org/#/c/246558/ - governance - added assert tags to swift (MERGED) | 18:14 |
| *** silor has quit IRC | 18:14 | |
| *** wasmum has quit IRC | 18:15 | |
| timburke | and i seem to recall asking about whether we *actually* tested it in the gate when he did that... | 18:15 |
| *** silor has joined #openstack-swift | 18:16 | |
| clayg | well I7ba9cdbe1b6ba026da374dd66e30611133538e91 sure makes it seem like dems' the rules and we sorta cheated - oops! | 18:19 |
| clayg | timburke: acoles: I'm going down this line of reasoning -> https://gist.github.com/clayg/29eb57bb93142dbfe64b | 18:23 |
| *** wasmum has joined #openstack-swift | 18:23 | |
| clayg | once we have the backend sockets shutdown things only go off the rails because the backends are blocked in wrong place - so if we drain the queues after we shutdown the sockets thing resolve fairly quickly? | 18:23 |
| notmyname | /headdesk | 18:24 |
| openstackgerrit | Eran Rom proposed openstack/swift: Add process level concurrency to container sync https://review.openstack.org/210099 | 18:25 |
| openstackgerrit | Eran Rom proposed openstack/swift: Add thread level concurrency to container sync https://review.openstack.org/225338 | 18:25 |
| clayg | timburke: i also added a diff on the test that might make it easier for see the error when running the test by itself | 18:25 |
| acoles | cschwede: thanks for all your work on the upgrade test! | 18:26 |
| clayg | timburke: or it might not be helping at all - i'm not sure I understand wft is going on | 18:26 |
| acoles | cschwede: i have been very frustrated by setting up grenade today - partly my usual problem of proxy settings causing stuff to fail:/ i feel like i am close. | 18:27 |
| acoles | clayg: makes sense when you put it in words like that. but also i'm not sure i grok it yet | 18:29 |
| timburke | clayg: that's how i felt through most of my debugging last time :/ | 18:29 |
| *** daemontool has quit IRC | 18:32 | |
| *** asettle has joined #openstack-swift | 18:43 | |
| *** Guest27982 has quit IRC | 18:46 | |
| *** asettle has quit IRC | 18:48 | |
| timburke | clayg: i think i may have been hasty to rip out the part_iter.close() in https://review.openstack.org/#/c/241778/2/swift/proxy/controllers/base.py -- what do you think about https://gist.github.com/tipabu/9e1b3b79db36dd964e34? | 19:02 |
| patchbot | timburke: patch 241778 - swift - Close EC fragment iterators in the GreenThread tha... (MERGED) | 19:02 |
| timburke | still need to try functests with EC-by-default, though | 19:03 |
| *** daemontool has joined #openstack-swift | 19:07 | |
| notmyname | acoles: timburke: cschwede: current summary (since you may have seen the reviews): turns out the -partial mechanism isn't supposed to be used any more. but there's a different thing. multinode | 19:11 |
| notmyname | https://review.openstack.org/#/c/297311/ sets up the multinode job in a way that would allow for different versions to be installed | 19:11 |
| patchbot | notmyname: patch 297311 - openstack-infra/devstack-gate - Run swift services on subnode | 19:11 |
| notmyname | but it would require that we (1) add the multinode job to our gate (which should be easy) and (2) figure out how to get the deployed rings to use both services, which nobody has any idea on right now | 19:12 |
| notmyname | I'm talking it over with sdague in -infra | 19:14 |
| acoles | notmyname: have you managed to figure out yet - are the multi "nodes" actually mulitple vms? | 19:14 |
| notmyname | I'm led to believe that, but I'll check | 19:14 |
| clarkb | they are | 19:15 |
| acoles | i noticed suggestion 2 on that review "Do the more complicated bits as a swift devstack plugin." which got me wondering again about running different swift service/versions in differetn venvs | 19:15 |
| acoles | clarkb: thanks | 19:15 |
| *** _JZ_ has joined #openstack-swift | 19:15 | |
| notmyname | clarkb: thanks for arguing for the -partial method | 19:16 |
| clarkb | ya well... | 19:16 |
| notmyname | https://etherpad.openstack.org/p/swift-rolling-upgrade-multinode-testing <-- is sdague's suggestions | 19:19 |
| *** daemontool has quit IRC | 19:25 | |
| notmyname | clarkb: is there only one (data) drive on the nodes? sdb1? | 19:32 |
| notmyname | ah, nm. I see now | 19:33 |
| clarkb | I believe that devstack makes a single xfs device for you | 19:33 |
| clarkb | so ya | 19:33 |
| notmyname | that's one loopback device that is created in that script | 19:33 |
| clarkb | yup | 19:33 |
| clarkb | and formated xfs because thats what swift wants iirc | 19:33 |
| notmyname | yeah. really just needs xattrs support, but xfs is best | 19:34 |
| notmyname | clarkb: how much space is available on the VM? | 19:35 |
| clarkb | not a ton, its 80GB / with about 15 used by system, 8GB swap file/device, 7GB of git repos etc | 19:35 |
| notmyname | not as big of a deal as I thought because it's only setting up 1 replica. otherwise we'd need more disks | 19:37 |
| *** wasmum has quit IRC | 19:39 | |
| openstackgerrit | Peter Lisák proposed openstack/swift: Change schedule priority of daemon/server in config https://review.openstack.org/238799 | 19:40 |
| *** asettle has joined #openstack-swift | 19:40 | |
| *** asettle has quit IRC | 19:45 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: Refactor server side copy as middleware https://review.openstack.org/156923 | 19:57 |
| openstackgerrit | Thiago da Silva proposed openstack/swift: decouple versioned writes from COPY https://review.openstack.org/260179 | 19:57 |
| cschwede | notmyname: ok, so i expected something like this… anyways, i can continue working on this, following up with Seans proposals, but not before Monday. Let me know if you or someone else continues with, otherwise I continue next week | 19:59 |
| notmyname | cschwede: thanks | 20:00 |
| notmyname | timburke's also looking at stuff | 20:00 |
| *** cdelatte has quit IRC | 20:02 | |
| *** trifon has quit IRC | 20:03 | |
| *** gyee has joined #openstack-swift | 20:06 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Clean up some unnecessary variables https://review.openstack.org/296620 | 20:07 |
| clayg | timburke: Mar 24 20:10:15 saio proxy-server: STDERR: ValueError: generator already executing (txn: tx62b6255e778b49459dcbc-0056f449a7) (client_ip: 127.0.0.1) | 20:10 |
| clayg | sigh | 20:10 |
| clayg | timburke: I do think we have something wrong on ec_get_disconnect - your intuition was right about that message showing up | 20:11 |
| clayg | ... i think | 20:11 |
| clayg | timburke: at least I'm seeing way more ESTABLISHED connections than I would like after running this -> https://gist.github.com/clayg/e2501e6615d0a1eae4e8 | 20:12 |
| *** zhiyan has quit IRC | 20:19 | |
| openstackgerrit | Merged openstack/swift: Fix full_listing in internal_client https://review.openstack.org/257502 | 20:19 |
| *** mmcardle has joined #openstack-swift | 20:19 | |
| openstackgerrit | Merged openstack/swift: Check marker params in SimpleClient full listing requests https://review.openstack.org/296800 | 20:20 |
| notmyname | cool. I think those are the last things for this release | 20:20 |
| *** zhiyan has joined #openstack-swift | 20:20 | |
| notmyname | that means the SHA for swift 2.7.0 is e0bac5e9e874f0dbdda3ea94f34b1922fca9ba35 | 20:21 |
| timburke | clayg: right..generator-already-executing guy...hmmm... wrong finally, maybe? try latest revision at https://gist.github.com/tipabu/9e1b3b79db36dd964e34 | 20:21 |
| acoles | notmyname: i'm not sure what is required for the upgrade test, but i believe i have a proxy running HEAD and a/c/o running liberty on my saio | 20:22 |
| notmyname | acoles: that's really cool | 20:22 |
| clayg | timburke: i thought that's what I was running? | 20:22 |
| notmyname | acoles: did you see the etherpad I linked earlier? https://etherpad.openstack.org/p/swift-rolling-upgrade-multinode-testing | 20:23 |
| acoles | is there anywhere we log the version? i need reassurance :) | 20:23 |
| timburke | clayg: i updated it. like a jerk. | 20:23 |
| acoles | notmyname: just looked | 20:23 |
| clayg | timburke: oh nice | 20:23 |
| notmyname | acoles: it's in /info. also in code via 'import swift; swift.__version__' | 20:23 |
| notmyname | acoles: timburke: clayg: good with e0bac5e9e874f0dbdda3ea94f34b1922fca9ba35 as the release SHA? (current HEAD of master) | 20:24 |
| acoles | notmyname: well let me just run it by my upgrade test... ;) | 20:25 |
| notmyname | :-) | 20:25 |
| acoles | notmyname: that's Merge "Check marker params in SimpleClient full listing requests", looks good to me | 20:25 |
| clayg | timburke: still generator-already-executing, still established sockets from proxy->storage, now put_fragments_in_queue logs [Errno 9] Bad file descriptor (well some of them do, another still logged ChunkWriteTimeout | 20:26 |
| clayg | notmyname: if it's good by acoles it's good by me | 20:26 |
| clayg | notmyname: do we get to keep making swift better or is this like our last chance? | 20:27 |
| notmyname | we will have next week to land backports, if necessary | 20:27 |
| *** sheel has quit IRC | 20:27 | |
| clayg | ok, so we can keep going, phew | 20:27 |
| clayg | i need more coffee | 20:27 |
| *** jmccarthy1 has joined #openstack-swift | 20:30 | |
| clayg | timburke: i think it's weird for a generator to throw already running when you call close - it's like no joke - that's why i'm closing your %^&hole | 20:30 |
| notmyname | swift mitaka release request https://review.openstack.org/#/c/297352/ | 20:32 |
| patchbot | notmyname: patch 297352 - releases - Swift 2.7.0 release | 20:32 |
| acoles | notmyname: so /info tells me *proxy* is at 2.6.1dev260 while in a/c/o venv swift.__version__ is '2.5.1.dev2', plus when i started a/c/o they barfed because i have liberasurecode_rs_vand config'd, so i had to rollback the config to jerasure, so i am gaining confidence that i am running two versions | 20:34 |
| acoles | just running func tests | 20:34 |
| acoles | i think the sure way will be for me to create a branch that spits out errors and run that in a/c/o venv | 20:35 |
| notmyname | acoles: I think it seems like the way to go is with the multinode setup in devstack. do you think the multiple venvs offer anything that the multinode setup doesn't? | 20:36 |
| acoles | notmyname: i can't argue against it. i have spent today battling to setup grenade/devstack without success, it took me 5 mins to spin up two venvs, so i am wary of the learning curve wrt devstack | 20:38 |
| acoles | but if someone knows how to get it all to work then great! | 20:38 |
| acoles | of course my way would still require some devstack hackery | 20:39 |
| acoles | notmyname: i was also curious to see how the tests do fare in a partial upgrade scenario - we'll have to choose a reasonable set of tests to run. i see some functest failure (which may well be expected) | 20:43 |
| notmyname | yeah, I think it's a good thing to have the tests, and something we'll definitely be able to use to improve | 20:44 |
| * notmyname steps away for a bit | 20:44 | |
| acoles | notmyname: i'm out mon/tues, i may only check in briefly tomorrow, but looks like the release is good to go :) | 20:46 |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** openstackgerrit has joined #openstack-swift | 20:49 | |
| *** acoles is now known as acoles_ | 20:51 | |
| *** acoles_ is now known as acoles | 20:51 | |
| *** acoles is now known as acoles_ | 20:55 | |
| *** silor has quit IRC | 21:07 | |
| *** sileht has quit IRC | 21:07 | |
| *** sileht has joined #openstack-swift | 21:08 | |
| notmyname | acoles_: thanks | 21:13 |
| *** panda has quit IRC | 21:20 | |
| *** trifon has joined #openstack-swift | 21:29 | |
| *** asettle has joined #openstack-swift | 21:33 | |
| *** ntata has quit IRC | 21:33 | |
| *** zacksh has quit IRC | 21:33 | |
| *** zacksh has joined #openstack-swift | 21:35 | |
| *** nadeem has quit IRC | 21:38 | |
| *** nadeem has joined #openstack-swift | 21:38 | |
| *** openstack has joined #openstack-swift | 21:50 | |
| *** elmiko has left #openstack-swift | 21:51 | |
| *** ntata has joined #openstack-swift | 21:54 | |
| *** ametts has quit IRC | 21:57 | |
| *** fbo has joined #openstack-swift | 22:00 | |
| *** kota_ has joined #openstack-swift | 22:00 | |
| *** wolfe.freenode.net sets mode: +v kota_ | 22:00 | |
| *** Zyric has quit IRC | 22:01 | |
| *** Zyric has joined #openstack-swift | 22:03 | |
| *** dmorita has quit IRC | 22:04 | |
| *** dmorita has joined #openstack-swift | 22:05 | |
| *** asettle has quit IRC | 22:12 | |
| openstackgerrit | Merged openstack/swift: Handle tempurl Content-Disposition header missing from HEAD https://review.openstack.org/296936 | 22:19 |
| *** Zyric has quit IRC | 22:23 | |
| *** NM has joined #openstack-swift | 22:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Add Expires header for successful GETs using tempurls https://review.openstack.org/279737 | 22:38 |
| *** trifon has quit IRC | 22:39 | |
| *** NM has quit IRC | 22:43 | |
| *** nadeem has quit IRC | 22:43 | |
| *** CaioBrentano has quit IRC | 22:44 | |
| *** jmccarthy1 has quit IRC | 22:44 | |
| *** jmb__ has quit IRC | 22:47 | |
| *** chlong has joined #openstack-swift | 22:55 | |
| *** km has joined #openstack-swift | 22:58 | |
| *** Zyric has joined #openstack-swift | 22:59 | |
| *** _JZ_ has quit IRC | 23:06 | |
| *** mmcardle has quit IRC | 23:12 | |
| *** dmorita has quit IRC | 23:20 | |
| *** dmorita has joined #openstack-swift | 23:20 | |
| *** dmorita has quit IRC | 23:21 | |
| *** dmorita has joined #openstack-swift | 23:21 | |
| *** jmb__ has joined #openstack-swift | 23:22 | |
| *** lyrrad_ has joined #openstack-swift | 23:23 | |
| *** lyrrad has quit IRC | 23:23 | |
| *** lyrrad_ is now known as lyrrad | 23:23 | |
| *** rickyrem has joined #openstack-swift | 23:25 | |
| *** jmb__ has quit IRC | 23:27 | |
| *** kei_yama has joined #openstack-swift | 23:28 | |
| clayg | timburke: so i was wrong - you can close the part_iter up there if you want - but I still need to add the sleep() call - it just switches from needing it for a slow client to needing it for a fast client | 23:32 |
| openstackgerrit | Tim Burke proposed openstack/swift: Fix versioned_writes functional test skipping https://review.openstack.org/265017 | 23:33 |
| *** vint_bra has quit IRC | 23:37 | |
| *** mingdang1 has joined #openstack-swift | 23:38 | |
| *** mingdang1 has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
