| notmyname | timburke: lol. I didn't actually *run* the patch. but it looks ok and the idea seems sound. and the +1 seems as strong as my previous "hmm... that's odd" when you first showed it to me | 00:02 |
|---|---|---|
| zaitcev | I didn't land it because I'm actually no sure what is more likely to break. If you monkey-patch something system, it's less likely to change, while internals of eventlet are more fluid. But then monkey-patching has more side effects, if anything goes wrong it's harder to realize it's involved. So it's a bit of a Buridan's Ass dilemma. | 00:02 |
| notmyname | that was previous versions. current one doesn't rely on eventlet monkeypatching nor stdlib monkeypatching | 00:04 |
| notmyname | also, TIL https://en.wikipedia.org/wiki/Buridan%27s_ass | 00:04 |
| notmyname | zaitcev: I like the current version of the patch because it also helps with a subsequent timburke patch that helps us parse our headers without having to either monkeypatch stdlib or wait for py38 | 00:10 |
| notmyname | timburke: which one was that? | 00:10 |
| *** gyee has quit IRC | 00:16 | |
| *** itlinux has joined #openstack-swift | 01:24 | |
| *** mvkr has quit IRC | 01:36 | |
| *** mvkr has joined #openstack-swift | 01:38 | |
| *** mvkr has quit IRC | 01:49 | |
| *** itlinux has quit IRC | 01:51 | |
| *** mvkr has joined #openstack-swift | 01:52 | |
| *** openstackstatus has quit IRC | 02:22 | |
| *** openstack has joined #openstack-swift | 02:24 | |
| *** ChanServ sets mode: +o openstack | 02:24 | |
| openstackgerrit | Michele Valsecchi proposed openstack/swift master: Fix a typo https://review.openstack.org/642644 | 04:57 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: Change grpcio version in requirements.txt to >=1.11.0 https://review.openstack.org/639365 | 04:59 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: test which gates run in the gerrit https://review.openstack.org/642645 | 05:09 |
| *** tristanC has quit IRC | 06:02 | |
| *** tristanC2 has joined #openstack-swift | 06:04 | |
| *** mahatic has quit IRC | 06:07 | |
| *** DHE has quit IRC | 06:07 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 06:11 |
| *** dr_gogeta86 has quit IRC | 06:16 | |
| *** DHE has joined #openstack-swift | 06:17 | |
| *** e0ne has joined #openstack-swift | 06:18 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 06:46 |
| openstackgerrit | Merged openstack/swift master: Do not use self in classmethod https://review.openstack.org/642359 | 06:48 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 06:54 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 07:02 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 07:16 |
| *** pcaruana has joined #openstack-swift | 07:39 | |
| *** hseipp has joined #openstack-swift | 07:42 | |
| *** pcaruana has quit IRC | 07:43 | |
| *** pcaruana has joined #openstack-swift | 07:43 | |
| *** tkajinam has quit IRC | 08:07 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 08:10 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: test to add override-checkout to pull the master https://review.openstack.org/642680 | 08:11 |
| *** e0ne has quit IRC | 08:17 | |
| *** ccamacho has joined #openstack-swift | 08:21 | |
| *** e0ne has joined #openstack-swift | 08:23 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 08:24 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 08:28 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: DNM: trying to set override-checkout https://review.openstack.org/642645 | 08:30 |
| kota_ | sorry, the noisy patch stream. I'm trying the black box testing to understand what happens in the zuul. | 08:31 |
| kota_ | and it looks like finally I got the dsvm gate in the losf feature branch. | 08:32 |
| *** e0ne has quit IRC | 08:39 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: Enable swift-dsvm-functional tests at losf feature branch https://review.openstack.org/642645 | 08:51 |
| kota_ | alecuyer, rledisez, notmyname: ^^^ | 08:52 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Stop overwriting reserved term https://review.openstack.org/639490 | 08:53 |
| *** mikecmpbll has joined #openstack-swift | 09:05 | |
| zigo | Hi. | 09:48 |
| zigo | I'm getting this error in swift-proxy: http://paste.openstack.org/show/747594/ | 09:48 |
| zigo | What can this be about? | 09:48 |
| *** hseipp has quit IRC | 10:01 | |
| *** [diablo] has joined #openstack-swift | 10:07 | |
| *** e0ne has joined #openstack-swift | 10:16 | |
| *** ybunker has joined #openstack-swift | 11:42 | |
| ybunker | hi everyone, still battle with the acct cont issue from yesterday :-( | 11:43 |
| ybunker | the other thing that im worry about is that besides replication, the nodes can't write acct or container to the actual node, for example: http://pasted.co/9856ba8e | 11:58 |
| *** e0ne has quit IRC | 12:36 | |
| *** ybunker has quit IRC | 12:50 | |
| *** ybunker has joined #openstack-swift | 12:53 | |
| *** FlorianFa has quit IRC | 13:03 | |
| ybunker | anyone? | 13:18 |
| *** irclogbot_2 has quit IRC | 13:26 | |
| *** e0ne has joined #openstack-swift | 13:28 | |
| *** irclogbot_2 has joined #openstack-swift | 13:29 | |
| ybunker | zaitcev? | 13:39 |
| *** openstack has joined #openstack-swift | 15:39 | |
| *** ChanServ sets mode: +o openstack | 15:39 | |
| *** e0ne has quit IRC | 15:45 | |
| timburke | zigo, looks like a client disconnect that just happened to be on a chunk boundary? i guess? | 15:57 |
| *** e0ne has joined #openstack-swift | 15:58 | |
| *** guimaluf has joined #openstack-swift | 15:59 | |
| *** ybunker has quit IRC | 16:03 | |
| *** mikecmpbll has quit IRC | 16:03 | |
| *** mikecmpbll has joined #openstack-swift | 16:04 | |
| *** ybunker has joined #openstack-swift | 16:20 | |
| *** gyee has joined #openstack-swift | 16:27 | |
| *** ccamacho has quit IRC | 16:29 | |
| ybunker | hi, anyone can give me a hand with the acct cont replication issue? | 16:46 |
| ybunker | at this point i think that any acct container is being replicated at all on the cluster :-( | 16:53 |
| *** e0ne has quit IRC | 16:55 | |
| ybunker | ?? | 17:15 |
| ybunker | is there any mailing list that i can send the query? i really need to get this problem solved asap :S | 17:22 |
| tdasilva | ybunker: hi, so where are things at? | 17:23 |
| tdasilva | from the chat with zaitcev yesterday it sounded like he was suggesting to merge the two separate configs into one? to simplify things? | 17:24 |
| ybunker | tdasilva: objects are replicating fine.. but accounts and containers not, i already check many things but with no luck at all, also for some reason, besides replication, i think that is not writing container or account info inside the local /srv/node directories | 17:25 |
| tdasilva | ybunker: right, so did you make any changes to your configs yesterday with the suggestion from zaitcev ? | 17:28 |
| *** hseipp has quit IRC | 17:28 | |
| ybunker | tdasilva: yes, and it doesn work :(, let me send you the config files | 17:28 |
| zaitcev | tdasilva: I only meant it long-term. The short-term suggestion was to get rid of extra replicators that did no work anyway, only prompted error messages in logs. | 17:31 |
| tdasilva | zaitcev: yeah, i think those are good suggestions! simplify first to get a good handle on it running! i'd say have just one network for both types of traffic and just one config file | 17:32 |
| tdasilva | per service per node if possible | 17:32 |
| tdasilva | but that will require both changes to config files and to ring | 17:32 |
| ybunker | http://pasted.co/b36127d7 | 17:33 |
| ybunker | tdasilva: also i think that replication is also not working on existing nodes | 17:36 |
| *** mikecmpbll has quit IRC | 17:37 | |
| ybunker | tdasilva: if you send me a brief of the changes that i need to made to simple use one service per node and the changes that i need to do on the ring i can try that | 17:40 |
| tdasilva | ybunker: honestly i'd start by simplifying your configuration, right now it seems you have one configuration file for each disk. I would change to have just one configuration for a given node/service | 17:43 |
| ybunker | tdasilva: ok, lets go in that direction, for that i guess that i need to change also the rsyncd.conf file right? | 17:45 |
| *** renich has joined #openstack-swift | 17:45 | |
| renich | Good day, swifters! o/ | 17:45 |
| tdasilva | so i'd back up your /etc/swift dir in your 10.1.1.11 node. then you should only have one config for the account server: account-server.conf | 17:46 |
| renich | I am wondering, I have an openstack-swift installation (keystone and swift) and it's on the latest version (Liberty). I was asked to do some bencharking but ssbench doesn't work anymore since it doesn't support v3 auth. How can I do this? | 17:46 |
| tdasilva | delete the files under /etc/swift/account-server/* | 17:46 |
| tdasilva | renich: liberty is not latest version | 17:47 |
| ybunker | tdasilva: http://pasted.co/f7533638 | 17:47 |
| renich | tdasilva: oh, hehee... well, thank you for the reality shot ;D | 17:47 |
| renich | tdasilva: oh, I meant Rocky | 17:48 |
| renich | Sorry... ~_~ | 17:48 |
| ybunker | tdasilva: bkp -> ok, delete files unders account-server -> ok | 17:49 |
| tdasilva | ybunker: now you need to adjust your account-server.conf to bind the correct ip:port and then change the ring for that node so that all disks have the same ip:port combo | 17:53 |
| tdasilva | for both data and replication ip:port | 17:53 |
| ybunker | tdasilva: also do i need to change the rsyncd.conf file? | 17:54 |
| tdasilva | renich: so taking a quick look at the ssbench here: https://github.com/swiftstack/ssbench/blob/master/ssbench/swift_client.py#L231 | 17:54 |
| renich | tdasilva: yep | 17:54 |
| tdasilva | renich: and checking the swift client here: https://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L564,L648 | 17:54 |
| renich | tdasilva: ok | 17:55 |
| tdasilva | it doesn't seem like it would be super difficult to patch up a v3 support, but i don't have the time to do that right now | 17:55 |
| tdasilva | renich: would you be able to send a PR to ssbench? | 17:55 |
| tdasilva | ybunker: pretty sure you will, but let's take one thing at a time, let's first the services running and rings updated | 17:56 |
| tdasilva | then rsync | 17:56 |
| renich | tdasilva: my python skills don't add up to that much... ! | 17:56 |
| renich | tdasilva: but I understand. Any other options other than ssbench that support v3? Or, how can I enable v2 auth in keystone just for that test? | 17:57 |
| ybunker | tdasilva: ok, for the ip:port relationship i need to specify rsync_module = 10.2.1.19::account{port} for each of the ports? | 17:57 |
| ybunker | tdasilva: or using vm_test_mode = yes will adapt with the rsync config ? | 17:59 |
| tdasilva | ybunker: i don't think you need vm_test_mode, that's really for a SAIO type of setup IIRC, for your rsync config i think you could have something like here: https://github.com/openstack/swift/blob/master/etc/rsyncd.conf-sample#L6,L10 | 18:00 |
| tdasilva | renich: if you are running liberty swift+keystone, doesn't that support v2 already? you should just be able to use that....OTOH i *think* for latest keystone, they have removed support for v3 | 18:01 |
| notmyname | tdasilva: he's using rocky | 18:01 |
| renich | tdasilva: I was wrong. I am running rocky. The latest one. | 18:02 |
| *** e0ne has joined #openstack-swift | 18:02 | |
| ybunker | tdasilva: ok so account-server.conf file is like this: http://pasted.co/b886d80a | 18:02 |
| ybunker | tdasilva: and rsync conf: http://pasted.co/16351cf9 | 18:05 |
| tdasilva | ybunker: i changed your paste to openstack paste just so i can refer to line numbers: http://paste.openstack.org/show/747633/ | 18:06 |
| *** e0ne has quit IRC | 18:06 | |
| ybunker | tdasilva: oh ok, great | 18:06 |
| notmyname | renich: looking... | 18:07 |
| tdasilva | ybunker: for now commment out line 5 and let's add healthcheck to the pipeline in line 26 | 18:07 |
| tdasilva | ybunker: then also add this: https://github.com/openstack/swift/blob/master/etc/account-server.conf-sample#L109,L110 | 18:08 |
| tdasilva | after line 32 | 18:09 |
| tdasilva | ybunker: have you changed your rings yet? | 18:09 |
| tdasilva | account rings only for now | 18:09 |
| ybunker | tdasilva: done | 18:09 |
| ybunker | tdasilva: on the rsync do i leave it like the last one? or just change the account on the old one? | 18:10 |
| tdasilva | ybunker: oh sorry, for the rsync, i meant to change only the account first | 18:11 |
| tdasilva | you said object servers is working fine, so i'd prefer not to touch it | 18:11 |
| ybunker | tdasilva: ok, no problem | 18:11 |
| tdasilva | ybunker: can you post your latest account-server, account-ring output and rsync.conf file again? also are you making changes to just one node for now? | 18:15 |
| renich | notmyname: thanks! | 18:15 |
| ybunker | tdasilva: yes, just one node | 18:15 |
| notmyname | renich: so the good news is that swiftclient supports keystone v3 just fine. the bad news is that at some point in the past ssbench inlined the client code | 18:15 |
| notmyname | renich: I don't know if there were modifications or if it were just done to avoid dependencies. but if there were modifications, then it might be tricky to update the client. really just depends on what they were, if any | 18:16 |
| *** hoonetorg has joined #openstack-swift | 18:17 | |
| ybunker | tdasilva: account-server.conf (http://paste.openstack.org/show/747634/) | 18:17 |
| notmyname | renich: for a basic test, maybe changing the `import ssbench.swift_client as client` lines to use swiftclient would be a good start for a test | 18:18 |
| notmyname | ie change it to `from swiftclient import client` | 18:18 |
| ybunker | tdasilva: account-ring: http://paste.openstack.org/show/747635/ | 18:19 |
| notmyname | renich: there are 3 instances of that line in the ssbench code. should be pretty easy to change those and see what breaks | 18:19 |
| renich | notmyname: OK, I'll try that | 18:19 |
| ybunker | tdasilva: and finally the rsync: http://paste.openstack.org/show/747636/ | 18:20 |
| *** e0ne_ has joined #openstack-swift | 18:21 | |
| tdasilva | ybunker: let's try making changes to rings...do you have a back up of your builder files? | 18:23 |
| tdasilva | if not, please make one | 18:23 |
| ybunker | tdasilva: yesç | 18:23 |
| tdasilva | ybunker: which node did you make the account-server config changes? | 18:23 |
| ybunker | 10.1.1.19 - 10.2.1.19 | 18:24 |
| *** zaitcev_ has joined #openstack-swift | 18:24 | |
| *** ChanServ sets mode: +v zaitcev_ | 18:24 | |
| tdasilva | ybunker: ok, so you need to change the ip port for both data and replication to just use 10.1.1.10:4000 | 18:25 |
| tdasilva | sorry | 18:25 |
| tdasilva | 10.1.1.19:4000 | 18:25 |
| tdasilva | you will stop use 10.2.1.19 for replication for now, ok | 18:26 |
| renich | notmyname: https://paste.fedoraproject.org/paste/PAlzdapq3nnt9uUmmXKwXQ | 18:26 |
| ybunker | tdasilva: ok, for the 3 disks right? /srv/node/1, /srv/node/2 and /srv/node/3 | 18:26 |
| *** gmann is now known as gmann_afk | 18:26 | |
| tdasilva | ybunker: correct | 18:26 |
| tdasilva | ybunker: do you know the swift-ring-builder command to change the ip:port ? the --help flag can be useful.... | 18:27 |
| ybunker | tdasilva: if you got the command out there will appreciated :-) | 18:27 |
| *** zaitcev has quit IRC | 18:27 | |
| renich | notmyname: so, now, to change all instances of client.DEFAULT_NETWORK_TIMEOUT to 20.0? | 18:28 |
| ybunker | tdasilva: do i need to remove it and then add it again? or is there any parameter and action to just change it? | 18:30 |
| tdasilva | ybunker: nope, just change, try this: swift-ring-builder account.builder set_info --ip 10.1.1.19 --change-port 4000 | 18:30 |
| tdasilva | then run `swift-ring-builder account.builder` to see if it worked | 18:30 |
| notmyname | renich: I think there's a better way. it seems the inlined client set up a `network_timeout` but the upstream client now has a `timeout` | 18:31 |
| renich | OK | 18:31 |
| renich | I was about to edit client.py | 18:32 |
| tdasilva | ybunker: if it does, you can then run: `swift-ring-builder account.builder set_info --ip 10.1.1.19 --change-replication-ip 10.1.1.19 --change-replication-port 4000` to update replication ip:port | 18:32 |
| notmyname | renich: so the change needs to be changing line 230 in worker.py to pass that in, but also update the references to where it's set | 18:32 |
| renich | <gulp>... OK. Will try that. | 18:32 |
| renich | notmyname: so change: network_timeout=network_timeout) to network_timeout=timeout? | 18:33 |
| renich | there is a connect_timeout as well. I saw that constant in ssbench's version as well | 18:34 |
| ybunker | tdasilva: done.. http://paste.openstack.org/show/747638/ | 18:34 |
| *** mikecmpbll has joined #openstack-swift | 18:36 | |
| tdasilva | ybunker: just for sanity your drives are mounted as `/srv/node/[1..3]`, correct? | 18:36 |
| ybunker | tdasilva: yes | 18:36 |
| ybunker | tdasilva: for acct cont, and then `/srv/node/[4..12]` for objects | 18:37 |
| renich | notmyname: not proficient in python so I need a bit more help, if you are willing to... :S | 18:37 |
| tdasilva | ybunker: ok, good, just for clarify, it's probably a good idea to add these two lines to your account-server.conf: https://github.com/openstack/swift/blob/master/etc/account-server.conf-sample#L8,L9 | 18:37 |
| notmyname | renich: no, change it to `timeout=network_timeout` | 18:38 |
| renich | OK | 18:38 |
| tdasilva | ybunker: you ring looks good, now i'd run `swift-ring-builder account.builder write_ring` and distribute the account rings to all your nodes | 18:38 |
| ybunker | tdasilva: done | 18:38 |
| tdasilva | then i'd try to run a healthcheck on the ip port from all your nodes | 18:39 |
| tdasilva | you can also check rsync to make sure it's working ok | 18:39 |
| tdasilva | if it all works, just repeat the process for all nodes | 18:39 |
| *** pcaruana has quit IRC | 18:39 | |
| renich | notmyname: looks like this: http://paste.openstack.org/show/747640/ | 18:40 |
| *** e0ne_ has quit IRC | 18:41 | |
| renich | And, when ran, it gives this: http://paste.openstack.org/show/747641/ | 18:41 |
| notmyname | renich: that's good, but above there where network_timeout is set, you'll need to change that too. | 18:42 |
| notmyname | right | 18:42 |
| notmyname | yeah, the upstream version doesn't set a default timout | 18:42 |
| renich | OK, I will change all network_timeout instances to timeout, right? | 18:43 |
| notmyname | so the fastest thing to do is change every instance of `client.THE_TIMEOUT` to the actual default number that's in the ssbench client (either 10 or 20 seconds) | 18:43 |
| renich | it's 20.0 | 18:43 |
| ybunker | tdasilva: with what command can i run the healthcheck? | 18:45 |
| tdasilva | `curl -i http://10.1.1.19:4000/healthcheck` | 18:45 |
| tdasilva | ybunker: please run that from all your swift nodes to make sure they can all talk to this ip:port | 18:46 |
| ybunker | tdasilva: Ok on all | 18:48 |
| renich | notmyname: OK, changed all client.DEFAULT*TIMEOUT to 20.0. Now, I get: swiftclient.exceptions.ClientException: Authorization Failure. Authorization Failed: Not Found (HTTP 404) (Request-ID: req-08639d7a-bd13-49b0-8ae2-41c7296a0247) | 18:48 |
| notmyname | that's good! | 18:48 |
| tdasilva | ybunker: same for rsync? | 18:49 |
| notmyname | at least it's not syntax errors or missing variables | 18:49 |
| renich | notmyname: yeah! Just FYI, `swift --os-auth-url http://localhost:5000/v3 --os-tenant-name redacted --os-username redacted --os-password redacted stat` works | 18:49 |
| notmyname | nice! | 18:49 |
| tdasilva | ybunker: i think something like `rsync rsync://pub@10.1.1.19/` | 18:49 |
| notmyname | renich: so same creds with ssbench are getting a 404 | 18:50 |
| ybunker | tdasilva: rsync up | 18:50 |
| renich | notmyname: yep | 18:50 |
| notmyname | renich: I'd probably look at cluster logs or turn on debug logging in ssbench next. see what's getting sent so I can find why the 404 is happening | 18:51 |
| renich | OK, I'll try that. | 18:51 |
| ybunker | tdasilva: rsync: failed to connect to 10.1.1.19 (10.1.1.19): Connection refused (111) | 18:52 |
| tdasilva | ybunker: mmm..and you restarted rsync right? after changes to the config file | 18:52 |
| ybunker | tdasilva: got it, the address was pointing to the replica, instead of the storage ip | 18:53 |
| notmyname | renich: I need to head to lunch. I won't be on irc for a bit | 18:53 |
| ybunker | tdasilva: http://paste.openstack.org/show/747642/ | 18:53 |
| tdasilva | ybunker: same output when running from all other nodes? | 18:54 |
| ybunker | tdasilva: yes | 18:54 |
| renich | notmyname: man, thanks a ton for the help. | 18:55 |
| tdasilva | ok, how's the logs? do you see any traffic going into that account server? | 18:55 |
| renich | notmyname: I'll keep at it. | 18:55 |
| ybunker | tdasilva: no, no traffic at all :-( | 18:55 |
| ybunker | in fact inside the /srv/node/[1...3] there's no `accounts` directory | 18:56 |
| tdasilva | well, we started from account-server and since nothing is being written to container server, there might be nothing to write to account server | 18:57 |
| tdasilva | so i'd make the same changes to the container server | 18:57 |
| ybunker | tdasilva: do i need the changes on container also, right¿ | 18:57 |
| tdasilva | you will need to use another port | 18:57 |
| ybunker | 5000 | 18:57 |
| renich | So, it seems ssbench is still looking for /v3/tokens: http://paste.openstack.org/show/747643/ | 18:59 |
| ybunker | tdasilva: http://paste.openstack.org/show/747644/ | 18:59 |
| ybunker | its ok? | 18:59 |
| renich | It seems it should be /v3/auth/tokes | 19:00 |
| tdasilva | ybunker: looks ok, just comment out line 3 | 19:01 |
| ybunker | tdasilva: ok | 19:01 |
| ybunker | tdasilva: applying changes on the ring file and replicating config | 19:04 |
| ybunker | tdasilva: ok, well, now im getting on the swift_container_server.log file: container-replicator: ERROR rsync failed with 10 | 19:07 |
| renich | I'll be back in a while. Going to get lunch as well. | 19:08 |
| tdasilva | ybunker: well you changed only one node, is the rsync error for that node? could be for another node | 19:09 |
| ybunker | tdasilva: let me change it on the other new nodes | 19:11 |
| tdasilva | ybunker: make sure that you make the change to the ring and then distribute the copy to all nodes | 19:12 |
| ybunker | tdasilva: ok, and then restart all the services? or no need for that? | 19:13 |
| tdasilva | ybunker: yeah because you changed the config | 19:13 |
| tdasilva | ybunker: just for future reference, when changing just the ring you don't need to restart services | 19:13 |
| ybunker | tdasilva: ok :) | 19:14 |
| ybunker | tdasilva: no accounts replication for the moment.. :( | 19:18 |
| tdasilva | ybunker: just to recap, are you changing all your container server config files/rings/rsync config? | 19:21 |
| ybunker | tdasilva: yes, but just on new nodes, do you want me to change also on the old nodes? | 19:21 |
| tdasilva | ybunker: can you post your latest ring? | 19:22 |
| tdasilva | container ring | 19:22 |
| ybunker | tdasilva: container ring http://paste.openstack.org/show/747646/ | 19:24 |
| tdasilva | ybunker: are those weights representative of your disk sizes? | 19:25 |
| ybunker | tdasilva: yes, but for the new nodes (10.1.1.19, 10.1.1.20 and 10.1.1.21) i need to set them to 50% of weight | 19:25 |
| ybunker | because those disks are 400G and the old ones are 200G | 19:26 |
| tdasilva | ybunker: so on nodes like 10.1.1.[11-16] they are 200G ? and 19,20,21 they are 400G? | 19:27 |
| tdasilva | or vice-versa? | 19:27 |
| ybunker | tdasilva: yes 10.1.1.[11-16] are of 200G | 19:27 |
| ybunker | tdasilva: and 19,20,21 400G | 19:27 |
| tdasilva | oh wow, so your weights are really messed up too | 19:28 |
| ybunker | tdasilva: yep | 19:28 |
| tdasilva | ybunker: can you change the weight on 200 on 19,20,21 ? but this time instead of write-ring you will need to rebalance | 19:29 |
| ybunker | tdasilva: ok | 19:29 |
| tdasilva | ybunker: was thinking something like: http://paste.openstack.org/show/747648/ | 19:33 |
| tdasilva | after rebalance, distribute rings to all nodes | 19:34 |
| ybunker | tdasilva: done | 19:38 |
| notmyname | timburke: note the ML message about pike extended maintenance | 19:38 |
| tdasilva | ybunker: can you post output of swift-ring-builder for account and container rings? | 19:38 |
| ybunker | tdasilva: accounts: http://paste.openstack.org/show/747650/ | 19:40 |
| ybunker | tdasilva: containers: http://paste.openstack.org/show/747651/ | 19:41 |
| tdasilva | ybunker: ok, have you done a rebalance and distributed rings to all nodes | 19:42 |
| tdasilva | ybunker: eventually you should adjust weight on 17 and 18 ... | 19:42 |
| ybunker | tdasilva: and then i got another problem... major problem... the objects...take a look in here: http://paste.openstack.org/show/747652/ | 19:43 |
| ybunker | tdasilva: on that | 19:43 |
| ybunker | tdasilva: do i need to change also the account and container on the OLD nodes? | 19:43 |
| timburke | notmyname, looks fairly clean... what do we want to do about https://review.openstack.org/#/c/629402/ ? | 19:44 |
| patchbot | patch 629402 - swift (stable/pike) - object-server can 409 in response to x-if-delete-at - 1 patch set | 19:44 |
| tdasilva | ybunker: yeah the weights on the object nodes look wrong too, but let's take one step at a time... | 19:44 |
| ybunker | tdasilva: nodes 10.1.1.11 to 10.1.1.14 are 4TB (and some nodes are at 98% of used disk), and then 10.1.1.15 to 10.1.1.21 are 2TB | 19:45 |
| ybunker | tdasilva: yep | 19:45 |
| timburke | i *guess* maybe there should be a client release for https://review.openstack.org/#/c/505092/ ? i'm kinda inclined to skip it and tell people to just upgrade to latest... python-swiftclient doesn't change much (for better or worse) and it should *always* work on old swift | 19:45 |
| patchbot | patch 505092 - python-swiftclient (stable/pike) - Add support for versionless endpoints (MERGED) - 1 patch set | 19:45 |
| tdasilva | ybunker: sorry, lost track a bit...let's put object-server to the side for a bit, where are you at with account/container server? | 19:47 |
| ybunker | tdasilva: account/container on new nodes is working! :) thanks!!!! :-), but on old nodes do i also need to change to this new scheme of just one config file for account/container? | 19:48 |
| tdasilva | ybunker: you seeing data go into account/container dirs??? | 19:48 |
| ybunker | tdasilva: yes! :) | 19:49 |
| tdasilva | ah cool | 19:49 |
| notmyname | timburke: I agree about swiftclient. I dont' see any need to release old versions | 19:49 |
| tdasilva | yeah, make same change to old nodes now | 19:49 |
| ybunker | tdasilva: ok | 19:49 |
| tdasilva | ybunker: how many proxy nodes do you have? | 19:49 |
| ybunker | tdasilva: 5 | 19:49 |
| tdasilva | ok, if they are different from any of these nodes, make sure you are copying rings to them too | 19:50 |
| ybunker | tdasilva: the thing is that rsync is using 10.1.1.19 for account container.. but on the object-ring it still using the replication network 10.2.1.19 | 19:53 |
| tdasilva | ybunker: mmm...can you post the configs for the object server? | 20:04 |
| ybunker | tdasilva: sure | 20:04 |
| ybunker | tdasilva: http://paste.openstack.org/show/747655/ | 20:08 |
| ybunker | tdasilva: and object.builder (http://paste.openstack.org/show/747656/) | 20:09 |
| ybunker | tdasilva: i've to go now, can we continue tomorrow? i really appreciate your help with this :-) thanks!! i still cant believe the cluster that the old guy left to me.. he was wise to leave the company hahaha lol | 20:11 |
| tdasilva | ybunker: you are welcome, looks like you are almost ther | 20:12 |
| *** e0ne has joined #openstack-swift | 20:20 | |
| *** ybunker has quit IRC | 20:34 | |
| notmyname | timburke: no, I don't have strong feelings about pavel's old backport to pike. having just done the backports, I didn't particularly enjoy the hoops I had to jump through | 20:45 |
| *** gmann_afk is now known as gmann | 20:54 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Stop using stdlib's putrequest(); it only does ASCII https://review.openstack.org/642893 | 21:19 |
| *** e0ne has quit IRC | 21:26 | |
| openstackgerrit | Merged openstack/swift master: Stop overwriting reserved term https://review.openstack.org/639490 | 21:35 |
| *** rcernin has joined #openstack-swift | 22:01 | |
| mattoliverau | morning, wow alot of scrollback lately. | 22:07 |
| *** hoonetorg has quit IRC | 22:09 | |
| *** hoonetorg has joined #openstack-swift | 22:10 | |
| *** mikecmpb_ has joined #openstack-swift | 22:15 | |
| *** mikecmpbll has quit IRC | 22:16 | |
| *** ianychoi_ is now known as ianychoi | 22:47 | |
| *** tkajinam has joined #openstack-swift | 22:53 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Fix quoting for large objects https://review.openstack.org/571905 | 23:11 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: alternate approach for DLO https://review.openstack.org/642920 | 23:11 |
| timburke | i really *really* like the progress that p 642520 and p 642893 represent. there's something very reassuring about being able to run both unit and func tests for a middleware that's being ported... | 23:19 |
| patchbot | https://review.openstack.org/#/c/642520/ - swift - Get functional/tests.py running under py3 - 5 patch sets | 23:19 |
| patchbot | https://review.openstack.org/#/c/642893/ - swift - py3: Stop using stdlib's putrequest(); it only doe... - 1 patch set | 23:19 |
| *** tristanC2 is now known as tristanC | 23:36 | |
| *** itlinux has joined #openstack-swift | 23:37 | |
| openstackgerrit | Merged openstack/swift master: Fix a typo https://review.openstack.org/642644 | 23:45 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!