| *** jlwhite has quit IRC | 00:29 | |
| *** rderose has joined #osic | 00:40 | |
| *** rmevans has quit IRC | 02:17 | |
| *** rmevans has joined #osic | 02:24 | |
| *** asettle has quit IRC | 02:33 | |
| *** sgundur has left #osic | 03:04 | |
| *** rmevans has quit IRC | 03:17 | |
| *** nadeem has joined #osic | 03:20 | |
| *** onthecly has joined #osic | 04:04 | |
| *** onthecly has quit IRC | 04:08 | |
| *** karthikp has joined #osic | 04:17 | |
| *** rderose has quit IRC | 04:53 | |
| *** asettle has joined #osic | 05:28 | |
| *** karthikp has quit IRC | 06:02 | |
| *** nadeem has quit IRC | 07:02 | |
| *** xek has quit IRC | 07:24 | |
| *** asettle has quit IRC | 08:09 | |
| *** jlwhite has joined #osic | 10:26 | |
| *** ccneill has quit IRC | 10:56 | |
| *** sgundur has joined #osic | 12:16 | |
| *** b3rnard0_away is now known as b3rnard0 | 12:17 | |
| *** inc0 has joined #osic | 12:57 | |
| *** xek has joined #osic | 13:24 | |
| *** onthecly has joined #osic | 13:28 | |
| *** onthecly has joined #osic | 13:29 | |
| *** onthecly has quit IRC | 13:29 | |
| *** onthecly has joined #osic | 13:30 | |
| toan | good morning osicers! | 13:39 |
|---|---|---|
| claco | mornings | 13:41 |
| sarafraj | good morning | 13:44 |
| lbragstad | o/ | 13:45 |
| dasm | good morning! | 13:46 |
| *** rderose has joined #osic | 13:49 | |
| *** ametts has joined #osic | 13:55 | |
| electrocucaracha | morning | 13:57 |
| *** KrishR has joined #osic | 13:58 | |
| *** jthorne has joined #osic | 14:00 | |
| izaakk | Morning! | 14:01 |
| *** woodster_ has joined #osic | 14:03 | |
| anush | Good morning from sunny Arizona | 14:05 |
| dasm | anush: good morning from cloudy San Antonio | 14:06 |
| *** KLevenstein has joined #osic | 14:06 | |
| *** KLevenstein has quit IRC | 14:08 | |
| kencjohnston | o/ | 14:11 |
| *** pushkaru has joined #osic | 14:11 | |
| pumaranikar | good morning | 14:12 |
| *** jlwhite has quit IRC | 14:12 | |
| *** anush_ has joined #osic | 14:13 | |
| *** Mudpuppy has joined #osic | 14:14 | |
| *** spotz_zzz is now known as spotz | 14:20 | |
| *** anush_ has quit IRC | 14:26 | |
| spotz | morning | 14:26 |
| *** muralia_ has joined #osic | 14:28 | |
| *** muralia has quit IRC | 14:30 | |
| *** karthikp has joined #osic | 14:34 | |
| Mudpuppy | O/ | 14:36 |
| *** onthecly has quit IRC | 14:39 | |
| *** homerp_ has joined #osic | 14:40 | |
| *** sfinucan has joined #osic | 14:44 | |
| *** pumarani__ has joined #osic | 14:46 | |
| *** rahulunair has joined #osic | 14:46 | |
| *** pushkaru has quit IRC | 14:48 | |
| sneti | Morning!! | 14:49 |
| *** karthikp has quit IRC | 14:51 | |
| *** karthikp has joined #osic | 14:52 | |
| *** inc0 has quit IRC | 14:59 | |
| *** asingh has joined #osic | 15:08 | |
| *** inc0 has joined #osic | 15:16 | |
| *** homerp_ has quit IRC | 15:18 | |
| *** homerp_ has joined #osic | 15:21 | |
| *** navid_ has joined #osic | 15:30 | |
| *** asingh has quit IRC | 15:34 | |
| navid_ | hi | 15:35 |
| navid_ | how can i reset the password on osic clster | 15:36 |
| jthorne | navid_: are you locked out or you want to change your password? | 15:37 |
| *** ccneill has joined #osic | 15:37 | |
| navid_ | I havent activated the password at the time so the link is expired | 15:37 |
| jthorne | navid_: i can reset it. what's your email address? | 15:37 |
| navid_ | navid.pustchi@intel.com | 15:39 |
| navid_ | jthorne, thanks | 15:39 |
| jthorne | navid_: sending the new link to your email | 15:39 |
| *** navid_ has quit IRC | 15:46 | |
| *** asingh has joined #osic | 15:47 | |
| pdardeau | swift ptl will be at castle tomorrow and wed to help jumpstart osic swift within swift community | 15:58 |
| pdardeau | set up an etherpad for it: https://etherpad.openstack.org/p/SG5RJKiZr9 | 15:58 |
| pdardeau | anyone is welcome to drop by at any time | 15:59 |
| pdardeau | locations TBD | 15:59 |
| *** inc0_ has joined #osic | 16:08 | |
| *** inc0 has quit IRC | 16:11 | |
| *** nadeem has joined #osic | 16:12 | |
| *** ankur-gupta-f has quit IRC | 16:14 | |
| *** david-lyle_ is now known as david-lyle | 16:15 | |
| *** anush_ has joined #osic | 16:20 | |
| *** yarkot_ has joined #osic | 16:23 | |
| *** ankur-gupta-f has joined #osic | 16:24 | |
| *** karthikp has quit IRC | 16:29 | |
| *** karthikp has joined #osic | 16:29 | |
| *** homerp_ has quit IRC | 16:34 | |
| *** nadeem has quit IRC | 16:37 | |
| *** jthorne has quit IRC | 16:41 | |
| *** jthorne has joined #osic | 16:42 | |
| *** jthorne has quit IRC | 16:43 | |
| *** sfinucan has quit IRC | 16:43 | |
| *** jthorne has joined #osic | 16:44 | |
| *** homerp_ has joined #osic | 16:44 | |
| *** inc0_ has quit IRC | 16:44 | |
| *** ankur-gupta-f has left #osic | 16:48 | |
| *** rderose has quit IRC | 17:00 | |
| *** inc0 has joined #osic | 17:04 | |
| *** jlwhite has joined #osic | 17:06 | |
| *** jlwhite has quit IRC | 17:20 | |
| *** asingh has quit IRC | 17:22 | |
| *** asingh has joined #osic | 17:23 | |
| *** anush_ has quit IRC | 17:29 | |
| *** homerp_ has left #osic | 17:36 | |
| *** karthikp has quit IRC | 17:47 | |
| *** karthikp has joined #osic | 17:48 | |
| *** yarkot_ has quit IRC | 17:48 | |
| *** sfinucan has joined #osic | 17:50 | |
| *** homerp_ has joined #osic | 17:51 | |
| *** jlwhite has joined #osic | 18:01 | |
| *** anush_ has joined #osic | 18:02 | |
| *** nadeem has joined #osic | 18:04 | |
| *** nadeem has quit IRC | 18:05 | |
| *** ankur-gu_ has joined #osic | 18:05 | |
| *** jlwhite has quit IRC | 18:05 | |
| *** jlwhite has joined #osic | 18:06 | |
| *** nadeem has joined #osic | 18:06 | |
| *** karthikp has quit IRC | 18:09 | |
| *** karthikp has joined #osic | 18:09 | |
| *** inc0 has quit IRC | 18:11 | |
| *** ccneill has quit IRC | 18:13 | |
| *** ccneill has joined #osic | 18:17 | |
| *** homerp_ has left #osic | 18:41 | |
| gmmaha | hi neutron experts, had a quick Q.. have you ever come across an issue where neutron fails with 'Authorization failed for token' when scheduling multiple VMs | 18:42 |
| gmmaha | and never recovers from that state.. | 18:42 |
| gmmaha | i started a 1000 VM schedule and after a certain limit neutron starts throwing 'Authorization failed for token' and just wont recover from this state | 18:43 |
| *** anush_ has quit IRC | 18:44 | |
| hockeynut | token expired perhaps? | 18:44 |
| gmmaha | hockeynut: if the token expired, shouldnt it get a new one by authenticating? | 18:45 |
| hockeynut | one would think - unless you authenticate once then use that token for a time period longer than the token's lifespan | 18:46 |
| ankur-gu_ | gmmaha: my guess is token authentication only happens upon initial request, so if it times out theres no call to re-authenticate | 18:46 |
| hockeynut | weve had that issue with some barbican tests - token lives for X and we try to use it at X+1 | 18:46 |
| gmmaha | hockeynut: ankur-gu_: ohh wow.. ok | 18:51 |
| gmmaha | didnt know that it wont request for token when it expires | 18:51 |
| ankur-gu_ | but im no expert. So don't take my word for it. Just an assumption | 18:52 |
| *** karthikp has quit IRC | 18:57 | |
| *** karthikp has joined #osic | 18:58 | |
| *** yarkot_ has joined #osic | 19:00 | |
| *** jlwhite has quit IRC | 19:05 | |
| *** ankur-gu_ has quit IRC | 19:18 | |
| *** ccneill has quit IRC | 19:19 | |
| *** ccneill has joined #osic | 19:20 | |
| *** jlwhite has joined #osic | 19:21 | |
| gmmaha | dolphm: lbragstad: is the above claim true? shouldnt services get fresh tokens if the current one in use expires? | 19:35 |
| gmmaha | hockeynut: thanks for the info.. i thought i was deploying it all wrong and was scrubbing my setup | 19:36 |
| hockeynut | np - good luck! | 19:38 |
| dolphm | gmmaha: services should definitely get fresh tokens if the one they have expires | 19:45 |
| dolphm | gmmaha: i wonder if you need to increase the token lifespan in keystone.conf? it's possible it's set to a shorter duration than your long-running operation requires | 19:45 |
| dolphm | gmmaha: i believe the default is just an hour (which is aggressively low to discover these issues - you probably want something longer like 2-72 hours, depending on security concerns, etc) | 19:46 |
| gmmaha | dolphm: thanks.. | 19:46 |
| dolphm | gmmaha: keystone.conf [token] expiration | 19:46 |
| gmmaha | i can increase the timeout of the key.. but have you ever heard of any service (neutron in this case) running into a situation liek this? | 19:47 |
| gmmaha | dolphm: will increase it and see if i get something | 19:47 |
| gmmaha | if the token fails to auth and if say the token hasnt expired yet | 19:48 |
| gmmaha | and i ran the test twice.. both times the no. of successful VMs were very close. So wondering if there is a security failsafe where so many requests within a given period of time, we block that token from requesting anymore | 19:50 |
| *** rmevans has joined #osic | 19:54 | |
| *** anush_ has joined #osic | 19:58 | |
| dolphm | gmmaha: i suspect it's the client's token that is expiring, not neutron's | 19:59 |
| dolphm | gmmaha: if it was a quota issue, you should get a different status code / message | 20:00 |
| dolphm | gmmaha: or if it was a sudden change in authorization, it'd be a 403 (you're authenticated, but not allowed to do that) | 20:00 |
| gmmaha | dolphm: aah ok.. | 20:00 |
| gmmaha | dolphm: the way i create the 1000 VMs was one command.. 'openstack server list --min 1000 --max 1000 ......' | 20:01 |
| gmmaha | and that command is done and it stops | 20:01 |
| dolphm | gmmaha: how long did that operation take before you saw a failure? | 20:01 |
| gmmaha | dolphm: ohh a hour past and i got 484 Vms active | 20:02 |
| gmmaha | After that nothing | 20:02 |
| gmmaha | all theother just failed since | 20:02 |
| *** inc0 has joined #osic | 20:05 | |
| dolphm | gmmaha: token is expired :) | 20:06 |
| dolphm | gmmaha: again, the default token lifespans are aggressively low | 20:07 |
| gmmaha | dolphm: is there a limit on no. of tokens keystone can issue in a given time period? | 20:07 |
| dolphm | gmmaha: 24 hours is a popular value for production deployments | 20:07 |
| gmmaha | I am curious how come the system never recovered from this state | 20:07 |
| dolphm | gmmaha: good question - that's one reason why the current default is low as well | 20:07 |
| gmmaha | i let it sit for anthoer 3 hours while it was slowly erroring out on the remaining 516 VMs | 20:08 |
| dolphm | gmmaha: with UUID tokens, yes, you'll fill up the database. a low expiration means fewer active tokens | 20:08 |
| dasm | dolphm: 24h as default for production? seems very long. i thought it's pretty low value to prevent possible breakouts. | 20:08 |
| dolphm | gmmaha: if you switch to Fernet tokens, there's no limit, and you can increase the default lifespan much higher with no performance penalty | 20:08 |
| dolphm | dasm: breakouts as in compromised tokens? | 20:08 |
| dasm | dolphm: and... you will talk about fernet tokens at Summit (ad :P) | 20:09 |
| dasm | dolphm: yes. | 20:09 |
| *** yarkot_ has quit IRC | 20:09 | |
| dolphm | dasm: that's a security / user experience tradeoff that you have to weigh yourself as a deployer | 20:09 |
| gmmaha | dolphm: so since the scheduling was done with 'A' token when it expired, noen ofthe subsequent requests bothered to change the token to a non-expired one? | 20:10 |
| gmmaha | Interesting!! :) | 20:10 |
| dolphm | dasm: but to your point, an hour is so low that no one will ever bother us upstream to lower the default any lower :P | 20:10 |
| dolphm | gmmaha: they can't -- your client generated the token | 20:10 |
| gmmaha | dolphm: haha.. :D | 20:10 |
| gmmaha | dolphm: aaaah | 20:10 |
| gmmaha | ok.. that makes sense now | 20:10 |
| dolphm | gmmaha: and at some point, the services just noticed that your token was expired, so they started disregarding the subsequent requests | 20:11 |
| dasm | dolphm: :) | 20:11 |
| dolphm | gmmaha: that's a fun problem if you're interested in solving it :P | 20:11 |
| gmmaha | dolphm: :) | 20:11 |
| gmmaha | dolphm: not going to say no to that. :P | 20:12 |
| gmmaha | dolphm: thanks much for taking the time to explain.. | 20:14 |
| gmmaha | makes a lot of sense now | 20:14 |
| *** inc0 has quit IRC | 20:16 | |
| dstanek | dolphm: i don't think you know what fun means | 20:17 |
| dasm | dstanek: probably everything depends on point of view ;) | 20:18 |
| *** rmevans has quit IRC | 20:19 | |
| dstanek | dasm: true, so it's fun for dolphm to watch gmmaha solve it :=) | 20:29 |
| * gmmaha has started the tests again with increased timeout | 20:29 | |
| gmmaha | dstanek: haha.. if only gmmaha knew how to | 20:30 |
| dasm | dstanek: xD | 20:30 |
| *** rderose has joined #osic | 20:31 | |
| *** anush_ has quit IRC | 20:42 | |
| *** ccneill has quit IRC | 20:44 | |
| *** nadeem has quit IRC | 20:54 | |
| *** ccneill has joined #osic | 20:55 | |
| *** Mudpuppy has quit IRC | 20:58 | |
| *** anush_ has joined #osic | 21:02 | |
| *** homerp_ has joined #osic | 21:04 | |
| *** ankurgupta has joined #osic | 21:21 | |
| *** ChanServ sets mode: +o ankurgupta | 21:21 | |
| *** ankurgupta has left #osic | 21:21 | |
| *** homerp_ has left #osic | 21:32 | |
| *** sfinucan has quit IRC | 21:33 | |
| *** b3rnard0 is now known as b3rnard0_away | 21:54 | |
| *** anush_ has quit IRC | 21:59 | |
| *** anush_ has joined #osic | 22:00 | |
| *** raddaoui has joined #osic | 22:16 | |
| *** jlwhite_ has joined #osic | 22:28 | |
| *** jlwhite has quit IRC | 22:28 | |
| *** asettle has joined #osic | 22:36 | |
| *** KrishR has quit IRC | 22:41 | |
| *** spotz is now known as spotz_zzz | 22:42 | |
| *** anush_ has quit IRC | 22:46 | |
| *** jthorne has quit IRC | 22:48 | |
| *** ametts has quit IRC | 22:48 | |
| *** asingh has quit IRC | 23:02 | |
| *** asingh has joined #osic | 23:03 | |
| *** karthikp has quit IRC | 23:08 | |
| *** asingh has quit IRC | 23:11 | |
| *** jlwhite_ has quit IRC | 23:21 | |
| *** ccneill has quit IRC | 23:46 | |
| *** rahulunair has quit IRC | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!