| *** angdraug has joined #openstack-solar | 00:44 | |
| *** angdraug has quit IRC | 01:45 | |
| -openstackstatus- NOTICE: gerrit is going to be restarted due to bad performance | 07:24 | |
| *** ChanServ changes topic to "gerrit is going to be restarted due to bad performance" | 07:24 | |
| *** ChanServ changes topic to "#openstack-solar" | 07:27 | |
| *** dshulyak has joined #openstack-solar | 08:05 | |
| pigmej | hello ! | 09:00 |
|---|---|---|
| pigmej | dshulyak: what about that versioning fixes? | 09:12 |
| dshulyak | hi | 09:14 |
| dshulyak | i didnt make them | 09:14 |
| pigmej | k | 09:14 |
| pigmej | :) | 09:14 |
| dshulyak | hm, actually what versioning fixes? | 09:16 |
| dshulyak | version in resource? | 09:16 |
| dshulyak | there wasnt any problem with that | 09:16 |
| *** salmon_ has joined #openstack-solar | 09:19 | |
| pigmej | .. | 09:22 |
| pigmej | i fixed parsing | 09:22 |
| pigmej | previous 8.0 was valid | 09:22 |
| salmon_ | rly? :D | 09:24 |
| pigmej | ;P | 09:28 |
| pigmej | salmon_: well, in some parst:P | 09:28 |
| pigmej | https://review.openstack.org/285336 | 09:29 |
| pigmej | we didn't merge it yet because of f2s | 09:29 |
| salmon_ | hmm, so 8.0 is valid or not? | 09:30 |
| pigmej | it is for now, but it shouldn't | 09:30 |
| pigmej | because 8.0.0 is valid for semver | 09:30 |
| salmon_ | ah, ok. I misunderstood you at the beginning | 09:31 |
| pigmej | salmon_: coffee!! | 09:32 |
| salmon_ | tee | 09:32 |
| salmon_ | *tea :) | 09:32 |
| pigmej | and see what it did to you... | 09:37 |
| openstackgerrit | Merged openstack/solar: Re-visit visited predecessors for react_on events https://review.openstack.org/288447 | 09:39 |
| pigmej | guys, can you review changes ? | 09:39 |
| pigmej | we have pretty a lot not merged stuff... | 09:39 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Added naive method to check if object already exists https://review.openstack.org/288560 | 09:40 |
| salmon_ | reviewing now | 09:42 |
| salmon_ | astute :P | 09:42 |
| pigmej | :> | 09:44 |
| dshulyak | what astute? | 09:48 |
| dshulyak | pigmej: ok, i thought that 8.0 is valid, but anyway i changed everything to 8.0.0 for 3-4 days | 09:49 |
| dshulyak | 3-4 days ago | 09:49 |
| dshulyak | salmon_: btw what happened with 3 controllers? | 09:50 |
| salmon_ | dshulyak: this one from astute https://review.openstack.org/#/c/288113/3 | 09:50 |
| salmon_ | dshulyak: 3 controllers failed with error in heat :( | 09:51 |
| pigmej | dshulyak: then we can safely merge this https://review.openstack.org/285336 | 09:51 |
| salmon_ | dshulyak: but it looks like error in fuel-library again | 09:51 |
| pigmej | salmon_: "always blame others (tm)" | 09:52 |
| salmon_ | pigmej: last time it was an error in libraryr :P | 09:52 |
| dshulyak | maybe it fails because we dont have retries, and in astute puppet will be retried ateast once | 09:52 |
| pigmej | shouldn't solar o retry "fix" it then? | 09:53 |
| salmon_ | dshulyak: I retried manually a few times | 09:53 |
| dshulyak | solar o retry is manual operation, there should be retries for each task that will be triggered in runtime | 09:54 |
| dshulyak | salmon_: have u created all 3 at once? | 09:55 |
| pigmej | yeah dshulyak sure | 09:55 |
| dshulyak | there is quite a lot of problems with updates | 09:56 |
| salmon_ | sure we should have it but it's not a case here. I retried it few times and there are retries in the task. | 09:56 |
| salmon_ | dshulyak: yes 3 at the same time | 09:56 |
| *** bogdando has quit IRC | 10:01 | |
| *** bogdando has joined #openstack-solar | 10:03 | |
| openstackgerrit | Merged openstack/solar: Added systemd solar-worker.service https://review.openstack.org/287744 | 10:04 |
| openstackgerrit | Maciej Kwiek proposed openstack/solar: Get task start and end time with defaults https://review.openstack.org/289243 | 10:12 |
| mkwiek | salmon_: please review this one: https://review.openstack.org/289243 , it should help with your KeyError, but I couldn't find the root cause, as I can't even reproduce it. I think we shouldn't really merge such fixes, but I am out of ideas for this one. | 11:33 |
| pigmej | maybe backend issue? | 11:34 |
| pigmej | afair lukasz uses sqlite | 11:35 |
| salmon_ | mkwiek: I agree with dshulyak | 11:38 |
| pigmej | yaeh me too, just wtf with that error then? | 11:38 |
| mkwiek | pigmej: I am also using sqlite | 11:38 |
| pigmej | salmon_: can you reproduce it somehow? | 11:39 |
| mkwiek | yeah, I agree with you all :D | 11:39 |
| salmon_ | pigmej: I don't need to, I'm hiting it all the time :) | 11:39 |
| pigmej | :| | 11:39 |
| mkwiek | salmon_: can you send me your database? | 11:39 |
| pigmej | still we need somehow to figure out the reason for it | 11:40 |
| salmon_ | mkwiek: wait a few minutes, I will try to reproduce it and will send you a db | 11:41 |
| mkwiek | cool, thanks a lot | 11:41 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Separate vagrantfile for the docker provider https://review.openstack.org/288525 | 11:55 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 11:55 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 11:58 |
| salmon_ | dshulyak: pigmej how can I fix it https://bpaste.net/show/2eb7522f383b? | 12:01 |
| salmon_ | https://bpaste.net/show/2eb7522f383b | 12:01 |
| pigmej | salmon_: solar o release_lock {id} | 12:02 |
| pigmej | how did you lokc it though? | 12:02 |
| pigmej | solar o release-lock | 12:03 |
| salmon_ | pigmej: I saved vm state :) | 12:03 |
| pigmej | ah | 12:04 |
| pigmej | ok :) | 12:04 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 12:42 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 12:44 |
| dshulyak | salmon_: have u tried to generate graph for 3 controllers? it just hangs for me | 13:07 |
| dshulyak | and it hangs not even in solar.. it is tred part | 13:08 |
| salmon_ | dshulyak: sure it worked | 13:08 |
| dshulyak | ok, maybe i need to wait | 13:09 |
| salmon_ | dshulyak: hmm, It was quite fast. max 1-3s delay | 13:09 |
| pigmej | ok 30% speedup with pg gained :D | 13:32 |
| pigmej | ok and no longer 100% CPU | 13:34 |
| pigmej | :) | 13:34 |
| salmon_ | yupi | 13:34 |
| salmon_ | :D | 13:34 |
| pigmej | still checking :) | 13:34 |
| pigmej | to verify :) | 13:34 |
| pigmej | hmm | 13:35 |
| pigmej | Total Delta: 13.1823182106 | 13:35 |
| pigmej | Total Time: 93.4946820736 | 13:36 |
| pigmej | wtf ? | 13:36 |
| pigmej | mkwiek: I'm pretty sure that somethign is still wrong in our time calculations | 13:36 |
| mkwiek | delta should be higher then total time, right? | 13:38 |
| pigmej | should be but in this example everythign is sequential | 13:39 |
| pigmej | but everytime I restart this job | 13:39 |
| pigmej | total delta is added to total time | 13:39 |
| pigmej | Total Delta: 14.3580310345 | 13:39 |
| pigmej | Total Time: 104.088195086 | 13:39 |
| dshulyak | :) | 13:39 |
| mkwiek | what do you mean added? | 13:39 |
| pigmej | Total Delta: 11.2815508842 | 13:40 |
| pigmej | Total Time: 180.409070015 | 13:40 |
| dshulyak | pigmej: so u are doing restarts or what? | 13:40 |
| pigmej | yup | 13:40 |
| pigmej | "but everytime I restart this job" :) | 13:40 |
| mkwiek | I guess it makes sense? | 13:40 |
| dshulyak | yeah, i just wanted to clarify | 13:40 |
| pigmej | mkwiek: that total time for the same job increases ? | 13:40 |
| mkwiek | because other tasks were already finished? | 13:40 |
| pigmej | restart starts from 0 | 13:41 |
| pigmej | if I would restart it tomorrow it would say me ~90000 seconds ? | 13:41 |
| dshulyak | there is still no reset of time on restart | 13:41 |
| pigmej | so total time is time of first task - time of last task ? | 13:42 |
| pigmej | it's total delta, right? | 13:42 |
| pigmej | so what's total time then? | 13:42 |
| dshulyak | no, total time is last - first | 13:42 |
| dshulyak | total delta is sum of (end - start) | 13:42 |
| dshulyak | so in your case it should equal | 13:42 |
| mkwiek | delta is time of all tasks combined, so if you have parallel tasks, you can end up with delta higher than total time | 13:42 |
| pigmej | ok, so total delta should be highher than totla time, isn't it? | 13:42 |
| pigmej | mkwiek: so we have a bug somewhere there | 13:43 |
| mkwiek | yeah, I guess so | 13:43 |
| pigmej | becuase I have total time higher than total delta ;p | 13:43 |
| dshulyak | as i said - there is no reset of start_time on reset | 13:43 |
| mkwiek | you are on master, right? | 13:43 |
| dshulyak | and end_time | 13:43 |
| pigmej | dshulyak: yup it seems so | 13:43 |
| dshulyak | i somehow broke graphviz with nfs :) | 13:44 |
| mkwiek | pigmej: should I submit it to launchpad, or will you? | 13:44 |
| mkwiek | I will take care of it | 13:44 |
| pigmej | mkwiek: ok I wil ldo it | 13:45 |
| pigmej | mkwiek: https://bugs.launchpad.net/solar/+bug/1554058 | 13:46 |
| openstack | Launchpad bug 1554058 in Solar "Total Time > Total Delta" [Undecided,New] | 13:46 |
| pigmej | beautiful report I know | 13:46 |
| mkwiek | thanks | 13:46 |
| dshulyak | :( no it is not related to nfs | 13:46 |
| pigmej | dshulyak: check if you have recent files, | 13:47 |
| mkwiek | pigmej: at least there is a way to reproduce it :D | 13:47 |
| pigmej | becuase for me pycs are sometimes broken on nfs | 13:47 |
| dshulyak | it hangs in graphviz programs, tred or dot, either on reads or brk, and it doesnt make any sense | 13:47 |
| pigmej | wtf monday :) | 13:48 |
| dshulyak | salmon_: what version of graphviz u are using? | 13:49 |
| pigmej | btw dshulyak what command exactly is not working? | 13:50 |
| pigmej | solar o dg last ? | 13:50 |
| dshulyak | pigmej: yes, but not in python part, it hangs on tred system_log.dot | dot … | 13:51 |
| mkwiek | pigmej: where can I find resource 'noop'? | 13:51 |
| salmon_ | dshulyak: dg commands are not working for me completly. It just generates dot file and then I'm generating svg on my local computer | 13:52 |
| mkwiek | pigmej: I guess I can write it myself :D | 13:52 |
| dshulyak | mkwiek: u can find one in f2s | 13:52 |
| dshulyak | salmon_: yes, strange that with 1 controller ( ~100 tasks) it works well | 13:53 |
| pigmej | mkwiek: wait I can give it you | 13:54 |
| salmon_ | dshulyak: I have 317 tasks | 13:54 |
| mkwiek | pigmej: cool | 13:54 |
| pigmej | but well | 13:54 |
| pigmej | https://bpaste.net/show/1ae37e73ad18 | 13:54 |
| pigmej | this is the meta :P | 13:54 |
| mkwiek | yeah, I guessed so :D | 13:55 |
| mkwiek | hanks | 13:55 |
| mkwiek | *thans even | 13:55 |
| mkwiek | *thanks | 13:56 |
| dshulyak | salmon_: btw what do you mean by completely? it hangs or what? | 14:03 |
| salmon_ | dshulyak: no, just generating dot file | 14:05 |
| mkwiek | pigmej: what do you mean by "then normal solar things"? | 14:05 |
| mkwiek | solar changes stage and process? | 14:05 |
| mkwiek | or sth else? | 14:05 |
| pigmej | stage / process | 14:05 |
| mkwiek | kk | 14:05 |
| mkwiek | thanks | 14:05 |
| salmon_ | I had some problem with installing package for dot/tred commands. I don't remember what was the issue | 14:06 |
| pigmej | dshulyak: ewetyhing works prefectly for me about graphs ... | 14:06 |
| pigmej | both o dg last or solar connections graph | 14:06 |
| dshulyak | for me the problems with generting dot graph starts with ~200 tasks | 14:07 |
| pigmej | I will create 500 and I will tell you :) | 14:08 |
| pigmej | works for me dshulyak ;/ | 14:08 |
| dshulyak | i suspect that the problem might be related to amount of edges.. | 14:08 |
| dshulyak | it doesnt work for me on local env either.. | 14:09 |
| pigmej | weird | 14:09 |
| pigmej | sqlite ? | 14:09 |
| pigmej | pg ? | 14:09 |
| pigmej | can you send me a dump? | 14:09 |
| mkwiek | pigmej: believe it or not, but I cannot reproduce this bug with incrementing total time :D | 14:09 |
| pigmej | pg backend | 14:10 |
| pigmej | I can reproduce it always | 14:10 |
| pigmej | have you used my script? | 14:10 |
| dshulyak | pigmej: why do u need database? | 14:10 |
| mkwiek | I had clear db in sqlite, I did exactly what you posted in the bug, and after each restart I am getting new value for total_time | 14:10 |
| pigmej | to check if the problem is DB or dot :) | 14:10 |
| pigmej | yo said that dot command hangs | 14:11 |
| pigmej | so it's rather not DB as long as it produces valid output... | 14:11 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 14:18 |
| mkwiek | pigmej: I mean, for me it works as it should, I get new total_time every time :/ | 14:20 |
| pigmej | I don't | 14:20 |
| dshulyak | pigmej: https://bpaste.net/show/c6533aea14d1, can u try to generate png from this dot graph? | 14:20 |
| pigmej | you used my scrip ? | 14:20 |
| dshulyak | or svg | 14:20 |
| pigmej | dshulyak: sur | 14:21 |
| dshulyak | tred system_log.dot | dot -Tsvg -o system_log.svg | 14:21 |
| mkwiek | pigmej: yes | 14:21 |
| dshulyak | this is what doesnt work for me | 14:21 |
| pigmej | hmm | 14:21 |
| pigmej | dshulyak: I have syntax error three | 14:21 |
| pigmej | Warning: /tmp/b.dot: syntax error in line 1 near 'igraph' | 14:21 |
| dshulyak | hm | 14:22 |
| dshulyak | it looks like u dropped 1st character | 14:22 |
| pigmej | yup | 14:23 |
| pigmej | so, it uses 100% CPU | 14:23 |
| pigmej | but maybe it will generate it;d | 14:23 |
| dshulyak | yeah, so it is the same as for me | 14:24 |
| pigmej | BUT | 14:24 |
| pigmej | it finished | 14:24 |
| dshulyak | crap ) | 14:24 |
| pigmej | it takes like 2minutes | 14:25 |
| pigmej | :D | 14:25 |
| pigmej | on my cpu, so... | 14:25 |
| dshulyak | its very strange, it is not that big | 14:26 |
| pigmej | |https://instant.io/#c92efb67cf19f8cfefdff18d9f9ecc91d47f50a7 | 14:26 |
| pigmej | dshulyak: what is your use case for 'exists' ? | 14:33 |
| dshulyak | if exists: return else create | 14:35 |
| pigmej | can I instead do create_if_not_exists? | 14:35 |
| pigmej | I don't want to expose raw exists | 14:35 |
| dshulyak | yes, sure, but i need to catch DBException | 14:35 |
| pigmej | with creaet_if_not_exists you would not need to | 14:36 |
| pigmej | you could have obj = create_if_not_exists(...) | 14:36 |
| pigmej | dshulyak: because this 'exists' stuff is kinda "experimental" I would prefer not expose public api for "exists" | 14:37 |
| dshulyak | sometimes i am calling cr.create and sometimes usual resource | 14:37 |
| pigmej | because then we will be not able to do something smarter with it | 14:37 |
| pigmej | ah ;/ | 14:37 |
| pigmej | crap ;/ | 14:37 |
| dshulyak | this tred takes long time to compile.. | 14:38 |
| pigmej | maybe with may_exists(resource_name): | 14:38 |
| pigmej | cr.create(resource_name,...) | 14:38 |
| pigmej | ? | 14:38 |
| pigmej | maybe this may_exists is not that stupid in fact | 14:39 |
| pigmej | it could support multiple arguments... | 14:39 |
| dshulyak | well, maybe u can leave it as it is, i am already doing res.load except NotFound | 14:47 |
| dshulyak | i will just switch order and exception | 14:47 |
| pigmej | it's not that I'm against .exists() but this current implementation is ... | 14:48 |
| pigmej | ok my pool seems working :D | 15:05 |
| pigmej | 30-40% boost on DB ops | 15:05 |
| pigmej | using pg | 15:05 |
| openstackgerrit | Maciej Kwiek proposed openstack/solar: Reset times in tasks on graph reset https://review.openstack.org/289396 | 15:31 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 15:38 |
| openstackgerrit | Jedrzej Nowak proposed openstack/solar: Implement better and more solar specific pool for PG https://review.openstack.org/289406 | 15:46 |
| dshulyak | i think i will have to implement this one https://blueprints.launchpad.net/solar/+spec/refactor-process-of-staging-changes , otherwise it is unclear how to handle partial updates of nailgun graph | 16:01 |
| dshulyak | and with this we will be able to re-run (insert in graph) part of already commited resources | 16:01 |
| pigmej | how much time will it take? | 16:02 |
| pigmej | roughly :) | 16:02 |
| dshulyak | day or two | 16:03 |
| pigmej | doesn't sound that bad then I think, it's worth to fix it now than later in fact... | 16:05 |
| pigmej | but is *everything* else from f2s ready enough? | 16:05 |
| dshulyak | what do we need to be ready from f2s? | 16:13 |
| dshulyak | plugins are not ready | 16:15 |
| pigmej | basic d eployment and prboably some LCM stuff | 16:15 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/solar: Add vbox/qemu packer build for centos7 https://review.openstack.org/289293 | 16:15 |
| dshulyak | what lcm stuff? | 16:15 |
| pigmej | *any* | 16:15 |
| pigmej | we need to support some class2 LCM | 16:15 |
| dshulyak | in f2s there is no support for this | 16:15 |
| dshulyak | and wont be actually :) | 16:16 |
| pigmej | I know but we need to select some (one?) use case and provide it | 16:16 |
| dshulyak | yes | 16:16 |
| pigmej | salmon_: what is missing in f2s ? | 16:16 |
| dshulyak | but i think the only way to do it is manually create composer file | 16:16 |
| pigmej | ;/ | 16:16 |
| pigmej | totally anti solar ;( | 16:16 |
| dshulyak | ? | 16:17 |
| dshulyak | there is no way to do it without manual composer file even with all solar features | 16:17 |
| pigmej | to change mysql pw I will need to create separate composer file ? | 16:17 |
| dshulyak | i am talking about moving mysql from one node to another | 16:17 |
| pigmej | ah yeah this one is different story :) | 16:17 |
| dshulyak | mysql pw is what? it is some parameter in mysql? | 16:18 |
| pigmej | yeah this one for sure requires some "policy engine like stuff" | 16:18 |
| pigmej | which may be just another composer file | 16:18 |
| pigmej | dshulyak: well, when we had the connections it was just a input for mysql_user resource afair | 16:18 |
| pigmej | and that resource was connected to several other resources | 16:18 |
| pigmej | so we need to support these "simple" use cases like change value and all connected resources are properly changed (we had connections for it) | 16:19 |
| dshulyak | this is all problematic.. | 16:19 |
| pigmej | AND we need somehow provide composer file / python file / whatever to show that "datach database" is also possible | 16:19 |
| pigmej | dshulyak: I know, but I spoke with salmon_ some time ago, and he said taht in the end connections *maybe* can be restored | 16:19 |
| pigmej | well, if we will not have *any* answer for even simple LCM, let's then stop even thinking about F2S at all, we need to show basic LCM that does not require any new resources (or removing old) and we need to show some class2 (like datach db) | 16:20 |
| dshulyak | f2s is not an answer to lcm, we still need it to bootstrap fuel composition in solar, after that everything should be done using solar api | 16:21 |
| pigmej | sure, I'm not saying about supporting it from Fuel UI :) | 16:22 |
| pigmej | just after you import it so solar, then all *solar* stuff should work, so basic LCM, and we should have answer for advanced one | 16:22 |
| pigmej | :) | 16:23 |
| *** angdraug has joined #openstack-solar | 17:45 | |
| *** angdraug has quit IRC | 18:01 | |
| *** angdraug has joined #openstack-solar | 18:19 | |
| *** angdraug has quit IRC | 19:29 | |
| openstackgerrit | Merged openstack/solar: Reset times in tasks on graph reset https://review.openstack.org/289396 | 20:03 |
| *** angdraug has joined #openstack-solar | 21:17 | |
| *** dshulyak has quit IRC | 21:47 | |
| *** angdraug_ has joined #openstack-solar | 22:02 | |
| *** dshulyak has joined #openstack-solar | 22:03 | |
| *** angdraug has quit IRC | 22:05 | |
| *** dshulyak has quit IRC | 22:08 | |
| *** angdraug_ has quit IRC | 22:42 | |
| *** angdraug has joined #openstack-solar | 22:42 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!