Thursday, 2016-01-21

*** dshulyak_ has joined #openstack-solar07:56
*** salmon_ has joined #openstack-solar08:33
openstackgerritDmitry Shulyak proposed openstack/solar: Add init script for custom worker  https://review.openstack.org/27066508:44
openstackgerritLukasz Oles proposed openstack/solar: Use exported env name, change default name  https://review.openstack.org/27067909:17
pigmejsup guys?09:22
bogdandofolks, where is the "gate-solar-docs" script?09:26
bogdandoit calls "tox -evenv -- python setup.py build_sphinx" I need it as tox -edocs instead09:27
salmon_bogdando: here https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml#L250 :)09:28
salmon_it's not defined by us09:28
openstackgerritLukasz Oles proposed openstack/solar: Use exported env name, change default name  https://review.openstack.org/27067909:43
salmon_guys, please take a look at this https://review.fuel-infra.org/#/c/16320/2 it adds jenkins job to run example test09:45
pigmejI looked09:55
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-solar10:02
bogdandosalmon_, https://review.openstack.org/#/c/27069810:05
salmon_thx10:05
openstackgerritBogdan Dobrelya proposed openstack/solar: Add tox checkniceness,checksyntax for docs  https://review.openstack.org/26955910:06
pigmejhttps://review.openstack.org/#/c/269653/ + https://review.openstack.org/#/c/270247/  review it guys10:07
salmon_pigmej: does this two depends on each other?10:20
pigmejyup10:20
pigmej208 => 24710:20
pigmej247 depends on 20810:20
pigmej208 is simple it just adds one exception in 'else'10:21
pigmej247 uses this exception ;)10:21
*** mihgen has quit IRC10:29
*** mihgen has joined #openstack-solar10:39
openstackgerritJedrzej Nowak proposed openstack/solar: Unified exception in repo create  https://review.openstack.org/27020811:06
pigmejdshulyak_: check this again, salmon wanted to change exception string :)11:06
openstackgerritJedrzej Nowak proposed openstack/solar: Unified exception in repo create  https://review.openstack.org/27020811:28
pigmejwrrr, pep8 dshulyak_ salmon_ I fixed that pep8....11:28
bogdandosalmon_, tha infra patch throws Job gate-solar-tox-checkniceness not defined11:55
bogdandoJob gate-solar-tox-checksyntax not defined11:55
bogdandoI'm totally confused now, which places are to define gates11:55
salmon_bogdando: no idea, it's better to ask them on IRC11:55
bogdandolet's at least accept my patch and later think about making a check gate for it as well11:55
dshulyak_catched siblings error11:59
dshulyak_SiblingsError: No empty object for resolution not sure what to do with siblings: ['{"status": "INPROGRESS", "task_type": "solar_resource", "target": "", "args": ["data_resource_2_6", "run"], "timelimit": 0, "childs": ["system_log:a377dc6b-f3df-4002-b1a4-0409d7eac130~data_resource_3_6.run"], "parents": ["system_log:a377dc6b-f3df-4002-b1a4-0409d7eac130~data_resource_1_6.run"], "execution": "system_log:a377dc6b-f3df-4002-b1a4-0409d7eac130", "errmsg": ""11:59
dshulyak_"name": "data_resource_2_6.run"}', '{"status": "PENDING", "task_type": "solar_resource", "target": "", "args": ["data_resource_2_6", "run"], "timelimit": 0, "childs": ["system_log:a377dc6b-f3df-4002-b1a4-0409d7eac130~data_resource_3_6.run"], "parents": ["system_log:a377dc6b-f3df-4002-b1a4-0409d7eac130~data_resource_1_6.run"], "execution": "system_log:a377dc6b-f3df-4002-b1a4-0409d7eac130", "errmsg": "", "name": "data_resource_2_6.run"}']11:59
pigmejdshulyak_: on what ?11:59
dshulyak_i have test11:59
dshulyak_https://review.openstack.org/#/c/269166/10/solar/test/functional/test_complete_solar_workflow.py12:00
dshulyak_but i also changed zeromq puller implementation12:00
pigmejdoes you have concurrency there ?12:01
pigmejdo you have*12:01
dshulyak_yeah, scale in that tests represents number of concurrent chains12:01
dshulyak_and in puller as well12:01
dshulyak_lock spams like crazy :)12:02
pigmejit would mean that lock is not working properly OR some other issue ...12:02
pigmejhttps://bpaste.net/show/d9cd2b8ff93312:03
pigmejdshulyak_: ^12:03
pigmejit should basically print always list with the same value on every index12:04
pigmejhttps://github.com/Mirantis/solar-resources/pull/512:08
openstackgerritDmitry Shulyak proposed openstack/solar: Add implementation of puller with pool of threads and tracking of tasks  https://review.openstack.org/27076512:25
openstackgerritDmitry Shulyak proposed openstack/solar: Implement timelimit mechanism based on tracking of tasks  https://review.openstack.org/27076612:25
pigmejdshulyak_: can we focus on release today ? please ?12:26
pigmej:)12:26
dshulyak_y, i’m not asking to debug it12:33
pigmejdshulyak_: yeah I know, but we need to document stuff still12:36
dshulyak_which stuff?12:36
pigmejconfiguration options for example12:36
dshulyak_for celery :) ?12:37
pigmejand we have 4 pending change sets12:37
pigmejdshulyak_: yeah12:37
dshulyak_oh, nobody cares about that12:37
salmon_about what?12:38
pigmejwe have also undocummented "how to run solar without vagrant"12:38
pigmej( salmon_ has it in TODO though, isn't it ?)12:38
salmon_pigmej: which part?12:39
pigmejsolar without vagrant12:39
salmon_nope12:39
pigmejyou said that you will test it and add to readmes :)12:39
salmon_ah, I will test it :)12:39
pigmejyou said that you will add it to readme :P12:39
salmon_ok, I remember ;)12:40
openstackgerritMerged openstack/solar: Use isolated filebased sqlite for each test process  https://review.openstack.org/27018112:40
pigmejhttps://github.com/Mirantis/solar-resources/pull/612:42
pigmejwe need to improve and note things that we can change / adjust etc, because otherwise there will be flame war, why the fuck do you use sqlite for celery not redis, etc.12:45
salmon_pigmej, please add such notes in this etherpads12:45
salmon_https://etherpad.openstack.org/p/os-dev-annoucement-solar-0.0.112:45
salmon_https://etherpad.openstack.org/p/mos-dev-annoucement-solar-0.0.112:45
pigmejwhat do you mean?12:46
pigmejI'm now adding simple FAQ to our documentation (about key/password fuckup)12:46
salmon_ah, ok. FAQ is better12:46
salmon_can someone merge it https://review.openstack.org/#/c/270062/ ?12:52
pigmejI add +2 yesterday :P12:55
pigmejadded*12:55
openstackgerritMerged openstack/solar: Unified exception in repo create  https://review.openstack.org/27020813:00
dshulyak_pigmej: i’m seeing this error with hosts_file example - http://paste.openstack.org/show/484562/13:01
dshulyak_it is with unified results13:02
pigmejyeah because you wanted None there13:02
pigmejisn't it ?13:03
dshulyak_aha, but rsync returns None None None even if everything is ok13:03
pigmejah, ok I will check it asap then, because it shouldn't13:03
pigmejweird13:04
openstackgerritLukasz Oles proposed openstack/solar: Set correct classifier  https://review.openstack.org/27078513:05
salmon_required for pypi^13:05
salmon_afk,lunch13:05
pigmejah ok I see the fuckup dshulyak_ :)13:05
openstackgerritJedrzej Nowak proposed openstack/solar: Unify returned transport results  https://review.openstack.org/26965313:12
pigmejdshulyak_: ^13:12
openstackgerritJedrzej Nowak proposed openstack/solar: Added EGroup click group which can handle errors  https://review.openstack.org/27024713:13
pigmej^ rebased13:13
openstackgerritJedrzej Nowak proposed openstack/solar: Unify returned transport results  https://review.openstack.org/26965313:17
dshulyak_pigmej: you have prints there :) https://review.openstack.org/#/c/269653/6/solar/core/transports/base.py13:20
pigmejdshulyak_: removed already :D13:20
pigmejin next changeset :)13:20
dshulyak_now error is nicer13:20
dshulyak_hosts_file2.run -> ERROR :: ssh: connect to host 10.0.0.4 port 22: No route to host13:20
dshulyak_rsync: connection unexpectedly closed (0 bytes received so far) [sender]13:20
dshulyak_rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]13:21
dshulyak_hosts_file1.run -> ERROR :: ssh: connect to host 10.0.0.3 port 22: No route to host13:21
dshulyak_rsync: connection unexpectedly closed (0 bytes received so far) [sender]13:21
dshulyak_rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]13:21
dshulyak_oh13:21
pigmej:)13:21
openstackgerritJedrzej Nowak proposed openstack/solar: Added simple FAQ to our docs  https://review.openstack.org/27079113:24
pigmejbrb dinner is there :)13:43
openstackgerritDmitry Shulyak proposed openstack/solar: Add documentation about celery configuration options  https://review.openstack.org/27080613:48
dshulyak_hm, i will abandon my patch, that faq is more than enough13:48
salmon_dshulyak_: workflow +1 ;) https://review.openstack.org/#/c/270062/214:12
openstackgerritMerged openstack/solar: Move all text from README to documentation  https://review.openstack.org/27006214:15
salmon_thx14:16
openstackgerritLukasz Oles proposed openstack/solar: Remove last `/` from urls  https://review.openstack.org/27082514:23
openstackgerritJedrzej Nowak proposed openstack/solar: Added simple FAQ to our docs  https://review.openstack.org/27079114:26
pigmejsalmon_: changed what you wanted14:26
pigmejsalmon_: https://review.openstack.org/#/c/269653/ workflow please14:27
pigmejdshulyak_ check this please https://review.openstack.org/#/c/270247/14:27
openstackgerritMerged openstack/solar: Added EGroup click group which can handle errors  https://review.openstack.org/27024714:34
salmon_pigmej: https://review.openstack.org/#/c/270679/14:39
salmon_and https://review.openstack.org/#/c/270785/14:39
salmon_and all https://review.openstack.org/#/c/270825/14:40
pigmejsalmon_: yeah doing it, we already discussed some of them ;P14:40
pigmejsalmon_: should we link to 'latest' ?14:41
pigmejI missclicked that one :D14:42
salmon_for now we don't have any other14:44
pigmejawesome!14:46
pigmejI wonder wtf https://review.openstack.org/#/c/267453/14:46
pigmejsalmon_: you're moving all stuf on etherpad ?14:47
salmon_yup14:48
openstackgerritDmitry Shulyak proposed openstack/solar: Zerorpc worker for orchestration modules  https://review.openstack.org/26916614:48
openstackgerritDmitry Shulyak proposed openstack/solar: Add implementation of puller with pool of threads and tracking of tasks  https://review.openstack.org/27076514:51
openstackgerritMerged openstack/solar: Use exported env name, change default name  https://review.openstack.org/27067914:51
openstackgerritMerged openstack/solar: Set correct classifier  https://review.openstack.org/27078514:52
openstackgerritMerged openstack/solar: Unify returned transport results  https://review.openstack.org/26965314:54
openstackgerritMerged openstack/solar: Add tox checkniceness,checksyntax for docs  https://review.openstack.org/26955914:54
pigmejdshulyak_: https://github.com/Mirantis/solar-resources/pulls14:54
pigmejsalmon_: dshulyak_ I cannot fix / improve "parse puppet log from STDOUT" because I lack puppet knowlegde now, or do you have some resources where I can check what can I expect on stdout ?14:57
salmon_hmm, what do we need to do here? I thought each puppet resource has logs in separate file14:58
dshulyak_and also in stdout14:58
dshulyak_but idont remember what it writes to stdout and what to stderr14:58
salmon_btw. if we are requiring  sqlalchemy now can we drop peewee?15:01
dshulyak_maybe someone have solar log with puppet task?15:01
pigmejsalmon_: sqlalchemy is "BIG" and we require it only for celery '15:02
pigmejso the answer is no :)15:02
salmon_ah, ok15:02
salmon_dshulyak_: https://review.openstack.org/#/c/270825/ https://review.openstack.org/#/c/270791/15:03
salmon_last two commits required for release15:03
dshulyak_i think its ok to merge small patches wihtout 2 reviews15:04
pigmejI think the same15:04
salmon_rules breakers!15:04
pigmejbecause we're overflowed otherwise15:04
salmon_but I agree :)15:04
pigmejand now define 'small' :)15:04
salmon_hmm, where is jenkins?15:04
pigmejsalmon_: it has a lag now15:04
salmon_;(15:04
pigmejbut hmm15:05
pigmejhttps://review.openstack.org/#/c/270791/ this looks weird15:05
salmon_yup15:05
openstackgerritLukasz Oles proposed openstack/solar: Added simple FAQ to our docs  https://review.openstack.org/27079115:05
salmon_I rebased it15:05
salmon_do we need anything else to commit/merge? Except maybe some docs fixes?15:06
pigmejprobably not15:07
pigmejbut I will check all thigns again15:07
salmon_ok, so let's test it :)15:07
salmon_I will try wordpress example without vagrant15:08
pigmejremember about running celery ;)15:08
salmon_yeah15:08
salmon_and please update this https://etherpad.openstack.org/p/solar-release-0.1.015:10
openstackgerritMerged openstack/solar: Remove last `/` from urls  https://review.openstack.org/27082515:14
pigmejsalmon_: it is updated15:14
salmon_good :)15:15
pigmejsalmon_: https://review.openstack.org/#/c/270791/ wtf is there ?15:22
pigmejI will rebase in on my local machine15:23
openstackgerritJedrzej Nowak proposed openstack/solar: Added simple FAQ to our docs  https://review.openstack.org/27079115:23
pigmejrebased salmon_15:24
salmon_it's in a queue15:24
salmon_http://status.openstack.org/zuul/15:24
pigmejnow it it15:24
pigmejit is15:24
pigmejsalmon_: just remember to run celery from the same user as cli commands15:26
salmon_I know15:30
dshulyak_altered test a bit - http://paste.openstack.org/show/484585/15:31
pigmejdshulyak_: it's incorrect15:32
dshulyak_why?15:32
pigmejall items should be equal15:32
pigmejah you have retries...15:33
dshulyak_no15:33
pigmejbut I'm not sure if you will be able to get all locks15:33
pigmejwithin time15:33
dshulyak_if im not able to get lock - then lock should fail, isnt it?15:33
pigmejyeah, but it will be not equal range(100) then15:33
dshulyak_yes, but i what i see is that sometimes there is 2 lockers with 115:34
pigmejperfect15:34
dshulyak_no :)15:34
pigmejwhy?15:34
pigmejit's perfectly fine if you see multiple items15:34
dshulyak_18, 20, 21, 21,15:34
dshulyak_21 and 1915:34
pigmejand ?15:34
pigmejdshulyak_: you coded that lock in a bit different way.. :)15:34
pigmejwho_is_locking determines who owns lock15:35
dshulyak_and 19 thought that it acquired lock15:35
pigmejhmm, how ?15:35
dshulyak_but in fact it was acquired by 2115:35
pigmejdo you have who_is_locking() == 19 ?15:35
dshulyak_thats exactly the problem15:36
pigmejyour acquire doesn't break when acquire fails, it returns `lk`15:36
pigmejand at the end you need to check if lk.who_is_locking() == me15:36
dshulyak_but i shouldnt enter context manager at all15:36
dshulyak_it should fail with exception or acquire lock15:36
dshulyak_in __enter__15:36
pigmejhmm15:36
pigmejah15:37
pigmejwhy clear_cache ?15:37
dshulyak_on range of 100 - there 3 cases like this15:37
dshulyak_but still my worker fails even if i have concurrency > 2 :(15:37
pigmejdshulyak_: you shouldn't clean cache between __enter__ and __exit__ that's for sure15:38
pigmej(it may be noop though for this situation but still)15:38
dshulyak_but that just means that 19 and 21 will simutaneously acquire a lock, no?15:41
dshulyak_if they entered15:41
pigmejgenerally who_is_locking should determine what's going on15:41
pigmejbut for me they don't enter15:41
pigmejor hmm15:42
pigmejwait it's hard to intepret the results;D15:42
pigmejhmm15:43
pigmejok, BUT, why you lock again inside ?15:43
pigmejyeah, you're breaking it15:44
pigmejdshulyak_: return lk in __enter__15:45
pigmejand with RiakLock(...) as l:15:45
pigmejand remove your additional acquire15:45
pigmejhttps://bpaste.net/show/f0b5bddc101415:45
pigmejworks for me15:45
dshulyak_but thats doesnt tell you if there is 2 concurrent locks15:46
pigmejit does15:46
pigmejp.map() is parallel15:46
pigmejyou're acquiring one lock twice within one loop, and you're breaking release method probably,15:46
pigmejaka: untested behaviour15:46
dshulyak_but i shouldnt even enter this block :)15:48
pigmejit enters it, and it's correct15:48
pigmejbut THEN you do additional acquire15:48
pigmejwhich breaks everything15:48
openstackgerritMerged openstack/solar: Added simple FAQ to our docs  https://review.openstack.org/27079115:48
pigmej(at least it seems so dshulyak_ )15:49
pigmejand because you use different stamp, you're changing lock ownership15:49
dshulyak_i changed it a bit, result is the same - http://paste.openstack.org/show/484589/15:50
dshulyak_ok, lets leave it15:51
dshulyak_i will do some additional debuging15:51
pigmejyou cannot do it dshulyak_ like that I think15:51
pigmejyou're destroying lock ;D15:51
pigmejremove that .get()15:51
pigmejand add return in __enter__15:51
dshulyak_ok, i will use separate object for tracking collision15:51
pigmej?15:52
pigmejhttps://bpaste.net/show/6a5dc73f390615:52
dshulyak_but afaiu there should be no 2 lockers with 1, and no -115:52
pigmej?15:52
pigmejwith that modification that I did, I have equality in res == range(100)15:53
pigmejI executed it 50 times and 50 times equality15:53
pigmejdshulyak_: what probably happens, when you _acquire lock again (or get it), you alter behaviour of __release__15:54
pigmejbecause different state will be saved now15:54
pigmejand weird things happens :D15:54
pigmejat least I think it happens...15:56
salmon_/usr/local/bin/celery multi stopwait 2 :(16:11
pigmejsalmon_: you can use service solar-celery start|stop16:12
salmon_does it require full path?16:12
pigmejah fuck16:12
pigmejyeah it should say 'celery' instead :D16:12
pigmej /usr/bin/env celery ;)16:13
pigmejand we're fine16:13
salmon_ImportError: No module named gevent :(16:13
salmon_I think we require gevent now :)16:13
pigmejno16:13
pigmejyou need to execute worker correctly16:13
salmon_./utils/solar-celery start is not correct? :P16:14
pigmejno because it requires gevent :D16:14
pigmejbecause it has -P gevent16:14
salmon_so we also require a gevent :P16:14
pigmejsalmon_: no, you can prefork16:14
pigmejsalmon_: /usr/bin/env celery start 1 -A solar.orchestration.runner -P prefork -c 2 -Q system_log,celery,scheduler --pidfile=/var/run/celery/%N.pid --logfile=/var/run/celery/%N.log16:16
pigmejor something like that16:16
salmon_pfff16:16
salmon_you recommend solar-celery scrip in FAQ!16:16
pigmejsalmon_: I'm extending that entry then16:18
salmon_ok16:19
*** salmon_ has quit IRC16:37
openstackgerritJedrzej Nowak proposed openstack/solar: Improved celery worker section in FAQ  https://review.openstack.org/27091616:49
openstackgerritJedrzej Nowak proposed openstack/solar: use /usr/bin/env in celery script  https://review.openstack.org/27091716:50
*** salmon_ has joined #openstack-solar17:28
dshulyak_pigmej: looks like my test was catching intermediate state17:53
pigmejdshulyak_: you mean?19:41
pigmejwhen you _acquire three is local state involved, which *could* be wrong (you used different stamp)19:41
dshulyak_this one log.debug("I was not locking, so removing me %s" % identity)19:42
pigmejthis is ok, in fact19:42
pigmejI mean it's not bad that it reaches that stage19:42
pigmejbut that additional _acquire for sure changes semantic, I don't know if it's good or bad though ;D19:43
pigmejbtw you're still working?19:43
dshulyak_i think that this Lock.get happened before this write, thats why i was able to get this illusion of conflict19:43
pigmejposssible19:47
*** salmon_ has quit IRC19:51
*** salmon_ has joined #openstack-solar20:42
openstackgerritMerged openstack/solar: Improved celery worker section in FAQ  https://review.openstack.org/27091620:51
openstackgerritMerged openstack/solar: use /usr/bin/env in celery script  https://review.openstack.org/27091720:51
*** dshulyak_ has quit IRC21:18
*** dshulyak_ has joined #openstack-solar21:26
openstackgerritLukasz Oles proposed openstack/solar: Fix celery start script  https://review.openstack.org/27105922:20
*** dshulyak_ has quit IRC22:22
openstackgerritMerged openstack/solar: Fix celery start script  https://review.openstack.org/27105922:41
*** salmon_ has quit IRC23:02

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!