Friday, 2015-01-09

fungizaro: i think jamespage will have to do it (or ken conley but i don't see him in here)00:00
*** ZZelle_ has quit IRC00:00
*** serverascode____ is now known as serverascode00:01
openstackgerritMerged openstack-infra/project-config: Cleanup tempest-lib job list
*** patrickeast has joined #openstack-infra00:05
*** dimtruck is now known as zz_dimtruck00:05
*** baoli has quit IRC00:05
*** mbacchi has quit IRC00:06
openstackgerritMerged openstack/requirements: Update oslo.middleware to 0.3.0
zarofungi: thanks.  shall i just go ahead and use an openstack project?  would hate to create a testing bug against one of those projects.00:08
fungizaro: i suppose you could temporarily register a lp project corresponding to one of the gtest projects in review-dev00:10
zarofungi: i think i would prefer using an openstack project :)00:12
*** teran has joined #openstack-infra00:12
*** markvoelker has quit IRC00:14
zarofungi: i looked into removing local mysql from review-dev.  not sure whether it's a good idea now.00:14
*** marun has quit IRC00:14
zarofungi: puppet is all configured to make local mysql the default connection for gerrit.  we've just overriden with connection to trove.00:15
fungizaro: yeah, seems fine to leave it on there00:16
openstackgerritMerged openstack/requirements: Add new angular deps to global-requirements.txt for Horizon
*** shashankhegde has joined #openstack-infra00:18
openstackgerritMerged openstack-infra/devstack-gate: create the ability to pass through devstack config
*** andreykurilin has quit IRC00:18
openstackgerritKhai Do proposed stackforge/python-jenkins: this is for testing, DO NOT REVIEW
*** sabeen has quit IRC00:22
ianwclarkb / jeblair : <- not updating?00:25
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Broaden pattern match for
fungijeblair: clarkb: ^ turns out that was also broken on trusty, for example
fungiand with that, i'm heading offline for the nighth00:28
*** sarob has quit IRC00:28
* fungi vanishes in a puff of logic00:28
openstackgerritJames E. Blair proposed openstack-infra/zuul: Cloner: handle missing ZUUL_BRANCH/_REF
jeblairclarkb, fungi, sdague: ^ that's the last bit the zuul cloner needed to support all the use cases we're doing in devstack-gate.  after that lands i will start to remove the git operations from d-g.00:33
*** sabari is now known as zz_sabari00:34
*** Kiall has quit IRC00:34
*** Kiall has joined #openstack-infra00:36
openstackgerritKhai Do proposed stackforge/python-jenkins: this is for testing, DO NOT REVIEW
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Move Nova V2.1 job from experimental to check/gate
jeblairianw: it looks current to me?00:38
*** otter768 has joined #openstack-infra00:39
*** Masahiro has joined #openstack-infra00:40
ianwjeblair: last entry is '2015-01-08 19:26:20,796 INFO '00:42
*** shayneburgess has quit IRC00:43
jeblairianw: yep, right after an exit 100:43
*** asettle-mtg is now known as asettle00:43
*** Masahiro has quit IRC00:44
jeblairianw: switching back to the main debug log, it then logged: 2015-01-08 19:26:20,803 ERROR nodepool.DiskImageBuilderThread: Exception building DIB image:00:45
jeblairianw: so i think that's correctly logged the end of a failed image build00:45
*** wuhg has joined #openstack-infra00:45
*** yaguang has joined #openstack-infra00:46
ianwjeblair: it hasn't successfully built a f21 image, i think i should be retrying00:46
ianwit should be retrying00:46
*** teran_ has joined #openstack-infra00:46
ianwall the f21 builds seem to stop with00:46
ianw015-01-08 19:23:03,324 INFO + sudo /bin/bash /root/system-config/install_modules.sh00:46
jeblairianw: oh, i bet they got stuck when clarkb restarted nodepool00:48
*** tqtran_ has quit IRC00:48
*** sarob has joined #openstack-infra00:49
*** teran__ has joined #openstack-infra00:49
ianwsounds likely...00:49
*** teran has quit IRC00:49
jeblairi'll see if deleting them from the command line fixes it.  we may need to add something to nodepool to delete all building images on startup like we do with nodes00:49
*** otter768 has quit IRC00:49
*** ivar-lazzaro has joined #openstack-infra00:50
*** teran_ has quit IRC00:51
jeblairianw: yeah, i think that's unsticking them00:51
zarojeblair: i'm seeing this problem in review-dev,
zaroyou have any idea which would be the prefered solution there?00:53
zaromordred, fungi: ^ maybe you know? relates to mysql00:54
jeblairzaro: unless i'm mistaken, that's been a perpetual problem on review-dev and will go away after a few failures.  it never happens on prod because we're so busy.00:54
jeblairzaro: there's a pool of maybe 4 or 5 connections, and once they all fail, you're good for a while00:55
*** ivar-lazzaro has quit IRC00:55
*** bhuvan_ has quit IRC00:55
zaroit's comes up after leaving the UI idel for a few mins.00:56
ianwjeblair: ok, image.log doesn't have anything new, yet00:56
zarobrowser refresh fixes it.00:56
*** gmurphy has joined #openstack-infra00:56
*** alugovoi has quit IRC00:58
zarojeblair: so it's unlikely to happen on review.o.o but should we not apply one of those fixes to eliminate the problem?00:59
*** Masahiro has joined #openstack-infra00:59
*** serverascode has left #openstack-infra00:59
jeblairzaro: maybe, if you want to look into the implications of the autoreconnect flag.  i have not used that with jdbc, so don't know how it will manifest01:00
*** mfink has joined #openstack-infra01:03
zaroi'm not real familiar with DBs so I thought i would just poke for advice on which suggestion in the thread would be best,, autoreconnect or removing the connection pool.01:04
*** chuckC_ has joined #openstack-infra01:04
jeblairzaro: i don't see a reason to remove the connection pool since that would have a small impact on performance and we're not hitting the problem in prod01:04
jeblairianw: 2015-01-09 01:01:48,477 DEBUG nodepool.SnapshotImageUpdater: Image id: 4375 waiting for server 04692001-0b2f-4070-9bf7-f5797885bc3c01:05
jeblairianw: the first one is just now getting a server01:05
jeblairianw: should see something soon01:05
*** xgerman has quit IRC01:05
*** ChuckC has quit IRC01:06
*** chuckC__ has joined #openstack-infra01:07
*** gyee has quit IRC01:08
*** dannywilson has quit IRC01:09
*** chuckC_ has quit IRC01:09
*** chuckC__ is now known as chuckC_01:09
jeblairzaro: the times in that error log are very short01:09
jeblairzaro: less than 5 minutes01:09
jeblairzaro: that's not the old issue at all.  something has changed.01:09
*** sputnik13 has quit IRC01:10
zarojeblair: here's the log from review-dev
*** jyuso716 has joined #openstack-infra01:11
jeblairzaro: yeah, i know, i was just looking at it.  :)01:11
jeblairzaro: "175,201 milliseconds ago" is 3 minutes01:12
zaroyeah that sounds right.01:12
jeblairzaro: that's not the normal 12 hour timeout01:12
zaroi think it's just shorter idle now.  but seems like same issue.01:13
jeblairzaro: the server determines the idle timeout01:13
openstackgerritTravis Tripp proposed openstack/requirements: Remove old deprecated xstatic angular packages
*** jyuso has quit IRC01:14
*** jyuso- is now known as jyuso01:14
zarojeblair: i assume you mean the db server?01:15
jeblairzaro: yep01:15
*** patrickeast has quit IRC01:16
*** mtanino has quit IRC01:17
zarojeblair: are review & review-dev using the same Trove instance?01:18
jeblairzaro: i certainly hope not01:18
*** patrickeast has joined #openstack-infra01:18
zaroso does that mean we need to update the db to set higher timeout?01:19
*** MaxV has joined #openstack-infra01:20
jeblairzaro: | wait_timeout               | 28800 |01:20
jeblairit's set to the default of 8 hours01:20
zarohmm, i see.01:20
jeblairzaro: don't use review-dev for a few mins01:21
*** lttrl has quit IRC01:23
jeblairzaro: okay, you can use it again01:23
jeblairzaro: the connections last for exactly 120 seconds01:23
jeblair(of idle time)01:23
*** lttrl has joined #openstack-infra01:24
*** MaxV has quit IRC01:24
*** nosnos has joined #openstack-infra01:27
jeblair01:27:00.071262 IP > Flags [F.], seq 36658, ack 8165, win 501, options [nop,nop,TS val 4139423225 ecr 2207611380], length 001:27
jeblair01:27:00.109899 IP > Flags [.], ack 36659, win 331, options [nop,nop,TS val 2207641365 ecr 4139423225], length 001:27
jeblairzaro: the disconnection is coming from the mysql server01:27
*** lttrl has quit IRC01:29
zarointeresting, sorry i can't be more of help01:30
jeblairi can't either, i have to run01:30
*** spzala has quit IRC01:31
*** lttrl has joined #openstack-infra01:32
zarojeblair: line 176
*** unicell has joined #openstack-infra01:35
zaronot sure how that works yet, but might be a clue.01:35
* zaro gotta go too01:35
*** lttrl has quit IRC01:36
*** melwitt has quit IRC01:36
*** signed8bit is now known as signed8bit_ZZZzz01:37
*** weshay has quit IRC01:40
*** banix has quit IRC01:40
*** wenlock has quit IRC01:42
*** chuckC_ has quit IRC01:42
*** dprince has joined #openstack-infra01:42
*** ivar-lazzaro has joined #openstack-infra01:45
*** bdpayne has quit IRC01:46
*** ivar-lazzaro has quit IRC01:50
*** yamahata has quit IRC01:53
*** dims__ has joined #openstack-infra01:55
*** dims__ has quit IRC01:55
*** dims__ has joined #openstack-infra01:55
*** xianghuihui has joined #openstack-infra01:59
*** xianghuihui has quit IRC02:05
*** amotoki has joined #openstack-infra02:05
*** dprince has quit IRC02:08
*** signed8bit_ZZZzz is now known as signed8bit02:11
*** eharney_ has quit IRC02:12
*** signed8bit has quit IRC02:13
*** shashankhegde has quit IRC02:13
*** ddieterly has joined #openstack-infra02:13
*** mohitsharma has joined #openstack-infra02:15
*** mohitsharma has quit IRC02:15
*** mohitsharma has joined #openstack-infra02:16
*** hdd has quit IRC02:17
*** oomichi has joined #openstack-infra02:18
*** asselin has quit IRC02:22
*** kmartin has quit IRC02:22
*** asselin has joined #openstack-infra02:23
*** kmartin has joined #openstack-infra02:23
*** sarob has quit IRC02:23
*** ddieterl_ has joined #openstack-infra02:25
*** ddieterly has quit IRC02:27
*** ddieterly has joined #openstack-infra02:27
*** jamielennox is now known as jamielennox|away02:30
*** ddieterl_ has quit IRC02:31
*** tkelsey has joined #openstack-infra02:32
*** pcaruana is now known as pcaruana|afk|02:34
*** jamielennox|away is now known as jamielennox02:37
*** tkelsey has quit IRC02:37
*** chuckC_ has joined #openstack-infra02:40
*** kaisers1 has joined #openstack-infra02:41
*** shashankhegde has joined #openstack-infra02:41
*** hdd has joined #openstack-infra02:42
*** ricolin has joined #openstack-infra02:43
*** kaisers has quit IRC02:43
*** amotoki has quit IRC02:45
*** sarob has joined #openstack-infra02:46
*** chlong has joined #openstack-infra02:49
*** otter768 has joined #openstack-infra02:50
*** jamielennox is now known as jamielennox|away02:51
*** otter768 has quit IRC02:54
*** ChuckC has joined #openstack-infra02:56
*** yamahata has joined #openstack-infra02:56
*** kmartin has quit IRC02:57
*** kmartin has joined #openstack-infra02:57
*** chuckC_ has quit IRC02:58
*** Ryan_Lane has quit IRC03:00
*** jamielennox|away is now known as jamielennox03:01
*** amotoki has joined #openstack-infra03:03
*** sarob has quit IRC03:04
*** andreaf has quit IRC03:04
*** andreaf has joined #openstack-infra03:04
*** chuckC_ has joined #openstack-infra03:05
*** hdd has quit IRC03:07
*** cpallares has quit IRC03:08
*** MaxV has joined #openstack-infra03:08
*** sarob has joined #openstack-infra03:10
*** MaxV has quit IRC03:13
*** patrickeast has quit IRC03:22
*** nosnos has quit IRC03:24
*** david-lyle has joined #openstack-infra03:29
openstackgerritKen'ichi Ohmichi proposed openstack/requirements: Avoid to use boto 2.35.0 for workaround
*** MaxV has joined #openstack-infra03:31
*** otter768 has joined #openstack-infra03:36
*** MaxV has quit IRC03:36
*** harlowja is now known as harlowja_away03:38
*** harlowja_at_home has joined #openstack-infra03:39
*** dims__ has quit IRC03:40
*** dims__ has joined #openstack-infra03:41
*** sarob has quit IRC03:41
*** dims__ has quit IRC03:45
*** rwsu has quit IRC03:53
*** teran__ has quit IRC03:54
*** nosnos has joined #openstack-infra03:56
*** xgerman has joined #openstack-infra04:01
*** oomichi has quit IRC04:01
*** teran has joined #openstack-infra04:04
*** links has joined #openstack-infra04:07
*** bradjones has quit IRC04:12
*** ddieterly has quit IRC04:13
*** oomichi has joined #openstack-infra04:14
*** mwagner_lap has joined #openstack-infra04:14
*** rlandy has quit IRC04:17
*** david-lyle has quit IRC04:17
*** shashankhegde has quit IRC04:21
*** stevemar has joined #openstack-infra04:23
*** shayneburgess has joined #openstack-infra04:31
*** tkelsey has joined #openstack-infra04:33
*** tkelsey has quit IRC04:38
*** harlowja_at_home has quit IRC04:49
*** harlowja_at_home has joined #openstack-infra04:57
*** lingxian has joined #openstack-infra05:02
*** zxiiro has quit IRC05:03
lingxianhi, there, could somebody provide some articles about how to setup my own CI infrastrcture(including gerrit)? I found the posts writen by jaypipes, but some contents are outdated. (e.g.
*** harlowja_at_home has quit IRC05:07
*** garyh has quit IRC05:09
charzlingxian: what problems that you face? more detail will be great.05:11
*** asettle has quit IRC05:15
lingxiancharz: jaypipes's article talks about setup CI using upstream Gerrit. But I want setup everything necessary that can work together, say gerrit, zuul, JJB, jenkins.05:15
lingxiancharz, and what I did is install gerrit by hand, and then using jaypipes's git repo to setup others.05:16
charzlingxian: Do you create a wiki page of your CI server in openstack/3party CI wiki?05:18
*** hdd has joined #openstack-infra05:19
*** yfried|afk is now known as yfried05:19
lingxiancharz, no. Maybe I don't make me understood. I just want a OpenStack CI infra used inside my company, it has nothing to do with the upstream.05:20
*** nosnos has quit IRC05:21
*** nosnos has joined #openstack-infra05:21
charzlingxian: ok, The document of you pasted is setup a 3party CI and work with upstream.05:23
charzlingxian: You also need to fetch code from upstream, right?05:24
lingxiancharz, the code is hosted in our git server inside.05:24
charzlingxian: ok, I got it.05:25
lingxiancharz, :)05:25
lingxiancharz, what's your suggestion?05:25
*** nosnos has quit IRC05:26
charzlingxian: I think that's no document to describe that. You need to find source code of accout stackforge in github or wiki.05:26
*** xgerman has quit IRC05:27
charzlingxian: Make sure you can retrieve push log from internal gerrit server, and you can flow the steps of document to setup others.05:29
*** bhunter71 has quit IRC05:29
lingxiancharz, so, what I want to do is install gerrit by myself, then install others using jaypipes' git repo(just need to modify something), to see if it works.05:29
lingxiancharz, yes, I can see the output through 'ssh -p 29418 gerrit stream-events'05:30
charzlingxian: ok, that's good.05:30
charzlingxian: try setup a job with gerrit plugin on jenkins.05:31
*** yfried is now known as yfried|afk05:32
lingxiancharz, ok, when jenkins is installed after the puppet task, I'll check.05:33
lingxiancharz, thanks!05:33
charzlingxian: np :)05:33
charzlingxian: r you run whole OpenStack CI in your internal server?05:34
lingxiancharz, yes05:34
lingxiancharz, is it workable?05:35
charzlingxian: That's a huge project, it must needs a lot of engineers.05:35
charzlingxian: I think that's workable, but how many servers on it?05:36
lingxiancharz, AFAIK, gerrit, zuul, jjb, jenkins(master+slave) will do everything I need. But I'm not sure.05:37
lingxiancharz, and the nodepool05:37
charzlingxian: cool!05:37
*** otter768 has quit IRC05:38
lingxiancharz, I can ignore the real devstack-gate in the early stage. I just want the workflow is oK05:39
*** MarkAtwood has joined #openstack-infra05:39
*** Longgeek has joined #openstack-infra05:40
lingxiancharz, say, I push a patch to gerrit, zuul got the event and dicide which jobs to run, and jenkins to his job05:40
lingxiancharz, that's it for the first step.05:41
lingxiancharz, and then I can consider the complacated parts of the infra05:41
*** ddieterly has joined #openstack-infra05:43
*** jyuso has quit IRC05:44
*** ddieterly has quit IRC05:45
*** ddieterly has joined #openstack-infra05:45
*** ddieterly has quit IRC05:50
*** nosnos has joined #openstack-infra05:53
*** mrmartin has joined #openstack-infra06:03
*** MarkAtwood has quit IRC06:05
*** mohitsharma has quit IRC06:05
*** tomoe has joined #openstack-infra06:09
*** mrmartin has quit IRC06:14
*** ivar-lazzaro has joined #openstack-infra06:21
*** zz_avozza is now known as avozza06:21
*** ivar-lazzaro has quit IRC06:27
*** teran_ has joined #openstack-infra06:32
*** pcrews has quit IRC06:32
*** mrmartin has joined #openstack-infra06:33
*** vigneshvar has joined #openstack-infra06:34
*** teran has quit IRC06:34
openstackgerritMerged openstack-infra/project-config: Make the Horizon integration tests job voting
*** sdake_ has joined #openstack-infra06:36
*** obondarev has joined #openstack-infra06:39
*** obondarev has quit IRC06:39
*** TravT_ has quit IRC06:41
*** ildikov has quit IRC06:41
*** TravT has joined #openstack-infra06:41
*** sarob has joined #openstack-infra06:41
*** ddieterly has joined #openstack-infra06:44
*** sarob has quit IRC06:46
*** camunoz has quit IRC06:46
*** nosnos has quit IRC06:48
*** ddieterly has quit IRC06:49
*** reed has quit IRC06:49
*** nosnos has joined #openstack-infra06:50
*** thingee has quit IRC06:56
*** Longgeek has quit IRC06:58
*** dims__ has joined #openstack-infra07:00
*** mrunge has joined #openstack-infra07:02
*** mohitsharma has joined #openstack-infra07:06
*** dims__ has quit IRC07:06
*** mohitsharma has quit IRC07:10
*** Longgeek has joined #openstack-infra07:11
*** jamielennox is now known as jamielennox|away07:15
*** mohitsharma has joined #openstack-infra07:18
*** tkelsey has joined #openstack-infra07:18
*** liam_ has joined #openstack-infra07:19
*** liam_ is now known as Guest1477207:20
*** berendt has joined #openstack-infra07:27
*** ildikov has joined #openstack-infra07:28
*** otter768 has joined #openstack-infra07:39
*** yfried|afk is now known as yfried07:39
*** alugovoi has joined #openstack-infra07:41
*** ddieterly has joined #openstack-infra07:44
*** otter768 has quit IRC07:44
*** coolsvap|afk is now known as coolsvap07:49
*** ddieterly has quit IRC07:49
*** hdd has quit IRC07:49
*** armax has quit IRC07:50
*** yfried is now known as yfried|afk07:54
*** Guest14772 has quit IRC07:55
*** pblaho_ has joined #openstack-infra07:55
*** jcoufal has joined #openstack-infra08:04
*** dhritishikhar has joined #openstack-infra08:06
*** ddieterly has joined #openstack-infra08:13
*** MaxV has joined #openstack-infra08:14
*** bhunter71 has joined #openstack-infra08:16
*** mpaolino has joined #openstack-infra08:17
*** ddieterly has quit IRC08:18
*** MaxV has quit IRC08:18
*** MaxV has joined #openstack-infra08:18
*** afazekas_ has joined #openstack-infra08:18
*** MaxV has quit IRC08:20
*** markus_z has joined #openstack-infra08:23
*** mrmartin has quit IRC08:23
*** e0ne has joined #openstack-infra08:24
*** yamahata has quit IRC08:24
*** hashar has joined #openstack-infra08:24
*** ivar-lazzaro has joined #openstack-infra08:25
*** e0ne has quit IRC08:27
*** tomoe has quit IRC08:27
*** alugovoi has quit IRC08:28
*** ivar-lazzaro has quit IRC08:30
*** vigneshvar has quit IRC08:30
*** vigneshvar has joined #openstack-infra08:31
*** dhritishikhar has quit IRC08:34
*** mpaolino has quit IRC08:40
*** luqas has joined #openstack-infra08:40
*** oomichi has quit IRC08:41
*** Hal_ has joined #openstack-infra08:43
*** yfried|afk is now known as yfried08:47
*** bhunter71 has quit IRC08:50
*** mpaolino has joined #openstack-infra08:52
*** MaxV has joined #openstack-infra08:53
*** dalgaaf has joined #openstack-infra08:57
*** mrmartin has joined #openstack-infra08:59
*** jpich has joined #openstack-infra09:01
*** Masahiro has quit IRC09:04
*** jistr has joined #openstack-infra09:05
*** Masahiro has joined #openstack-infra09:05
*** HeOS has joined #openstack-infra09:08
*** ankit_ag has joined #openstack-infra09:09
*** Longgeek_ has joined #openstack-infra09:10
*** Longgeek has quit IRC09:11
*** boris-42 has quit IRC09:13
*** berendt has quit IRC09:13
*** berendt has joined #openstack-infra09:15
*** derekh has joined #openstack-infra09:22
openstackgerritMike Heald proposed openstack-infra/infra-specs: Proposal for a 'fail-fast' option on jobs
*** marun has joined #openstack-infra09:28
*** ssam2 has joined #openstack-infra09:30
*** zz_johnthetubagu is now known as johnthetubaguy09:31
*** berendt has quit IRC09:32
*** dtantsur|afk is now known as dtantsur09:33
*** ZZelle has quit IRC09:34
*** ZZelle has joined #openstack-infra09:35
*** hashar has quit IRC09:36
*** mpaolino has quit IRC09:36
*** hashar has joined #openstack-infra09:37
openstackgerritCyril Roelandt proposed openstack-infra/project-config: Make the py33 gate voting for neutronclient
*** berendt has joined #openstack-infra09:40
*** otter768 has joined #openstack-infra09:40
*** stevemar has quit IRC09:44
*** andreaf has quit IRC09:44
*** smoser has quit IRC09:45
*** otter768 has quit IRC09:45
*** andreaf has joined #openstack-infra09:47
*** mpaolino has joined #openstack-infra09:48
*** andreaf has quit IRC09:49
*** andreaf has joined #openstack-infra09:49
*** smoser has joined #openstack-infra09:52
openstackgerritRoman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge
*** mohitsharma has quit IRC09:59
*** jp_at_hp has joined #openstack-infra10:05
*** KanagarajM has joined #openstack-infra10:07
ankit_agHello All tempest tests failing on
ankit_agdue to eeror message <Response><Errors><Error><Code>AuthFailure</Code><Message>Signature not provided</Message></Error></Errors><RequestID>req-6a3ea6ef-0658-4aaf-8996-0ef1277d9e0d</RequestID></Response>10:09
openstackgerritRoman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge
ankit_agI could not found any relative bug, do you guys have any idea10:10
afazekas_Is the boto version changed on pypi ?10:11
afazekas_I just seen this kind of issue..10:11
*** afazekas_ is now known as afazekas10:11
*** avozza is now known as zz_avozza10:16
afazekas it is just released10:16
openstackgerritafazekas proposed openstack/requirements: Deny boto==0.35.0
*** Masahiro has quit IRC10:26
*** ihrachyshka has joined #openstack-infra10:26
*** wznoinsk has joined #openstack-infra10:27
*** ivar-lazzaro has joined #openstack-infra10:27
*** zz_avozza is now known as avozza10:28
*** ivar-lazzaro has quit IRC10:32
*** e0ne has joined #openstack-infra10:36
gilliardafazekas: not sure if we have a policy to only deny reqs that we _know_ are broken.10:38
gilliardhence my comment on your patch.10:39
*** ldnunes has joined #openstack-infra10:40
*** unicell has quit IRC10:43
*** andreykurilin has joined #openstack-infra10:44
*** gilliard is now known as gilllliard10:46
*** unicell has joined #openstack-infra10:50
*** hashar has quit IRC10:54
afazekasI see the issue with 2.34.0,  the issue might be somewhere else10:57
*** pelix has joined #openstack-infra10:59
*** jlibosva has joined #openstack-infra11:00
*** aysyd has joined #openstack-infra11:02
gilllliard2.34.0 ?11:02
afazekasI logged in to other wise broken setup :( , but yes I saw 2.34.011:03
*** e0ne is now known as e0ne_11:03
openstackgerrityolanda.robla proposed openstack-infra/storyboard-webclient: Add maxlength validation for fields
*** pc_m has joined #openstack-infra11:09
*** KanagarajM has quit IRC11:12
*** links has quit IRC11:13
*** e0ne_ has quit IRC11:13
*** alexpilotti has joined #openstack-infra11:14
*** rfolco has joined #openstack-infra11:18
*** amotoki_ has joined #openstack-infra11:19
afazekasgilllliard, looks like the gate working with 146049 , is 'boto>=2.32.1,<2.35.0' ok as version spec ?11:21
DavieyIsn't this a bit much?  From: "A mailing list for the OpenStack Stable Branch test reports. <>"11:22
gilllliardafazekas: works for me! I don't know the policy - I was just asking.11:22
gilllliardBecause I think they are likely to release 2.35.x soon (perhaps) which would break us again11:23
*** rfolco has quit IRC11:23
*** e0ne has joined #openstack-infra11:23
*** rfolco has joined #openstack-infra11:25
*** adalbas has joined #openstack-infra11:26
*** Masahiro has joined #openstack-infra11:27
afazekasgilllliard: we usually open a ticket when something fails, in this case very likely it will be fixed in the next release11:29
*** rfolco has quit IRC11:30
*** rfolco has joined #openstack-infra11:30
*** Masahiro has quit IRC11:31
*** mpaolino has quit IRC11:34
*** rfolco has quit IRC11:35
*** rfolco has joined #openstack-infra11:35
*** rfolco has quit IRC11:39
*** ssam2 has quit IRC11:40
*** ssam2 has joined #openstack-infra11:40
*** otter768 has joined #openstack-infra11:41
*** tomoe has joined #openstack-infra11:44
openstackgerritafazekas proposed openstack/requirements: Limit boto version to <2.35.0
*** otter768 has quit IRC11:46
*** mohitsharma has joined #openstack-infra11:46
*** avozza is now known as zz_avozza11:49
*** andreykurilin has quit IRC11:49
*** jcoufal_ has joined #openstack-infra11:50
*** cdent has joined #openstack-infra11:50
*** mfink has quit IRC11:50
openstackgerritRadomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras
*** rfolco has joined #openstack-infra11:52
*** jcoufal has quit IRC11:52
*** tomoe has quit IRC11:52
*** nosnos has quit IRC11:54
*** rfolco has quit IRC11:56
*** erlon_ has joined #openstack-infra11:57
*** rfolco has joined #openstack-infra11:57
*** e0ne is now known as e0ne_11:58
*** mrmartin has quit IRC11:58
*** arxcruz has joined #openstack-infra11:59
*** rfolco has quit IRC12:02
*** mohitsharma has quit IRC12:03
*** e0ne_ has quit IRC12:03
*** rdopiera has joined #openstack-infra12:05
rdopierais Andreas Jaeger here?12:05
*** EmilienM|afk is now known as EmilienM12:07
*** mohitsharma has joined #openstack-infra12:09
openstackgerritRadomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras
*** e0ne has joined #openstack-infra12:13
*** rfolco has joined #openstack-infra12:14
*** dizquierdo has joined #openstack-infra12:15
*** boris-42 has joined #openstack-infra12:15
*** ldnunes has quit IRC12:17
*** rfolco has quit IRC12:18
*** ldnunes has joined #openstack-infra12:20
*** Longgeek_ has quit IRC12:21
*** ldnunes has quit IRC12:22
*** mrmartin has joined #openstack-infra12:22
*** ldnunes has joined #openstack-infra12:24
*** rfolco has joined #openstack-infra12:30
*** ivar-lazzaro has joined #openstack-infra12:31
*** chlong has quit IRC12:32
*** jp_at_hp has quit IRC12:33
*** mohitsharma has quit IRC12:34
*** rfolco has quit IRC12:35
*** jp_at_hp has joined #openstack-infra12:35
*** ivar-lazzaro has quit IRC12:36
*** rfolco has joined #openstack-infra12:36
*** hashar has joined #openstack-infra12:38
*** erlon_ is now known as erlon12:39
*** rfolco has quit IRC12:41
*** jedimike has joined #openstack-infra12:41
*** rfolco has joined #openstack-infra12:41
*** Longgeek has joined #openstack-infra12:47
*** EmilienM is now known as EmilienM|afk12:52
*** jp_at_hp has quit IRC12:55
*** rfolco has quit IRC12:57
*** jp_at_hp has joined #openstack-infra12:59
*** jp_at_hp has quit IRC12:59
*** jp_at_hp has joined #openstack-infra12:59
*** dhritishikhar has joined #openstack-infra12:59
*** Masahiro has joined #openstack-infra12:59
openstackgerritSam Betts proposed openstack-infra/project-config: Add networking-cisco-apic to stackforge
*** johnthetubaguy is now known as zz_johnthetubagu13:02
*** MarkAtwood has joined #openstack-infra13:03
*** MarkAtwood has quit IRC13:03
*** Masahiro has quit IRC13:04
*** baoli has joined #openstack-infra13:07
*** baoli_ has joined #openstack-infra13:08
*** baoli_ has quit IRC13:08
*** EmilienM|afk is now known as EmilienM13:09
*** baoli_ has joined #openstack-infra13:09
*** baoli has quit IRC13:12
*** bswartz has quit IRC13:14
*** radez_g0n3 is now known as radez13:19
*** rdopiera has quit IRC13:19
*** rdopiera has joined #openstack-infra13:19
*** dprince has joined #openstack-infra13:21
*** rlandy has joined #openstack-infra13:24
*** mpaolino has joined #openstack-infra13:24
openstackgerritRadomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras
*** hashar has quit IRC13:27
*** rdopiera has quit IRC13:27
*** dims__ has joined #openstack-infra13:28
jaypipeslingxian: pong. sorry, our timezones are quite offset. can you email me at and I will try to help you on your CI tasks as much as I can. thx!13:29
*** rdopiera has joined #openstack-infra13:29
*** ddieterly has joined #openstack-infra13:31
*** dims_ has joined #openstack-infra13:33
*** berendt has quit IRC13:34
*** dkranz has joined #openstack-infra13:34
*** hashar has joined #openstack-infra13:34
*** dims__ has quit IRC13:35
*** mbacchi has joined #openstack-infra13:35
*** dkliban_afk is now known as dkliban13:36
*** _nadya_ has joined #openstack-infra13:37
*** dims_ has quit IRC13:37
*** ankit_ag has quit IRC13:38
*** _nadya_ has quit IRC13:40
*** dims__ has joined #openstack-infra13:41
*** otter768 has joined #openstack-infra13:42
openstackgerritRadomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras
*** otter768 has quit IRC13:46
openstackgerritRadomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras
*** weshay has joined #openstack-infra13:49
*** _nadya_ has joined #openstack-infra13:51
*** dkranz has quit IRC13:52
*** pcaruana|afk| has quit IRC13:53
*** annegentle has joined #openstack-infra13:54
*** _nadya_ has joined #openstack-infra13:54
*** sigmavirus24_awa is now known as sigmavirus2413:55
ZZelleHi everyone13:57
ZZelleIt seems there is a trouble with boto in neutron tests13:58
rdopierawhat is the nature of said problem?13:59
ZZellerdopiera, boto version has been updated on pypi yesterday14:00
*** zz_johnthetubagu is now known as johnthetubaguy14:02
salv-orlandoI am pretty sure people here already know what's going on - it might be a coincidence but failures started at midnight14:02
salv-orlandoZZelle: not just neutron - all boto tests fail.14:03
salv-orlandoregardless of the job where they are executed14:03
ZZellesalv-orlando, indeed it seems tempest tests are no more working14:04
*** vigneshvar has quit IRC14:04
salv-orlandoZZelle: afazekas has a bug report ->
uvirtbotLaunchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,Confirmed]14:05
salv-orlandoZZelle: and a patch which is in the gate queue... so nothing to be worried about, it's going to be fixed soon14:05
*** rdopiera has left #openstack-infra14:06
*** mattfarina has joined #openstack-infra14:07
*** bswartz has joined #openstack-infra14:08
*** mfink has joined #openstack-infra14:08
*** coolsvap is now known as coolsvap|afk14:10
*** ddieterly has quit IRC14:11
*** signed8bit has joined #openstack-infra14:11
ZZellesalv-orlando, great14:11
*** dkranz has joined #openstack-infra14:11
mpaolinoZZelle, salv-orlando I have the same problem with
ZZellesalv-orlando, so it's a good time for a beer :)14:12
ZZellempaolino, patience is the response :)14:12
mpaolinoshould we "recheck" later?14:12
mpaolinoor the tests will be triggered automagically? :)14:13
ZZellempaolino, no, we should wait and then recheck manually14:14
salv-orlandompaolino: wait for this to merge otherwise rechecks are always going to fail
mpaolinothanks guys14:15
*** dustins has joined #openstack-infra14:19
*** coolsvap|afk is now known as coolsvap14:19
*** ryanpetrello has joined #openstack-infra14:20
*** amotoki_ has quit IRC14:20
*** e0ne is now known as e0ne_14:20
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Enable functional testing job for ironic-discoverd
openstackgerritMerged openstack/requirements: Limit boto version to <2.35.0
*** miqui_ has quit IRC14:26
*** _nadya_ has quit IRC14:30
*** e0ne_ has quit IRC14:31
*** mriedem has joined #openstack-infra14:32
*** ivar-lazzaro has joined #openstack-infra14:33
*** dhritishikhar has quit IRC14:33
*** ivar-lazzaro has quit IRC14:37
*** yfried has quit IRC14:38
*** amitgandhinz has joined #openstack-infra14:40
*** teran_ has quit IRC14:41
*** shayneburgess has quit IRC14:42
*** ddieterly has joined #openstack-infra14:43
*** armax has joined #openstack-infra14:46
*** Masahiro has joined #openstack-infra14:48
*** mjturek has joined #openstack-infra14:50
*** dannywilson has joined #openstack-infra14:52
*** Masahiro has quit IRC14:52
*** pcaruana|afk| has joined #openstack-infra14:52
openstackgerritMerged openstack-infra/elastic-recheck: Add query for infra/pypi mirror bug 1355050
uvirtbotLaunchpad bug 1355050 in openstack-ci "markupsafe package not found" [Medium,Triaged]
*** xyang0 has joined #openstack-infra14:54
*** rfolco has joined #openstack-infra14:54
openstackgerritMerged openstack-infra/elastic-recheck: add query for bug 1408814
uvirtbotLaunchpad bug 1408814 in cinder "Image could not be found after upload. The image may have been deleted during the upload." [Undecided,New]
*** rickerc has quit IRC14:56
*** zxiiro has joined #openstack-infra14:58
*** rickerc has joined #openstack-infra14:59
*** rfolco has quit IRC14:59
*** dannywilson has quit IRC14:59
*** rfolco has joined #openstack-infra14:59
fungiclarkb: jeblair: mordred: if you're okay with 145876 and 145839 then i'll go ahead and do the paste db migration maintenance this afternoon or tomorrow15:00
*** annegentle has quit IRC15:01
*** achanda has joined #openstack-infra15:02
*** annegentle has joined #openstack-infra15:02
*** packet has joined #openstack-infra15:03
*** pradk has joined #openstack-infra15:07
mordredfungi: in 145876, you refer to server package but the patch is removing python-mysqldb which is the python module15:08
fungimordred: oh, you are correct... thanks15:09
fungimordred: lemme see if i can confirm what package puppet was complaining about15:09
*** banix has joined #openstack-infra15:09
*** chuckC_ has quit IRC15:10
mordredwell, I think it's gonna be a conflict with include mysql::pythin15:10
mordredfungi: so I'm likely just quibbling over the commit message - but I wanted to make sure15:10
mordredthe other one  looks great15:10
fungiyeah, i think the patch is correct and i need to fix the commit message, but double-checking it all now that i clearly was coding under the influence of cold meds when i wrote that :/15:11
openstackgerritMerged openstack-infra/project-config: Set up venv for zuul-swift-logs
fungii think include mysql::python is cargo culted actually and i can drop it and abandon 14587615:12
*** TheJulia has quit IRC15:13
mordredoh cool!15:13
*** reed has joined #openstack-infra15:14
*** reed has quit IRC15:14
*** reed has joined #openstack-infra15:14
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a local MySQL database to paste.o.o
openstackgerritJeremy Stanley proposed openstack-infra/system-config: MySQL database cleanup for paste.o.o
fungiadded one other minor cleanup while i was at it15:16
sdaguesalv-orlando: is that something the neutron team is on board with? I don't recognize the commit author, and it's not got a very clear message15:18
*** banix has quit IRC15:19
openstackgerritMerged openstack-infra/project-config: Publish yaml2ical to PyPI
*** esker has joined #openstack-infra15:20
*** bhunter71 has joined #openstack-infra15:22
*** mrunge has quit IRC15:22
openstackgerritMerged openstack-infra/project-config: Use large-ops-testing to test reducing the number of services
*** tonytan4ever has joined #openstack-infra15:24
openstackgerritRoman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge
*** shayneburgess has joined #openstack-infra15:25
*** bswartz has quit IRC15:28
mordredfungi: the gozer folks were doing a log review thing yesterday that's part of internal HP needs - as part of it, they ran a report on the gerrit db that I found interesting, so I adapted it a little and put it into the file override_report.sql in root's homedir on review.o.o15:31
mordredfungi: basically, it's a report of all of the times someone force-merged something bypassing the gate15:32
*** dmsimard_away is now known as dmsimard15:32
mordredit's kinda a fun output to look at15:32
fungimordred: oh! neat idea15:32
salv-orlandosdague: first time I see that patch. I don't think this is a move agreed with the rest of them but non necessarily a bad one. Can infra-team wait for input from me, mestery, markmcclain, or armax before merging?15:33
mordredyah - I _almost_ want to make it slightly better and publish it as an HTML report anyone can look at15:33
*** bswartz has joined #openstack-infra15:33
armaxsalv-orlando: what patch?15:33
mordred(it would be nicer if the corresponding comment on the force-merge vote was included too)15:33
*** ihrachyshka has quit IRC15:34
*** andreaf has quit IRC15:34
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for boto 2.35 auth failure bug 1408987
uvirtbotLaunchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,Confirmed]
mordredfungi: but I don't _actually_ want to spend the effort to do that :)15:35
*** vhoward has joined #openstack-infra15:36
fungiand yeah, a quick review of the results looks reasonably like what i would expect15:37
openstackgerritRandy Bertram proposed openstack/requirements: Add term.js to global requirements
fungiroughly matches the people, frequency and projects i think it should15:37
*** wenlock has joined #openstack-infra15:37
*** arielb has joined #openstack-infra15:38
*** arielb has quit IRC15:38
*** arielb has joined #openstack-infra15:38
fungifor example, sahara force-merges around the time it was renamed from savanna, et cetera15:38
*** teran has joined #openstack-infra15:38
fungirequirements and devstack force-merges when setuptools 8 had us horribly broken15:38
*** achanda_ has joined #openstack-infra15:41
*** achanda has quit IRC15:41
*** cpallares has joined #openstack-infra15:42
*** otter768 has joined #openstack-infra15:43
*** teran has quit IRC15:43
fungimordred: if you're in the mood to look at more puppet, 144367 is needed so that i can make another try on deploying storyboard-dev15:44
fungionce that's in place i'll take another stab at bringing it up with 14046615:45
*** doug-fish has joined #openstack-infra15:46
openstackgerritMerged openstack-infra/elastic-recheck: Add query for boto 2.35 auth failure bug 1408987
uvirtbotLaunchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,Confirmed]
mordredfungi: I do agree with tim's comment about the comment - but not so much that I think we shoudl block that patch15:46
fungimordred: yeah, i figured i'd tweak the commen to be more accurate if the change ended up needing another patchset15:47
*** marun has quit IRC15:47
*** otter768 has quit IRC15:47
*** pblaho_ is now known as pblaho15:48
jeblairfungi: i approved the add-local-db change15:49
openstackgerritMerged openstack-infra/system-config: Add a local MySQL database to paste.o.o
jeblairopenstackgerrit makes jeblair redundant15:50
fungijeblair: i saw--thanks. i'll get going on the migration prep once that's showing up on the server15:50
*** afazekas has quit IRC15:50
fungiheh, and that ;)15:50
openstackgerritMerged openstack-infra/project-config: Prepare project-config for puppet module split
fungihave you been replaced by a small shell^H^H^H^H^Hpython script?15:50
jeblairand i wrote it :(15:50
fungii think i saw that movie15:51
fungicome with me if you want to live15:51
*** sputnik13 has joined #openstack-infra15:51
*** nelsnelson has joined #openstack-infra15:53
mordredjeblair: oh! you're here today15:53
mordredjeblair: got a sec to talk about the exception hiding stuff in shade? the other comments are easy to either address or respond to in the review15:54
*** jokke_ has joined #openstack-infra15:54
*** bnemec is now known as beekneemech15:56
jeblairmordred: sure.  i'm sorta here.  packing and flying later.15:56
openstackgerritMerged openstack-infra/puppet-storyboard: Allow overriding the cert/key file paths
*** zz_dimtruck is now known as dimtruck15:57
*** achanda_ has quit IRC15:58
mordredjeblair: k - so - in general there's two ideas driving that pattern across the library15:58
mordredjeblair: one is that there are more than one place where shade may be using one of many different backend clients based on what it's detected about your cloud - so the amount of information you'd need to catch the right base exception is a bit extreme (could be glanceclient, could be novaclient, etc)15:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix puppet gate check jobs naming
mordredjeblair: the other is the eventually I want shade to use openstacksdk instead of python-*client - but for people to not have to rewrite consuming code16:00
mordred_however_ I hate exception hiding :)16:00
mordredso I've been trying to debug log whenever we do that - and maybe even should just attach the original exception object to the raised exception?16:01
*** dhritishikhar has joined #openstack-infra16:01
mordredjeblair: so do you think the combo of exception logging when they're masked and attaching the original exception object would be good enough to handle your concern?16:02
jeblairmordred: yeah, if you do it in such a way that when you repr() the ecxception, the original gets printed too, that could aid in debugging... but i think the issue i raised in line 687 of 145603 is still an issue16:02
mordredlooking at that one16:02
*** annegentle has quit IRC16:03
mordredjeblair: nod - I'll take a stab at a different approach to that issue16:03
*** dhritishikhar has quit IRC16:03
jeblairmordred: ok.  i think to generalize, it's two sides of the same coin -- you have to be careful not to mask debug info from users, and also when exceptions are used for fine-grained error reporting, not mask that from developers16:04
mordredyup. totally agree16:05
mordredjeblair: also - I tested minimal installs of ubuntu, fedora and centos and all had md5sum and figured it would be more efficient than hashlib - but I suppose if the task at hand is "upload this image to the cloud" perhaps the extra 20 seconds of md5 processing isn't particularly concerning?16:05
jeblairmordred: also, i do like your goals.  i think it's probably worth doing.  it just might (<sigh>) be a bit of work16:05
*** openstackgerrit has quit IRC16:05
*** openstackgerrit has joined #openstack-infra16:06
mordredyah. golly, if only there were a consistent way to deal with this already ...16:06
*** bcourt has quit IRC16:07
mordredjeblair: re: etag in swift - I had originally tested some code that added the md5 as a property to the swift object - I could just go back to doing that instead16:07
*** dhritishikhar has joined #openstack-infra16:07
jeblairmordred: the oracle of stackoverflow suggested it would take like 2 seconds16:07
mordredand that means I could use not-md516:07
*** TravT has left #openstack-infra16:07
*** bcourt has joined #openstack-infra16:07
*** che-arne has joined #openstack-infra16:07
mordredjeblair: wel, md5sum itself on a centos image created by dib takes more than 216:07
fungiyeah, read speed on the file will be the killer16:08
mordredit takes 29 seconds to calculate the md5 of centos.vhd with md5sum16:08
*** yaguang has quit IRC16:09
fungisince you have to read the entirety of it to generate a checksum16:09
jeblairmordred: i think we ought to try not-md5.  it's easy to say "oh this isn't crypto".  but who knows what might be built on this and where some attacker might change an image or something with an md5 collision and then this would not notice16:09
mordredjeblair: which one do you think we should use?16:09
jeblairthat would be embarassing16:09
jeblairi think sha256 is the cool kid now, right?16:09
fungihonestly, i expect just about any checksumming algorithm to be ~= timewise, because you're going to be i/o bound on the file16:10
*** otter768 has joined #openstack-infra16:10
fungiso, yeah, sha256 should be fine16:10
fungi(technically that's sha2-256)16:10
jeblairthat's the one16:10
clarkbyou need md5 anyways because thats what glance checks iirc16:10
mordredclarkb: this is all shade16:11
jeblairwhich is sad16:11
clarkbtalking to glance right?16:11
jeblairi think clarkb was referring to the upload integrity check?16:11
mordredthis is how shade knows whether to re-upload16:11
fungiglance will probably add a more modern checksum algorithm eventually--there is a bug open for that16:11
jeblairit will probably not cost us anything more to generate 2 hashes at the same time16:12
jeblairsince we are more likely io-bound than cpu bound on that16:12
*** Ala has quit IRC16:13
mordredif I'm doing it with hashlib, I'll absolutely do that16:13
mordredin fact, then I can check to see if the swift etag matches _and_ if the user-provided sha256 matches - because also it'll be the same cost to do so16:13
fungihrm, there's a good chance it's going to want to read the file twice to be able to do that16:13
mordredfungi: the hashlib approach is to read the file in 8k chunks16:14
fungi(to be able to generate two different checksums i mean)16:14
mordredfungi: and then update the hash16:14
*** patrickeast has joined #openstack-infra16:14
mordredso I should be able to update two different hashes in that loop16:14
*** mtanino has joined #openstack-infra16:14
fungimordred: it can reuse the same read to update two hash objects?16:14
fungiyeah, that should be fine then16:14
mordredshould be able to16:14
mordredI'll test real quick16:14
jeblairmordred: on my machine, md5 of a 950MB trusty iso takes 1.5s with the command line, and 1.5s with python @ 4096b chunks16:15
jeblair(with a warm io cache)16:16
*** coolsvap is now known as coolsvap|afk16:16
jeblairer wow16:17
jeblairsha256 command line takes 4.1 seconds, python takes 3.716:17
openstackgerritMarton Kiss proposed openstack-infra/system-config: Backup groups.o.o with bup
jeblairpython is faster? ^16:18
jeblairthat's pretty much additive.  doing both in python is 5.6s16:18
*** rfolco has quit IRC16:19
jeblair8192b chunks is slightly faster at 4.9s16:19
mordredjeblair: I was doing my md5 calc of the 1.6G centos image on a cloud server :)16:19
*** ssam2 has quit IRC16:20
jeblairmordred: yeah, not only might the cpu be slower, but you might not be able to fit it all in ram to keep the io cache warm16:20
*** annegentle has joined #openstack-infra16:20
*** ssam2 has joined #openstack-infra16:20
mordredjeblair: a single loop in python using 1M chunks can calculate both checksums in 1.5 minutes16:20
jeblairmordred: but that's an issue no matter how it's generated, so as long as you're consistent, comparing the two approaches should be fine16:20
*** xgerman has joined #openstack-infra16:20
jeblairmordred: much more than 4096 isn't going to get you much and is just going to waste ram16:21
*** e0ne has joined #openstack-infra16:21
jeblair8192 seems to be the sweet spot16:21
mordredjeblair: yah - I was doing 1M because that was in the copy-pasta16:21
mordredjeblair: re-doing with 819216:21
openstackgerritRoman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge
*** marcusvrn has quit IRC16:25
*** ildikov has quit IRC16:27
*** ddieterly has quit IRC16:27
mordredjeblair: same time/cost for 8k16:28
jeblairyeah, so i'd go with that so you're not eating ram16:29
*** bswartz has quit IRC16:30
*** marcusvrn has joined #openstack-infra16:30
*** Hal_ has quit IRC16:30
*** otter768 has quit IRC16:30
cody-somervilleHey folks. Were the sqlalchemy dependency updated to handle new pip on the stable branches as well or was a different strategy taken there?16:30
*** bswartz has joined #openstack-infra16:31
fungicody-somerville: we updated the sqlalchemy dependency strings in global-requirements.txt on all branches16:31
fungicody-somerville: though the exact strings used differed between stable and master16:31
*** bswartz has quit IRC16:32
*** bswartz has joined #openstack-infra16:33
*** annegentle has quit IRC16:33
*** ddieterly has joined #openstack-infra16:33
cody-somervillefungi: And to be clear, no attempts to pin pip to older version was used or done anywhere, right?16:33
fungicody-somerville: it was temporarily but we've been able to revert that everywhere now afaik16:34
*** bcourt has quit IRC16:34
fungier, we didn't pin pip but we did pin setuptools16:34
fungiand have since rolled that back now that things are working16:34
fungicody-somerville: the sqlalchemy requirements issue was already solved by the time pip updated. we went through all that to accommodate setuptools 8.016:35
*** andreykurilin has joined #openstack-infra16:35
fungiwhich led the pip/virtualenv update a couple weeks16:35
*** annegentle has joined #openstack-infra16:35
*** zz_avozza is now known as avozza16:36
*** stevemar has joined #openstack-infra16:36
*** Masahiro has joined #openstack-infra16:37
*** rfolco has joined #openstack-infra16:37
*** dansmith is now known as superdan16:37
sdaguecody-somerville: yeh, it mostly requires synchronizing all the stable trees for projects I think16:38
*** teran has joined #openstack-infra16:39
*** pcrews has joined #openstack-infra16:40
*** dimtruck is now known as zz_dimtruck16:41
*** rfolco has quit IRC16:41
*** Masahiro has quit IRC16:41
*** zz_dimtruck is now known as dimtruck16:42
*** dangers_away is now known as dangers16:42
*** hdd has joined #openstack-infra16:43
*** teran has quit IRC16:44
*** marun has joined #openstack-infra16:44
*** dannywilson has joined #openstack-infra16:47
*** bcourt has joined #openstack-infra16:48
*** annegent_ has joined #openstack-infra16:48
*** TheJulia has joined #openstack-infra16:49
*** banix has joined #openstack-infra16:49
*** markus_z has quit IRC16:49
*** dkranz has quit IRC16:49
*** marun has quit IRC16:50
*** amrith has joined #openstack-infra16:50
*** annegentle has quit IRC16:50
amrithjeblair, ping.16:50
*** dannywilson has quit IRC16:50
*** dannywilson has joined #openstack-infra16:51
*** vigneshvar has joined #openstack-infra16:51
amrithmordred, ping.16:52
openstackgerritKhai Do proposed openstack-infra/system-config: Set the Gerrit change screen default to OLD_UI.
*** mfink has quit IRC16:53
mordredamrith: sup?16:53
*** rwsu has joined #openstack-infra16:53
amrithmordred, thx. looking into an issue in trove.16:54
amrithbuilding guest image, I get "ImportError: No module named unittest2"16:54
amrithI saw some talk about this 12/16 on openstack-infra16:54
amrithbetwee you and jeblair16:54
*** Guest47811 has joined #openstack-infra16:54
amrithwas wondering what y'all may have found16:54
amrithand how you got around it16:54
amrithI'm getting it on an install of oslo.concurrency16:54
amrithwhich requires16:54
amrithtesttools -> unittest216:54
jeblairamrith: i think the solution is to install a new version of virtualenv, which will have a new setuptools and then it should work16:55
amrith approx 2014-12-16T22:34:5116:55
*** Guest47811 is now known as rackertom16:55
mordredamrith: yes, what jeblair said16:55
*** dannywilson has quit IRC16:55
amrithwe currently have pip install -U 'setuptools<8.0'16:56
amrith pip install virtualenv16:56
amrithwhat's the preference on setuptools version?16:56
*** avozza is now known as zz_avozza16:56
*** annegentle has joined #openstack-infra16:57
*** yamahata has joined #openstack-infra16:57
mordredamrith: virtualenv bundles setuptools - do you need a new version of virtualenv16:57
mordredso do a pip install -U virtualenv16:58
mordredyou also should not be pinning setuptools to less than 8.016:58
mordredamrith: where is that?16:58
mordredamrith: _definitely_ remove the <8.0 from your pip install -U setuptools16:58
*** sdake has quit IRC16:58
amritham looking for that16:58
amrithI see it in my log16:59
amrithtrying to see where it comes from16:59
amrithit is in generating a guest image for trove16:59
*** fthigv has joined #openstack-infra16:59
*** sdake has joined #openstack-infra16:59
mordredamrith: is this driven by devstack?16:59
*** sdake has quit IRC16:59
*** sdake has joined #openstack-infra16:59
amrithyes, this part would be17:00
mordredbecause there _was_ a period where we were pinning setuptols in devstack17:00
amrithlooking, one second17:00
mordredso you could have a _very_ out of date devstack :)17:00
mordredmornign zaro17:00
*** annegent_ has quit IRC17:00
*** marcusvrn has quit IRC17:01
*** marcusvrn1 has joined #openstack-infra17:01
clarkbits also redstack and dib17:01
amrithmy devstack isn't all that old17:01
yjiang5fungi: Hi, I have a question to the gerrit stream event. I have a local mirror, which will pull the repo  from immediately after get the change-merged event. However, most of the time, it will get result of "Already up-to-date". So want to know if the event is sent out before or after the corresponding change are pushed to the openstack/github mirror?17:01
amrithand I'm building image through redstack17:01
amrithwhich invokes devstack for this part17:01
sdagueredstack ?17:02
*** dkranz has joined #openstack-infra17:03
*** jlibosva has quit IRC17:03
clarkbsdague fork thing of devstack from the reddwarf days17:03
sdagueah, gotcha17:03
*** TravT has joined #openstack-infra17:03
amrithconvenience for development in my sandbox17:03
*** EmilienM is now known as EmilienM|afk17:04
amrithquick and dirty way for me to build a dev environment17:04
*** kgiusti has quit IRC17:04
clarkbyjiang5 I think they happen concurrently17:04
clarkbyjiang5 unfortunately we need a newer gerrit to get replication events but zaro is testing that upgrade now17:05
*** jgrimm is now known as zz_jgrimm17:05
*** kgiusti has joined #openstack-infra17:06
*** alugovoi has joined #openstack-infra17:06
yjiang5clarkb: ok, so it means highly possible my git will be one changeset delay, till the replication event is done, right?17:06
*** sabeen1 has joined #openstack-infra17:06
clarkbyjiang5 yes17:06
*** ivar-lazzaro has joined #openstack-infra17:07
yjiang5clarkb: thanks for your help a lot. Currently I will workaround this issue by sleep 10 seconds before pull the git. Hope 10 seconds will be enough.17:07
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceilometer oslo.utils bug 1408737
uvirtbotLaunchpad bug 1408737 in ceilometer "broken unit tests after oslo.utils update" [High,In progress]
amrithmordred, jeblair ... brand new devstack and still the same., I'll try and figure out where this version pinning is coming from. thx17:09
*** Ryan_Lane has joined #openstack-infra17:11
*** dimtruck is now known as zz_dimtruck17:11
amrithmordred, jeblair looks like it comes from tripleo-image-elements/elements/pip-and-virtualenv/install.d/get-pip-py-source-install/01-install-pip17:13
*** shashankhegde has joined #openstack-infra17:14
*** bswartz1 has joined #openstack-infra17:14
*** bswartz has quit IRC17:16
*** gmurphy has left #openstack-infra17:16
*** beekneemech has quit IRC17:17
*** bnemec has joined #openstack-infra17:18
*** marcusvrn1 has quit IRC17:18
*** zz_dimtruck is now known as dimtruck17:19
*** cpallares has quit IRC17:19
*** garyh has joined #openstack-infra17:19
zaroclarkb: i'm wondering if the replication issue we've been having is from setting replicationDelay to too low of value?17:20
mordredamrith: ah - then that's almost certainly a bug in t-i-e17:20
*** MaxV has quit IRC17:20
mordredSpamapS, greghaynes: ^^17:20
*** otter768 has joined #openstack-infra17:20
zaroclarkb: read  lockErrorMaxRetries
amrithmordred, what version of setuptools are you expecting?17:20
amrithseems like it is endinng up with 7.017:20
amrithwhich is now up to 11.3 ish ;)17:21
zaroclarkb: the default is 15 seconds.17:21
mordredamrith: we expect latest setuptools17:22
*** dims__ has quit IRC17:22
amrithok, thanks!17:22
*** achanda has joined #openstack-infra17:22
mordredgreghaynes: you're pinning setuptools it seems in t-i-e?17:22
*** dims__ has joined #openstack-infra17:22
mordredamrith: ^^ there's your fix17:23
amrithmordred, thanks!17:23
clarkbzaro we can probably bump that value. should dig through git logs to see if they tell us why its set low though17:23
*** marcusvrn has joined #openstack-infra17:23
*** ldnunes has quit IRC17:24
*** ZZelle is now known as ZZelle|away17:24
*** ldnunes has joined #openstack-infra17:24
zaroclarkb: i'll take a look17:25
*** melwitt has joined #openstack-infra17:26
openstackgerritMerged openstack-infra/elastic-recheck: Add query for ceilometer oslo.utils bug 1408737
uvirtbotLaunchpad bug 1408737 in ceilometer "broken unit tests after oslo.utils update" [High,In progress]
*** dims__ has quit IRC17:27
*** dizquierdo has quit IRC17:28
*** jedimike has quit IRC17:28
*** arxcruz has quit IRC17:28
jeblairzaro, clarkb: i think we should have the delay to be low (eg, current value of 1 second), and retries to be higher than the default of 0, (eg, suggestion of 3)17:28
jeblairzaro, clarkb: is that what you are discussing?  or are you proposing increasing the delay?17:29
*** bhuvan_ has joined #openstack-infra17:29
*** bhuvan_ has joined #openstack-infra17:29
clarkbjeblair possibly both? not sure how long these locks will be contended for17:30
*** gyee has joined #openstack-infra17:30
*** e0ne is now known as e0ne_17:30
*** jistr_ has joined #openstack-infra17:31
*** dtantsur_ has joined #openstack-infra17:31
jeblairi want our git.o.o mirrors to be useful and fast, so i want to keep the delay low.  i don't mind the extra traffic.17:31
*** pblaho_ has joined #openstack-infra17:31
*** jcoufal has joined #openstack-infra17:32
*** ryanpetrello has quit IRC17:33
*** signed8bit has quit IRC17:33
*** jcoufal_ has quit IRC17:33
*** palar has joined #openstack-infra17:34
*** jcoufal_ has joined #openstack-infra17:34
*** dtantsur has quit IRC17:34
*** pblaho has quit IRC17:34
*** jaypipes is now known as leakypipes17:34
*** jistr has quit IRC17:35
*** pblaho__ has joined #openstack-infra17:37
*** pblaho__ is now known as pblah17:37
*** pblaho_ has quit IRC17:37
*** pblah is now known as pblaho17:37
*** jcoufal has quit IRC17:37
*** pblaho has quit IRC17:37
*** dtantsur_ has quit IRC17:37
*** jistr_ has quit IRC17:37
*** pblaho has joined #openstack-infra17:37
*** Ryan_Lane has quit IRC17:38
*** hdd has quit IRC17:39
*** otter768 has quit IRC17:39
zaroisn't setting replicationDelay low and retries high opposite to what the documentation recommends?17:39
*** aysyd has quit IRC17:40
zarojeblair: ohh actually i got mixed up.  i think your suggestion is reasonable.17:41
*** Longgeek has quit IRC17:42
*** jcoufal_ has quit IRC17:42
*** yamahata has quit IRC17:42
*** dtantsur has joined #openstack-infra17:42
*** hdd has joined #openstack-infra17:42
*** aysyd has joined #openstack-infra17:42
*** Longgeek has joined #openstack-infra17:44
*** andreykurilin has quit IRC17:44
*** chianingwang has joined #openstack-infra17:44
*** andreykurilin has joined #openstack-infra17:45
*** otter768 has joined #openstack-infra17:45
*** tkelsey has quit IRC17:45
*** ryanpetrello has joined #openstack-infra17:45
*** sputnik13 has quit IRC17:47
*** bcourt has quit IRC17:48
*** hashar has quit IRC17:48
*** dims__ has joined #openstack-infra17:49
*** e0ne_ is now known as e0ne17:50
*** dimtruck is now known as zz_dimtruck17:51
amrithjeblair, mordred ... I cherry-picked that change and re-ran redstack, failed in the same way. I now have setuptools (latest). will investigate17:51
*** derekh has quit IRC17:51
*** dannywilson has joined #openstack-infra17:51
zaroclarkb, jeblair : opps, i referenced the wrong doc.  ver 2.8.4 doesn't have the e17:51
*** dannywilson has quit IRC17:51
zarolockErrorMaxRetries option.
*** zz_dimtruck is now known as dimtruck17:52
*** dannywilson has joined #openstack-infra17:52
*** _shaps_ has quit IRC17:53
*** Longgeek has quit IRC17:53
*** dims__ is now known as dimsum__17:53
*** amitgandhinz has quit IRC17:53
*** sputnik13 has joined #openstack-infra17:54
*** Longgeek has joined #openstack-infra17:54
*** otter768 has quit IRC17:55
*** shashankhegde has quit IRC17:57
mordredamrith: did you also update virtualenv to latest?17:57
mordredamrith: or are you using virtualenv at all?17:57
reedare there already plans to remove this page and use infra manual instead?
amrithmordred, I'm using virtualenv but I'm using redstack to build a new image. so in creating a new image it gets the latest virtualenv also17:57
amrithwill get you versions in a second17:58
mordredamrith: ok. can you verify that it's latest virtualenv? cool17:58
openstackgerritKhai Do proposed openstack-infra/system-config: update replication setting for review-dev.o.o match review.o.o
*** Longgeek has quit IRC17:59
*** hdd has quit IRC17:59
*** bcourt has joined #openstack-infra18:00
krotscheckjeblair: Comments on yolanda’s core promotion have closed, and we’ve got no concerns. What’s the next step?18:00
*** Longgeek has joined #openstack-infra18:00
fungithrow a party?18:01
yolandafungi, sure! :)18:01
krotscheckParty at fungi’s place!18:02
*** Longgeek has quit IRC18:02
fungii think someone just adds her to the core group for that project and jeblair follows up to the nomination thread with a confirmation (or delegates it)18:02
krotscheckThat’s a projects-config thing?18:03
fungikrotscheck: i recommend coming in by seaplane18:03
*** e0ne has quit IRC18:03
*** aysyd has quit IRC18:03
*** AJaeger has joined #openstack-infra18:03
*** Longgeek has joined #openstack-infra18:04
*** Longgeek has quit IRC18:04
fungikrotscheck: it happens by editing,members though right now it looks like that's not a self-managed group (owned by administrators). not sure if that was intentional18:04
*** annegentle has quit IRC18:05
krotscheckfungi: hrm. Yeah, I can’t edit that.18:05
*** mfink has joined #openstack-infra18:05
pleia2haha, woke up congested, my husband goes "you look and sound horrible" - such a charmer :) and I might have a cold18:05
*** aysyd has joined #openstack-infra18:05
*** Longgeek has joined #openstack-infra18:05
*** tonytan4ever has quit IRC18:07
*** tonytan4ever has joined #openstack-infra18:08
AJaegerHi, I just noticed that logfiles are not complete. Is that a known issue with our swift logs?18:10
*** dtantsur is now known as dtantsur|afk18:10
fungiAJaeger: yeah, it seems there may be a race with log uploading where jenkins hasn't finished recording the console output. there was some discussion yesterday on ways to mitigate18:11
AJaegerfungi: thanks18:11
fungibest contender was to insert a canary at the end of the job output and then keep polling (with some reasonable timeout) until that pattern is found18:12
AJaegerthat sounds ugly ;(18:12
*** tonytan4ever has quit IRC18:12
fungiwelcome to jenkins integration!18:12
*** tqtran has joined #openstack-infra18:12
*** achanda has quit IRC18:13
*** chianingwang has quit IRC18:13
*** ivar-lazzaro has quit IRC18:14
tqtranHello, can i bug someone to take a look at We're trying to move along some of the 3rd party JS dependencies in horizon. Any help on this is greatly appreciated.18:14
dkranzfungi: Is supposed to fix the gate problem with tempest? Cause it looks like stable branches still fail there.18:14
fungisdague: ^ ?18:15
*** kgiusti has quit IRC18:16
fungidkranz: i _think_ it needs a similar patch in nova stable branches18:16
*** jpich has quit IRC18:16
*** kgiusti has joined #openstack-infra18:16
funginot actually following that issue very closely today18:16
fungiso i coule be way off base18:16
fungidkranz: or maybe just requirements stable branches will be enough18:17
clarkbfungi that sounds right. need to roll up a lot of these fixes with grenade and branchless tempest18:18
fungiaha, the juno patch hasn't landed for that yet...,n,z18:19
dkranzclarkb: that looks like but check-requirements-integration-dsvm is failing18:19
dkranzclarkb: maybe this from the log: 'oslo.context' is not in global-requirements.txt18:20
fungi "'oslo.context' is not in global-requirements.txt"18:20
fungilooks like oslo.log is in juno requirements and has added a requirement for oslo.context which is not in juno global requirements18:21
clarkbits all documented in the fix for the fix for the fix :)18:21
*** marcusvrn has quit IRC18:22
*** jp_at_hp has quit IRC18:23
dkranzfungi, clarkb : So the "cherry pick" was actually no18:24
fungii take it oslo.log is lacking requirements checks18:24
*** ssam2 has quit IRC18:24
openstackgerritMerged openstack/requirements: Use oslo.utils without namespace package
fungino, wait, that would be a lack of stable backward-compat checks for juno i guess18:24
fungithough i suppose we don't do backward compatibility requirements checks to catch this18:25
*** Masahiro has joined #openstack-infra18:25
openstackgerritMerged openstack/requirements: provide more meaningful output on
fungiso, based on a cursory time trial, it looks like i can accomplish the paste db migration in ~2 minutes if all goes smoothly, and can cut back in under a minute if it fails for some reason18:28
*** Masahiro has quit IRC18:30
openstackgerritKhai Do proposed openstack-infra/project-config: Update zuul layout to match production
fungithe plan is: 1. disable puppet agent, 2. remove the trove hostname from hiera, 3. stop apache, 4. mysqldump the trove db, 5. read the dump into the local db, 6. reenable puppet agent and apply, 7. start apache, 8. test18:31
*** jcoufal has joined #openstack-infra18:31
fungiassuming the test fails: 9. replace the trove hostname in hiera, 10. update puppet again18:32
clarkbwhen does point at localhost happen?18:32
fungiteh global site manifest provides localhost as a fallback value if the db hostname key is not present in hiera18:32
*** zz_sabari is now known as sabari18:33
fungithat's what that does, right? or am i misunderstanding how that works?18:33
*** ivar-lazzaro has joined #openstack-infra18:33
dkranzclarkb, fungi: is any one working on unblocking the gate? I would help but not sure what exactly to do.18:34
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Explicit Event API
openstackgerritMerged openstack/requirements: update test-requirements to current
sdaguedkranz: it's possible that stable/juno is wedged on 2 things needing to happen at once18:34
fungidkranz: sdague had a thread about it on the -dev ml, but not sure if he's still actively poking the fixes18:34
fungioh, there's an sdague18:34
sdagueyeh, sorry, was just scraping snow off the driveway18:35
dkranzsdague: :)18:35
openstackgerritJoe Gordon proposed openstack/requirements: Update to latest hacking (0.10.0)
sdaguethe master and icehouse fixes look like they are in18:35
clarkbfungi ya that is how the hiera lookup should work18:35
sdagueI haven't been following the juno fixes, one of the stable maint folks hopefully will18:36
*** jogo is now known as flashgordon18:37
*** amitgandhinz has joined #openstack-infra18:37
fungisdague: looks like it's presently deadlocked on the fact that oslo.log is in stable/juno projects.txt and has added a requirement for oslo.context which is not in stable/juno global-requirements.txt18:37
fungior something along those lines18:38
fungiadam_g: ^18:38
*** tonytan4ever has joined #openstack-infra18:38
fungiflaper87: ^18:39
*** marcusvrn has joined #openstack-infra18:39
*** Ryan_Lane has joined #openstack-infra18:40
*** tonytan4ever has quit IRC18:40
*** mpaolino has quit IRC18:40
*** tonytan4ever has joined #openstack-infra18:41
amrithmordred, I'm using  virtualenv-12.0.5 (latest).18:42
amrithmordred, ^^18:42
*** dkranz has quit IRC18:44
fungiso anyway, given that i can migrate paste in such a short window, i'm tempted to just go ahead and do it, not bother scheduling/announcing anything18:46
adam_gsdague, fungi, will take a look shortly18:46
fungior i can make openstackstatus announce it for me via notice to the irc channels and not bother with e-mail or alert/ok18:47
fungithat seems like a sane compromise18:47
clarkbfungi ya I think considering it fails enough on its own a short window like that is fine18:47
*** dustins has quit IRC18:51
*** bnemec is now known as beekneemech18:51
*** harlowja_away is now known as harlowja18:52
*** pblaho has quit IRC18:54
*** johnthetubaguy is now known as zz_johnthetubagu18:57
fungi#status notice is going offline for a database migration (duration: ~2 minutes)18:57
openstackstatusfungi: sending notice18:57
-openstackstatus- NOTICE: is going offline for a database migration (duration: ~2 minutes)18:58
*** dannywilson has quit IRC18:59
openstackstatusfungi: finished sending notice18:59
*** shashankhegde has joined #openstack-infra18:59
fungiit's migrated and back online now19:00
fungi is a paste i saved right before i stopped apache19:00
*** annegentle has joined #openstack-infra19:00
*** dannywilson has joined #openstack-infra19:01
mordredfungi: it seems to be working19:01
fungiyep, and i saved 156125 right after apache came back up. the trove db has 156124 but not 156125 while the local mysql db has both19:02
fungiso looks like it was successful19:02
*** alexpilotti has quit IRC19:03
*** yamahata has joined #openstack-infra19:05
*** e0ne has joined #openstack-infra19:06
*** dustins has joined #openstack-infra19:07
*** ivar-lazzaro has quit IRC19:07
*** dalgaaf has quit IRC19:07
*** sabari is now known as zz_sabari19:07
fungiclarkb: oh, not sure if you saw 145978 last night, my next installment in the "you can't sudo here" saga19:08
*** xyang0 has quit IRC19:08
adam_gfungi, +  not certain whether the latter is needed but seems to make sense19:09
*** imcsk8 has quit IRC19:09
*** imcsk8 has joined #openstack-infra19:10
openstackgerritMichael Krotscheck proposed openstack-infra/infra-specs: StoryBoard Streaming API specification
fungiadam_g: you'll likely need to switch the order on them to get them to pass19:10
krotscheckfungi: Looks like your recheck on passed.19:11
*** kgiusti has quit IRC19:11
fungiadam_g: i think the requirements integration job for juno is broken until 146180 is in place, so 146179 won't succeed19:11
fungikrotscheck: yep, i'm about to switch gears to storyboard-dev testing now that the paste migration is wrapped up19:11
adam_gfungi, ah, right. ill break em up19:11
krotscheckfungi: Thanks :D19:11
zarocan someone help me debug zuul?19:11
zaroit doesn't seem like zuul is sending jobs to gearman19:12
* krotscheck is still doing gerrit things. Will hopefully be done by lunch19:12
*** jcoufal has quit IRC19:12
zarofungi: help with zuul?19:12
fungizaro: does the debug log indicate it's initiating jobs?19:13
*** jcoufal has joined #openstack-infra19:13
*** signed8bit has joined #openstack-infra19:13
zarofungi: i see this in the zuul log, "2015-01-09 19:09:08,492 INFO zuul.Gerrit: Updating information for 5306,6"19:13
zarothat's it.19:13
fungizaro: oh, actually an easy cause is that you might have forgotten to start the zuul-merger daemon?19:13
fungizuul won't submit any jobs until the necessary merges for them complete19:14
zarofungi: nope zuul merger service is started and running. however i don't see anything in the merger logs19:14
fungizaro: and it's configured to use the built-in gearman service? have you checked the gearman status output?19:15
*** fthigv has quit IRC19:15
*** vponomaryov has quit IRC19:15
zaroi'm not sure how the zuul merger is configured.  but i'm looking at the gearman status and i never see the job getting added to the gearman queue.19:16
zaroso maybe zuul merger is where i should look?19:16
zarothis is on review-dev, so maybe merger didn't get setup?19:16
fungithat's possible... check the zuul.conf merger section19:17
*** AJaeger has quit IRC19:17
*** arielb has quit IRC19:17
fungimaybe this is a newish version of zuul with an out of date config19:17
zarozuul version is
ekarlso-anyone tell me what is wrong here ?
*** yamahata has quit IRC19:18
fungiekarlso-: the log got truncated uploading to swift. i'll see if i can find the full output19:19
zarofungi: well there is a merger section, so looks like the configuration is there and it seems good to me.19:20
*** mrunge has joined #openstack-infra19:21
fungiekarlso-: i wound up rechecking it to see if we can get a full log from the next run. the log was a day old so not likely possible to dig out of whichever jenkins master it ran on at this point19:21
fungizaro: and you see a zuul-merger process in the process table and a merger worker available in the gearman status?19:22
zarofungi: actually the setting zuul_url= seems incorrect.  should the 'p' be there at the end?19:22
zarofungi: yes, i see merger:merge and merger:update19:23
fungioh, when you said review-dev you meant zuul-dev? i thought you just had some personal zuul server connected to review-dev. i'll take a look on zuul-dev myself19:23
*** yamahata has joined #openstack-infra19:23
zaroyes, zuul-dev19:24
*** EmilienM|afk is now known as EmilienM19:24
fungiyep, i agree there's one merger worker showing available19:25
fungizaro: what job are you expecting to trigger?19:25
zarofungi: gtest19:27
*** hdd has joined #openstack-infra19:27
clarkbfungi: I had not. looking now19:27
*** Longgeek has quit IRC19:27
*** ddieterly has quit IRC19:27
fungizaro: yeah, so your change is got gtest-org/gtest which is in the projects list in layout.yaml and has noop pseudo-jobs defined for all pipelines19:28
zarofungi: ohh actually i think zuul-dev is just out of date.  looking at site.pp i see it's probably missing a few thingys.19:28
*** dustins has quit IRC19:28
*** Longgeek has joined #openstack-infra19:28
fungizaro: aha, probably a good avenue to check down, yes19:29
*** ddieterly has joined #openstack-infra19:29
pleia2should the dev servers .pp files typically be kept up to date with the prod ones? (it's something I've thought about when reviewing prod changes, but never did more than think :))19:30
zarofungi: thanks for your time.  i shore up zuul-dev and see if i can get it to work.19:30
*** dustins has joined #openstack-infra19:30
*** dkranz has joined #openstack-infra19:32
*** zz_sabari is now known as sabari19:32
fungizaro: you bet--let me know if you hit another dead end19:32
*** bswartz1 has quit IRC19:32
fungikrotscheck: i have a new storyboard-dev building now, based on 140466 applied to a dev env. should know if it's functional shortly19:32
*** Longgeek has quit IRC19:33
krotscheckfungi: Great.19:33
*** sabari is now known as zz_sabari19:33
*** wuhg has quit IRC19:34
*** pelix has quit IRC19:34
fungipleia2: generally would be a good idea (with some notable exceptions like when we leave out some support intentionally, or are testing some specific divergence on the dev server)19:34
*** vigneshvar has quit IRC19:34
* pleia2 nods19:34
clarkbdib builds are currently not working on new nodepool so I am looking into that19:35
clarkbhopefully can get hpcloud using dib images soon19:35
*** marcusvrn has quit IRC19:35
openstackgerritMerged openstack-infra/project-config: Broaden pattern match for
fungikrotscheck: it's 2001:4800:7818:104:be76:4eff:fe04:cc25
fungikrotscheck: built with no obvious errors19:37
*** zz_sabari is now known as sabari19:38
krotscheckfungi: Checking19:38
fungiself-signed cert and redirect from http->https seem to work as designed19:39
krotscheckfungi: Indeed. Just logged in without an issue.19:39
krotscheckAnd it autogranted me admin. Nice.19:39
fungiit knows what's good for it19:40
krotscheckI’m a little concerned that this will make the draft build no longer a seamless experience.19:40
clarkbmordred: you still around? it almost looks like we are leaking stuff into the cache via nodepool/elements/cache-devstack/extra-data.d/50-early-source-repo19:40
krotscheckOriginally it was: Click on link, it works!19:40
clarkbmordred: I will probably end up removing the cahce but maybe you want to look first?19:40
fungianyway, i think this is working enough that we can go ahead and approve 140466 if we can get a second pair of eye on it19:40
*** sabari is now known as zz_sabari19:41
fungikrotscheck: well, the self-signed cert is something you'll only need to accept once19:41
krotscheckNow, it’s going to be click on link, watch-storyboard-stutter-because-you-don’t-know-that-you-have-to-authorize-snakeoil.19:41
clarkbfungi: sure I can take a look19:41
*** bhunter71 has quit IRC19:41
*** zz_sabari is now known as sabari19:41
clarkbfungi: any cahnce you want to look at dib failures on nodepool before I clean out its caceh?19:41
fungiclarkb: any chance there's something there which isn't obvious from the image update log?19:42
krotscheckfungi: True, but it won’t be obvious for users using the draft build, since it’s sitting on a domain with a signed cert.19:42
clarkbfungi: the error is 'Permission denied' trying to access a thing in the cache19:42
fungiclarkb: i would probably just try to fire off a new image build and see where it dies, honestly19:42
fungiclarkb: oh19:42
fungitoo fun19:42
clarkbbut the cache path looks fine permissions too19:42
clarkbhowever that repo cache is dirty with all these manifest files that come out of our elements19:43
clarkbkrotscheck: fungi  in 140466 I think you need to update storyboard.o.o's node {} block to tell it where to get its cert files?19:44
clarkbwe default to undef in that cahgne but don't update storyboard.o.o19:44
*** kgiusti has joined #openstack-infra19:44
clarkbor am I missing something with how that fits together?19:44
*** TravT has quit IRC19:45
fungiclarkb: in puppet-storyboard if the content is non-empty but the filename is undef then it has a fallback to a hard-coded default filename19:46
fungiexcept for the chain file19:46
anteayahas anyone seen mordred around?19:47
fungier, no, even for the chain file, the logic around what to do with that if the content is undef is just a little more convoluted19:47
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Update query for ec2 bug 1408987
uvirtbotLaunchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,In progress]
fungianteaya: yep, he's been in and out today19:47
anteayaI am stuck in australia they won't let me transfere to my nz flight19:47
fungioh, that's no good19:47
clarkbanteaya: he was around earlier today but hasn't popped in for a bit19:47
anteayaI don't go throough the round machine19:48
anteayaso I can not go to nz19:48
anteayabut I don't know if I can get home19:48
clarkbfungi: thanks, I will approve shortly19:48
clarkbas soon as I double check that for my own sanity19:48
anteayaI tried texting him19:48
fungiclarkb: i'm trying to double-check that myswlf as well19:48
*** cdent has quit IRC19:49
fungiclarkb: it looks like you may be right, if we pass in an explicit undef19:49
clarkbanteaya: they don't have an opt out option in AU? /me is trying to remember what perth was like19:49
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Update query for ec2 bug 1408987
uvirtbotLaunchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,In progress]
anteayaclarkb: I'm told they don't19:50
anteayait didn't come up in perth or brisbane19:50
anteayaperhaps just sydney has a round machine19:50
anteayaI don't know19:51
*** mrunge has quit IRC19:51
*** sputnik13 has quit IRC19:51
clarkbfungi: ya I think the issue there is we are overriding the default19:52
clarkbfungi: because an explicit undef overrides19:52
*** palar has quit IRC19:52
*** vigneshvar has joined #openstack-infra19:53
fungianteaya: clarkb: looks like they changed it a couple years ago to be non-opt-out
*** dprince has quit IRC19:53
clarkbcrinkle: nibalizer: am I wrong about the default behavior for default params there19:54
mordredanteaya: hey! yikes ...19:54
*** e0ne has quit IRC19:54
mordredanteaya: I don't know much about what the options are there ... any chance you've tried pinging mikal or lifeless? they both know many things19:54
mordredalso - let me poke stewart19:55
anteayasitting and waiting19:55
mordredanteaya: oh - it's 7am there - I probably shouldn't text stewart ...19:55
mordredmaybe lifeless is awake though19:55
*** otter768 has joined #openstack-infra19:56
anteayathey are trying to understand my reasons19:56
anteayathey laws don't allow a person to make choices about their own health apparently19:56
openstackgerritMerged openstack-infra/elastic-recheck: Update query for ec2 bug 1408987
uvirtbotLaunchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,In progress]
*** TravT has joined #openstack-infra19:59
*** TravT_ has joined #openstack-infra19:59
*** TravT has quit IRC20:01
*** otter768 has quit IRC20:01
*** bhunter71 has joined #openstack-infra20:05
clarkbmordred: did you see the thing about the dib cache having trouble on nodepool.o.o?20:06
clarkbmordred: wanted to give you a chance to debug before I rm -rf source_repositories and make it start over20:06
openstackgerritJoe Gordon proposed openstack-infra/project-config: Reduce number of services running in large-ops
*** marun has joined #openstack-infra20:07
mordredclarkb: I did not20:07
mordredclarkb: /me reads scrollback20:07 error: cannot open /opt/dib_cache/source-repositories/ci_sandbox_6a5dd9d33a0c8ef15cf705206fa0d8765f0da322/.git/FETCH_HEAD: Permission denied20:08
fungiclarkb: i don't suppose we're running into a max nodes condition there? (~104k inodes in that one directory level)20:08
*** annegentle has quit IRC20:08
clarkbfungi: ya would that manifest as permission denied?20:08
mordredclarkb: when you say "leaking stuff into the cache" - what do you mean20:08
fungioh, this is in dib_cache not dib_tmp20:08
flashgordonsdague: good news, the large-ops change worked20:08
clarkbmordred: in /opt/dib_cache/source-repositories we have a bunch of dib git manifest files20:09
*** andreykurilin has quit IRC20:09
clarkbmordred: so the git caching thing you have that does the extradata dib magic isn't cleaning up after itself20:09
fungiclarkb: .git is root-owned there... was it trying to fetch as root?20:09
*** andreykurilin has joined #openstack-infra20:09
mordredrm -f $GIT_MANIFEST20:09
clarkbfungi: hrm it shouldn't but maybe something did?20:09
mattoliverauMorning all20:10
clarkbmordred: I think the problem is we abort before that point due to the permissions issue20:10
*** signed8b_ has joined #openstack-infra20:10
*** dhritishikhar has quit IRC20:10
fungimattoliverau: have any suggestions for anteaya's au airport body scanner dilemma?20:10
clarkbfungi: mordred: I think the simplest thing now is probably to nuke source-repositories and let it rebuild and see if we can narrow down if it happens again20:10
mordredI'm ok with that20:11
zarofungi: did a detailed check of zuul-dev configuration, seems ok.  i thought it needed to connect to a zuul merger server but it looks like zuul_dev.pp installs the merger on localhost.20:11
anteayamight have to go through it today to avoid prison20:11
clarkbfungi: anything else you want to check before I do that?20:11
*** signed8bit has quit IRC20:11
anteayasince I don't have an au visa20:11
fungiclarkb: yeah, the only other thing i can think of is if we had parallel threads trying to access that (for a while the source repos stuff was not concurrency-safe and would break if we ran multiple dib image creations concurrently)20:12
anteayabut I will be getting a medical certificate in future20:12
*** annegentle has joined #openstack-infra20:12
anteayasince if I have one that says I can't go throught the scanner I don't have to go through20:12
zarofungi: it seems like zuul picks up gerrit changes because i see it in the zuul.log -> "Updating information for 5306"20:12
clarkbfungi: mordred also clean up dib_tmp while I am at it20:13
clarkbI will go ahead and start this20:13
zarofungi: i'm not seeing any updates in the merger debug logs.20:13
fungizaro: well, since it has no jobs defined for that change's project it's not actually going to do any work requests (aside from the merge i think)20:13
mikalanteaya: so yeah, if you're talking about the full body scans I don't think they're optional20:13
*** aysyd has quit IRC20:13
lifelessanteaya: hi20:14
*** Masahiro has joined #openstack-infra20:14
lifelessthe US has mitigating things like the constitution for some of the crazy.. once it escapes into the wider world, it usually doesn't.20:14
mikalQuote from the FAQ:20:14
*** SotK has quit IRC20:14
mikal"If a person refuses to undergo a body scan, and they have no medical or physical condition which prevents them for undertaking a body scan, they will be refused clearance and not allowed to pass through the screening point. They therefore will not be allowed to board their aircraft. This policy not only applies to passengers, but also to pilots, other aircrew, government officers and airport staff."20:14
mattoliverauanteaya: what, that's annoying, I thought you could refuse and they'd find some other way. Damn Sydney.. I apologise for my dumb country :(20:14
*** ldnunes has quit IRC20:15
fungiclarkb: okay20:15
anteayalifeless: hi20:15
mattoliverauWell there you go then20:15
mikalAhhh, interesting. We have the same ban as the EU on x-ray scanners, but this isn't an x-ray machine apparently.20:15
*** SotK has joined #openstack-infra20:16
anteayayeah that is what they are telling me20:16
anteayamikal: that is there point20:16
anteayawith no scientific studies to back it up20:16
*** shashankhegde has quit IRC20:16
*** bswartz has joined #openstack-infra20:16
*** sputnik13 has joined #openstack-infra20:17
mikalanteaya: yeah, I get your objection (I have similar concerns) but I don't see any provision in these rules for the AU govt to let you opt out20:17
zarofungi: argg! i completely missed that.20:18
anteayamikal: medical cert20:18
timrcSomewhere Bruce Schneier is sighing.20:18
anteayawhich I don't have but am going to get20:18
fungizaro: well, i'd still expect zuul to update the change with a passing comment or something20:18
*** andreaf has joined #openstack-infra20:18
*** annegentle has quit IRC20:19
fungizaro: it's configured to operate on the change in that pipeline, just not configured to run any jobs for it20:19
clarkbok those dirs are cleaned up, now we wait for nodepool to start the next image build20:19
*** Masahiro has quit IRC20:19
fungiclarkb: is it possible the error they failed on was the source repos caching concurrency issue?20:20
zarodo you need to restart zuul to reload layout.yaml?20:20
clarkbfungi: maybe? I wouldn't expect permission denied from that though20:20
clarkbfungi: iirc it manifested in the past as "I don't know what this ref is"20:20
mikalanteaya: there will be an airport doctor. You could ask to talk to them?20:21
*** ZZelle has joined #openstack-infra20:21
anteayaoh good question20:21
anteayaI'm asking20:22
fungizaro: i think it automatically notices if you change the layout.yaml, but you could check the puppet module to see if it has a special exec subscribed to that file20:24
*** annegentle has joined #openstack-infra20:25
*** baoli has joined #openstack-infra20:25
*** baoli_ has quit IRC20:25
*** dimtruck is now known as zz_dimtruck20:27
lifelessjamielennox|away: sorry, didn't see the context from your ^20:27
anteayamikal: they tell me there is no doctor20:27
anteayathey are offering to call me an ambulance20:28
*** zz_dimtruck is now known as dimtruck20:28
*** spzala has joined #openstack-infra20:28
anteayaI don't need an ambulance20:28
*** baoli has quit IRC20:28
*** dustins has quit IRC20:29
*** baoli has joined #openstack-infra20:29
lifelessok planes20:30
*** dimtruck is now known as zz_dimtruck20:30
*** pack3t has joined #openstack-infra20:31
*** sabari is now known as zz_sabari20:35
*** alugovoi has quit IRC20:36
notmynameanyone else on the 9pm flight out of SFO to AUK today?20:36
*** dhritishikhar has joined #openstack-infra20:36
*** alugovoi has joined #openstack-infra20:36
*** thingee has joined #openstack-infra20:37
clarkbhrm those dib image builds are still running. I need to run errands and will check on them after20:38
mikalI have to drop offline for my flight20:38
mikalI will check IRC later20:38
jeblairanteaya: i did a quick search and do not see a workable SYD->NZ cruise ship option20:40
jeblairnotmyname: i depart SFO at 7:45pm (for AKL)20:41
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Added manifest for
fungiclarkb: i think that ^ solves it?20:41
clarkbfungi: yup20:42
crinklewas just going to comment on that20:43
fungiclarkb: as for the dib builds, if it _is_ the concurrency issue, the outcome i witnessed was that all but one of them would fail with the same error, whichever one managed to win the race for the lock20:43
clarkbfungi: all three are building according to nodepool dib-image-list20:43
*** unicell has quit IRC20:43
fungiyeah, in my case it was around the point where they all tried to clone nova about the same moment20:44
fungitook a while for it to get to that point20:44
*** armax has quit IRC20:46
*** hdd has quit IRC20:48
*** shashankhegde has joined #openstack-infra20:52
*** zz_avozza is now known as avozza20:52
*** avozza is now known as zz_avozza20:54
*** armax has joined #openstack-infra20:54
openstackgerritJoe Gordon proposed openstack/requirements: Update to latest hacking (0.10.0)
*** armax has quit IRC20:59
*** mrmartin has quit IRC21:00
*** armax has joined #openstack-infra21:02
*** annegent_ has joined #openstack-infra21:03
*** annegentle has quit IRC21:04
*** baoli has quit IRC21:06
*** baoli has joined #openstack-infra21:08
*** ryanpetrello_ has joined #openstack-infra21:10
*** ryanpetrello has quit IRC21:12
*** ryanpetrello_ is now known as ryanpetrello21:12
*** achanda has joined #openstack-infra21:13
*** bswartz has quit IRC21:14
*** bswartz has joined #openstack-infra21:16
*** baoli has quit IRC21:17
*** achanda has quit IRC21:18
*** baoli has joined #openstack-infra21:18
openstackgerritEndre Karlson proposed openstack-infra/project-config: Add python-cueclient
*** zul has quit IRC21:22
ekarlso-can I get a signoff on ^21:23
*** zz_dimtruck is now known as dimtruck21:23
*** zz_sabari is now known as sabari21:30
*** krtaylor has quit IRC21:35
*** TravT_ has quit IRC21:36
*** annegent_ has quit IRC21:37
*** rlandy has quit IRC21:37
*** armax has quit IRC21:37
*** TravT has joined #openstack-infra21:37
*** mjturek has quit IRC21:39
*** mjturek has joined #openstack-infra21:39
*** TravT has quit IRC21:41
*** TravT has joined #openstack-infra21:41
*** dhritishikhar has quit IRC21:42
*** armax has joined #openstack-infra21:44
*** dhritishikhar has joined #openstack-infra21:45
*** zz_gondoi is now known as gondoi21:45
*** krtaylor has joined #openstack-infra21:47
*** zz_avozza is now known as avozza21:47
*** andreykurilin has quit IRC21:51
*** andreykurilin has joined #openstack-infra21:51
*** shashankhegde has quit IRC21:53
*** bswartz has quit IRC21:55
*** otter768 has joined #openstack-infra21:57
*** dhritishikhar has quit IRC21:59
zarofungi: what's the chain of events that should happen after21:59
dimsum__looks like we defined "The OpenStack way"!22:00
zarofungi: zuul adds a trigger event?22:01
*** otter768 has quit IRC22:01
*** hdd has joined #openstack-infra22:01
*** dannywilson has quit IRC22:02
*** bcourt has quit IRC22:03
*** Masahiro has joined #openstack-infra22:03
*** thingee has quit IRC22:03
*** baoli has quit IRC22:05
*** baoli has joined #openstack-infra22:05
*** jamielennox|away has quit IRC22:05
*** sigmavirus24 is now known as sigmavirus24_awa22:05
*** sigmavirus24_awa is now known as sigmavirus2422:05
*** mbacchi has quit IRC22:06
*** jpeeler has quit IRC22:06
*** Masahiro has quit IRC22:08
*** hemna has quit IRC22:08
*** jamielennox|away has joined #openstack-infra22:09
*** jamielennox|away is now known as jamielennox22:09
*** shashankhegde has joined #openstack-infra22:09
*** tonytan4ever has quit IRC22:11
*** armax has quit IRC22:11
*** Hal_ has joined #openstack-infra22:14
*** TravT_ has joined #openstack-infra22:15
*** TravT has quit IRC22:17
*** pc_m has quit IRC22:19
*** dimsum__ has quit IRC22:19
*** dimsum__ has joined #openstack-infra22:20
fungizaro: the sequence seems to be: Adding trigger event, Done adding trigger event, Run handler awake, Fetching trigger event, Processing trigger event22:20
*** baoli has quit IRC22:20
jeblairfungi, zaro: something i can help with?22:20
fungijeblair: troubleshooting the zuul installation on zuul-dev.openstack.org22:21
fungiwhich is hooked to the upgraded review-dev22:21
*** baoli has joined #openstack-infra22:21
zarojeblair: i'm trying to debug zuul-dev it doesn't seem to be sending jobs to gearman22:22
zarowhen i push a change to review-dev.o.o the last message in zuul debug log is Done adding trigger event: <TriggerEvent patchset-created gtest-org/gtest master 5306,8>22:23
*** baoli has quit IRC22:23
zaroi'm not sure what should happen next.22:23
jeblair2015-01-09 19:09:08,610 DEBUG zuul.Scheduler: Project gtest-org/gtest not found22:24
jeblairthat means there is no configuration for that project in the layout22:24
*** baoli has joined #openstack-infra22:24
*** dimsum__ has quit IRC22:24
fungiinterestingly it's in the projects list22:24
jeblairmind if i restart it?22:25
fungicomparing the layout.yaml for zuul and zuul-dev i think the latter needs a lot of updates22:25
zaroohh wait, the layout file changed.  is puppet still running?22:26
jeblairfungi: probably.  i don't see the final configuration in the debug log, so i suspect it's invalid in some way22:26
*** jpeeler has joined #openstack-infra22:26
*** jpeeler has quit IRC22:26
*** jpeeler has joined #openstack-infra22:26
zaroi updated the layout.yaml and it somehow changed back to original.22:27
*** bswartz has joined #openstack-infra22:27
jeblairthen i think puppet is running :)22:27
fungiif you haven't deactivated puppet agent, it will keep reverting whatever you change in there22:27
fungias that file is definitely managed by puppet22:27
zaroi thought i did, it's 'sudo service puppet stop ' right?22:28
*** bhuvan_ is now known as bhuvan22:28
*** annegentle has joined #openstack-infra22:28
fungizaro: no, we don't run puppet agent as a service, haven't for months22:28
fungipuppet agent --disable22:28
*** Longgeek has joined #openstack-infra22:28
jeblairsigh.  i thought someone was going to document that.22:28
fungizaro: i think probably for sanity's sake, the next step ought to be porting the current production config and layout to the dev environment22:29
zaroughh! actually clark told me that, but nibilizer said it's sudo command so i thought both were same.22:29
*** mattfarina has quit IRC22:29
*** TravT_ has quit IRC22:29
*** patrickeast has quit IRC22:30
*** baoli has quit IRC22:30
zarofungi: it's in review
*** sabeen1 has quit IRC22:30
*** armax has joined #openstack-infra22:30
fungioh, awesome. looking22:30
*** otter768 has joined #openstack-infra22:31
zarocommand to start zuul is '/etc/init.d/zuul' start and '/etc/init.d/zuul-merger' start correct?22:31
openstackgerritJames E. Blair proposed openstack-infra/system-config: Document how to disable/enable puppet
*** gondoi is now known as zz_gondoi22:32
clarkbit should already be documented22:32
*** Longgeek has quit IRC22:32
jeblairclarkb: what should?22:33
*** banix has quit IRC22:33
jeblairoh, it's here:
*** annegentle has quit IRC22:34
zarojeblair: only way i was able to get review-dev to _not_ timeout is to change the connectionpool setting to false.22:35
*** packet has quit IRC22:35
*** pack3t is now known as packet22:35
jeblairzaro: please keep track of that problem.  we need to solve it correctly before going into production.22:35
jeblairi don't think disabling the connection pool without understanding what is going on or why is okay22:36
jeblair(i think it's fine to do that to proceed with your work, but we need to start keeping a "punch list" of things like that)22:36
*** packet has quit IRC22:37
*** mriedem has quit IRC22:38
zaroyeah, i'm keeping track of it all.  i'll put it on etherpad shortly so you can see.22:38
*** teran has joined #openstack-infra22:40
clarkbhrm those image builds are still going22:41
mordredclarkb: didn't we blow away the dib source-repositories cache earlier?22:42
clarkbmordred: yes22:43
clarkbbut thats all in the cache and shouldn't affect anything22:43
clarkbalso the build sseem to have gotten stuck before that22:43
*** teran has quit IRC22:44
*** HeOS has quit IRC22:44
clarkbI don't see any exceptions grepping for devstack-trusty-dib. Now to grep for more targetted info22:45
*** sabari is now known as zz_sabari22:49
clarkbI wonder if we dont handle that particular error properly in nodepool22:50
mordredclarkb: that seems less than informative22:50
*** signed8b_ is now known as signed8bit_ZZZzz22:50
*** baoli has joined #openstack-infra22:50
jeblairmordred, clarkb: the actual error will probably be in the image build log22:50
*** esker has quit IRC22:50
jeblairi think the usage of "unable to create /path" is more "i couldn't build this image" not "there is a problem opening this file".22:51
clarkbjeblair: right I know the actual error. Its the permissions denied thing22:51
clarkbjeblair: yup22:51
clarkbthe major problem is that nodepool still thinks it is running those builds so cleanup wasn't properly22:51
jeblairah, gotcha22:52
*** sarob has joined #openstack-infra22:52
*** andreaf has quit IRC22:52
clarkbI think I may see the problem22:52
*** pradk has quit IRC22:52
*** armax has quit IRC22:53
clarkbit was sufficient to simply return when catching that exception have to remove the previously commited data for the building state22:53
clarkb*wasn't sufficient22:53
*** otter768 has quit IRC22:53
*** dimtruck is now known as zz_dimtruck22:54
*** mfink has quit IRC22:54
mordredyou know - there are times when I think we should use fixtures for more things in non-testing land22:55
fungior exit traps22:55
jeblairi disagree.  there's a really simple exception handling structure in nodepool.22:58
openstackgerritK Jonathan Harker proposed openstack-infra/nodepool: Raise a more descriptive KeyError for debugging
*** jcoufal has quit IRC23:00
jeblairbasically, things should try to do something, and if they fail, try to undo it, and then let the top level catch every exception so that it logs it and tries again23:00
clarkbya this was tricky becaues before we weren't handling the error and my fix was insufficient23:01
*** EmilienM is now known as EmilienM|afk23:01
clarkbthat'll learn me23:01
jeblairthis is what exceptions are for -- cleanup and state change on error. we're not exiting the program or thread, so fixtures and exit traps are not a good fit.23:02
openstackgerritClark Boylan proposed openstack-infra/nodepool: Cleanup DIB images in DB if they fail to build
*** zz_sabari is now known as sabari23:02
fungii was referring to the possibility of exit traps in diskimage-builder, to optionally clean up when it fails23:03
jeblairi'm told dib cleans up after itself23:03
*** mfink has joined #openstack-infra23:03
clarkbya it should. I think the particular case here where it didn't is a bug in the git repo element23:04
clarkbthat mordred added23:04
clarkbin the psat we always marked it ready so we would transition state. Sometimes to the wrong state. Removing the db row should be what we want23:06
*** baoli has quit IRC23:07
*** baoli has joined #openstack-infra23:08
morganfainbergsdague, thanks for all the responses on that thread: KSC and middleware23:09
morganfainbergsdague, really appreciate it :)23:09
morganfainbergsdague, helps a lot with trying to get all of this cleaned up / resolved in a consistent manner (and one that wont break *theworld*)23:10
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Make DIB exceptions more robust
*** armax has joined #openstack-infra23:13
jeblairclarkb: lgtm (though there was a typo in commit msg i think).  i added another (untested) patch to bring logging and exception patterns more inline.23:14
clarkbya I am looking at your change now23:16
*** amitgandhinz has quit IRC23:16
*** dkliban is now known as dkliban_afk23:16
*** dannywilson has joined #openstack-infra23:18
openstackgerritMerged openstack-infra/nodepool: Cleanup DIB images in DB if they fail to build
fungiin 146222 the method no longer sets self.disk_image at all. is that going to cause a problem or was it not necessary to track in the object?23:19
clarkbI thought it wasn't needed in the class object23:21
*** zz_dimtruck is now known as dimtruck23:21
fungiyeah, i'm not finding any other places accessing that attribute23:22
clarkbI can do a quick restart and delete the three rows in the db when those merge23:22
*** armax has quit IRC23:23
sdaguemorganfainberg: no prob23:27
openstackgerritMerged openstack-infra/nodepool: Make DIB exceptions more robust
*** ChuckC_ has joined #openstack-infra23:28
clarkbcool once puppet update sthe nodepool install I will do a restart23:28
*** banix has joined #openstack-infra23:29
*** ChuckC has quit IRC23:29
*** baoli has quit IRC23:30
* jeblair transits23:33
jeblairhave a nice weekend23:33
*** jeblair is now known as corvus23:33
*** vigneshvar has quit IRC23:34
*** hdd has quit IRC23:34
*** ryanpetrello has quit IRC23:36
*** ryanpetrello has joined #openstack-infra23:37
clarkbok nodepool is up to date going to restart it now23:37
*** ChuckC_ has quit IRC23:39
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Add debtcollector project configuration
*** dimsum__ has joined #openstack-infra23:40
*** dimsum__ has quit IRC23:41
*** ryanpetrello has quit IRC23:42
*** dimsum__ has joined #openstack-infra23:42
*** ivar-lazzaro has joined #openstack-infra23:43
*** pcaruana|afk| has quit IRC23:46
zarofungi: i can't figure out why zuul says it cannot find gtest-org/gtest, layout looks correct to me.23:46
zarois there a way to query zuul for the projects it knows about?23:46
fungiit enumerates the configuration it has parsed when it starts, in the debug loh23:47
*** mriedem has joined #openstack-infra23:47
*** dmsimard is now known as dmsimard_away23:47
fungineat, we have a zombified zuul-server process there too23:47
fungimind if i stop all the processes and try to start them fresh?23:48
zaronope, go ahead23:48
fungioh, yeah, also missing pidfile23:48
zaroi probably started them all wrong.23:49
fungiit's still not writing a pidfile either, so it's likely pretty ill. checking the logs now23:51
*** Masahiro has joined #openstack-infra23:52
*** sarob has quit IRC23:54
*** sarob_ has joined #openstack-infra23:54
*** dangers is now known as dangers_away23:54
*** ddieterly has quit IRC23:54
*** banix has quit IRC23:55
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Add debtcollector project configuration
*** teran has joined #openstack-infra23:56
mriedemanyone else see this before?
*** Masahiro has quit IRC23:56
mriedemlogstash shows this quite a bit and it's not always a failure23:56
mriedemblocking the backport of the pin to boto on stable/juno - which blocks all other stable/juno nova changes23:56
*** ChuckC_ has joined #openstack-infra23:57
*** dimsum__ has quit IRC23:58
*** dimsum__ has joined #openstack-infra23:58
fungimriedem: i believe someone filed a bug against httpretty earlier in the week noting that it is unable to properly wheel23:58
clarkbno I did't because httpretty's latest release is able to do it23:59
clarkbthe version we pin to is not23:59
*** melwitt has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!