Monday, 2015-08-31

*** baoli has quit IRC00:02
*** tiswanso has quit IRC00:04
*** achanda has joined #openstack-infra00:06
openstackgerritKai Qiang Wu(Kennan) proposed openstack-infra/project-config: Add docs and publish jobs for kolla
jamielennoxlifeless: when using pip install .[dep] syntax here my (previously ~empty) 3gb /tmp partition gets filled and then the process crashes, works if i do those deps normally00:07
jamielennoxlifeless: is this something you've seen before?00:07
lifelessjamielennox: you're filling 3GB installing keystone!00:10
lifelessjamielennox: so a) no. b) what uses the sapce up ?00:11
jamielennoxlifeless: it appears to be a /tmp/pip-XXX-build00:11
jamielennoxif i swap out the .[XXX] syntax with the packages they actually point to it doesn't happen though00:12
lifelessyeah - whats in there?00:12
*** stevemar has quit IRC00:13
*** yushiro has joined #openstack-infra00:13
*** dimtruck is now known as zz_dimtruck00:15
jamielennoxgah, tmp fills everything dies00:16
*** sflanigan has joined #openstack-infra00:19
*** sflanigan has joined #openstack-infra00:19
*** oomichi has joined #openstack-infra00:21
jamielennoxlifeless: ok,
*** sigmavirus24 is now known as sigmavirus24_awa00:22
lifelessthats odd00:23
jamielennoxlifeless: so i ran that after rm -rf /tmp/pip-*00:23
jamielennoxlifeless: right - and it's creating 3 of them00:23
lifelesssomething is running tox within that00:23
lifelessand checking out from git...00:24
lifelessjamielennox: so this is super weird :)00:24
jamielennoxoh good :)00:25
lifelessjamielennox: possibly your pip debug log would help, but I'm mostly worried about wtf tox is doing in there00:25
jamielennoxthis change seems to have happened a while ago, i haven't heard of it as a problem from other keystone people00:25
jamielennoxbut it's been hitting me for a while00:25
lifelessI wonder00:25
lifelessyou're installing .[foo] right ?00:26
lifelessok, I think I know00:26
lifelessdon't do .[foo] .[bar]00:26
lifelessthats bonkers00:26
lifeless.[foo,bar] and you should remove requirements.txt entirely00:26
lifelesssince its implied by keystone itself, which is what . refers to00:27
lifeless.[foo] is 'keystone + the keystone foo extra'00:27
lifelessI am not sure why you've got a full copy of your current dir being made00:27
jamielennoxso it's doing 3 keystone installs to figure out what is specified in each of the [XXX] reqs00:27
lifelessand copying the working tree as part of that00:28
lifelesswhich is where your space usage is coming from00:28
lifelessfixing the redundancy will make things much snappier up front00:28
jamielennoxah, yes my keystone/.testrepository is apparently 994M00:28
lifelessyou might want to trim or reset your testr database too00:28
jamielennoxnfi why00:29
lifelessthere's no built in gc00:29
jamielennoxok, so tox requirements should simply be "-r test-requirements .[XXX,YYY,ZZZ]" and let requirements be sourced from "."00:30
jamielennoxand kill my .testrepository00:30
jamielennoxlifeless: if you want to comment on anything00:40
openstackLaunchpad bug 1490354 in Keystone "Tox exhausting /tmp partition" [Undecided,New]00:40
jamielennoxlifeless: thanks00:40
*** achanda has quit IRC00:44
yushiroHi all :)00:49
yushiroI'd like to review .00:49
*** bswartz has joined #openstack-infra00:52
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add DIB_KEEP_PACKAGE_CACHE variable
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add DIB_USE_PACKAGE_CACHE variable
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add #!/bin/bash to library functions
*** chlong has joined #openstack-infra00:55
*** vivekd has quit IRC01:04
*** vivekd has joined #openstack-infra01:05
*** tbarron has quit IRC01:07
*** legoktm has quit IRC01:07
*** salv-orlando has joined #openstack-infra01:07
*** yuanying-alt has joined #openstack-infra01:10
*** salv-orlando has quit IRC01:13
*** baoli has joined #openstack-infra01:13
*** salv-orlando has joined #openstack-infra01:13
*** yuanying-alt has quit IRC01:14
openstackgerritMerged openstack-infra/project-config: Promote horizon selenium test job
*** baoli has quit IRC01:19
*** zz_dimtruck is now known as dimtruck01:24
*** zhenguo has joined #openstack-infra01:24
*** salv-orlando has quit IRC01:26
*** ddieterly has quit IRC01:28
*** dimtruck is now known as zz_dimtruck01:29
*** tiswanso has joined #openstack-infra01:29
*** vivekd has quit IRC01:32
*** baoli has joined #openstack-infra01:33
*** marzif__ has joined #openstack-infra01:33
*** tiswanso has quit IRC01:34
*** kambiz has quit IRC01:37
*** eantyshev has quit IRC01:37
*** marzif_ has quit IRC01:37
*** achanda has joined #openstack-infra01:37
*** flepied has joined #openstack-infra01:38
*** flepied1 has quit IRC01:38
*** jeadre has quit IRC01:39
*** jeadre has joined #openstack-infra01:40
Kennanhi infra-cores01:40
KennanCould anyone take some to help review it ?01:41
*** kambiz has joined #openstack-infra01:43
*** baoli has quit IRC01:46
*** chlong has quit IRC01:49
*** baoli has joined #openstack-infra02:00
*** armax has joined #openstack-infra02:10
*** yamamoto has quit IRC02:12
*** tbarron has joined #openstack-infra02:20
*** achanda has quit IRC02:22
*** sigmavirus24_awa is now known as sigmavirus2402:23
*** achanda has joined #openstack-infra02:24
*** stevemar has joined #openstack-infra02:25
*** salv-orlando has joined #openstack-infra02:28
*** ddieterly has joined #openstack-infra02:29
*** stevemar has quit IRC02:30
*** ddieterly has quit IRC02:33
*** gildub has joined #openstack-infra02:33
mordredlifeless: should we spend any mental energy on getting pip to not copy .testrepository as part of copying the working tree?02:34
*** asettle has quit IRC02:40
*** salv-orlando has quit IRC02:41
*** asettle has joined #openstack-infra02:44
*** tiswanso has joined #openstack-infra02:45
*** FallenPegasus has joined #openstack-infra02:45
*** tiswanso has quit IRC02:50
*** sigmavirus24 is now known as sigmavirus24_awa02:53
*** gildub has quit IRC02:56
*** ddieterly has joined #openstack-infra02:56
*** yamamoto has joined #openstack-infra02:58
*** yuanying-alt has joined #openstack-infra02:59
*** gildub has joined #openstack-infra02:59
*** gildub has quit IRC03:00
openstackgerritMerged openstack-infra/puppet-apps_site: Generate assets file from assets.yaml
*** yuanying-alt has quit IRC03:04
*** gildub has joined #openstack-infra03:08
*** larainema has joined #openstack-infra03:11
*** FallenPe_ has joined #openstack-infra03:12
*** FallenPegasus has quit IRC03:13
*** d34dh0r53 has quit IRC03:25
*** eglute has quit IRC03:25
*** Trozz_ has quit IRC03:25
*** sigmavirus24_awa has quit IRC03:25
lifelessmordred: possibly03:25
lifelessmordred: the issue is that if you don't copy dotfiles, for instance, .git is missing and boom03:26
*** jroll has quit IRC03:26
lifelessmordred: I don't recall why its copied in the first place, which would be the obvious optimisation03:26
jheskethmarzif__: sorry for the delay, I've reviewed 193125 for you03:26
jheskethKennan: I've taken a look fyi03:27
*** sputnik13 has joined #openstack-infra03:27
mordredlifeless: yeah03:27
Kennanthanks jhesketh:03:27
mordredlifeless: or, at some point wasn't pip going to move to doing an sdist first then an install of the sdist as part of install?03:27
mordredlifeless: because that way the manifest structure already in place should handle copying/not-copying the right things03:27
*** apmelton has quit IRC03:28
*** apmelton has joined #openstack-infra03:30
*** jroll has joined #openstack-infra03:31
*** eglute has joined #openstack-infra03:31
*** Trozz_ has joined #openstack-infra03:31
*** larainema has quit IRC03:31
*** madong_ has joined #openstack-infra03:31
*** d34dh0r53 has joined #openstack-infra03:32
*** sigmavirus24_awa has joined #openstack-infra03:33
openstackgerritMerged openstack-infra/project-config: Switch Horizon to Javascript-jobs
*** links has joined #openstack-infra03:40
*** salv-orlando has joined #openstack-infra03:43
*** FallenPe_ has quit IRC03:46
Kennanhi jhesketh:03:47
Kennannot sure if it is right channel to ask question about openstack/requirements project03:48
KennanI found another my code change to just add project to requirement namespace, it hit03:48
KennanAssertionError: False is not true : Invalid mac address detected.03:48
Kennanit seems nova issue03:48
KennanI did not think it is related with project added change03:49
*** cody-somerville has joined #openstack-infra03:50
*** cody-somerville has quit IRC03:50
*** cody-somerville has joined #openstack-infra03:50
*** Piet has joined #openstack-infra03:55
*** Piet has quit IRC03:56
*** salv-orlando has quit IRC03:56
*** Piet has joined #openstack-infra03:56
*** legoktm has joined #openstack-infra03:58
*** legoktm has quit IRC03:58
*** legoktm has joined #openstack-infra03:58
*** Piet has quit IRC03:59
*** sputnik13 has quit IRC04:04
*** ddieterly has quit IRC04:06
jheskethKennan: It doesn't look related, but there could be some subtleties in novas dependencies04:06
*** baoli has quit IRC04:07
*** bharathm has joined #openstack-infra04:12
openstackgerritSteve Baker proposed openstack/requirements: netaddr!=0.7.16 due to MAC validation regression
stevebakerthis change ^ should unwedge gate-heat-python27, which may mean it deserves being put at the head of the queue04:19
jheskethKennan: looks like it's hitting a few projects in the last day04:20
*** sputnik13 has joined #openstack-infra04:21
openstackgerritKen'ichi Ohmichi proposed openstack/requirements: Encap netaddr-0.7.16
*** stevemar has joined #openstack-infra04:23
jheskethstevebaker, oomichi: looks like you both have fixes for netaddr ^04:24
oomichijhesketh: oh, thanks04:27
oomichijhesketh: I send a tempest side patch also,,n,z04:27
oomichijhesketh, stevebaker: I abondoned my requirement patch and update tempest side based on that.04:28
jheskethoomichi: cool, thanks04:30
oomichijhesketh: tempest side patch is done on
oomichijhesketh: thanks for the notice :)04:33
*** sputnik13 has quit IRC04:36
*** sabeen1 has joined #openstack-infra04:40
yushirogood afternoon, infra-team.04:43
lifelessmordred: its kindof-planned04:43
lifelessmordred: but all the attempts so far have rolled back, IIRC04:43
stevebakeroomichi: beat you by 5 minutes :)04:44
*** tiswanso has joined #openstack-infra04:46
oomichistevebaker: yeah, nice job ;)04:48
*** yuanying-alt has joined #openstack-infra04:48
*** sabeen1 has quit IRC04:49
*** deepakcs has joined #openstack-infra04:49
*** sergmelikyan has joined #openstack-infra04:50
*** tiswanso has quit IRC04:51
*** salv-orlando has joined #openstack-infra04:52
*** yuanying-alt has quit IRC04:52
*** yfried|afk has quit IRC04:53
*** Sree has joined #openstack-infra04:53
*** Sree has quit IRC04:54
*** Sree has joined #openstack-infra04:55
*** abregman has joined #openstack-infra04:55
*** Sukhdev has joined #openstack-infra04:56
*** salv-orlando has quit IRC04:57
*** salv-orlando has joined #openstack-infra04:57
*** ig0r__ has joined #openstack-infra04:59
*** ig0r_ has quit IRC04:59
*** Sree has quit IRC05:03
*** puranamr has joined #openstack-infra05:05
*** ddieterly has joined #openstack-infra05:06
*** asettle has quit IRC05:06
*** asettle has joined #openstack-infra05:08
*** sergmelikyan has quit IRC05:09
*** salv-orlando has quit IRC05:10
*** ddieterly has quit IRC05:11
*** sabeen1 has joined #openstack-infra05:13
*** asettle has quit IRC05:14
*** asettle has joined #openstack-infra05:14
*** Sree has joined #openstack-infra05:21
*** sdake_ has quit IRC05:25
*** sdake has joined #openstack-infra05:26
openstackgerritStanislaw Pitucha proposed openstack-infra/project-config: Add pypy build to Anchor
*** oomichi has quit IRC05:27
*** sdake has quit IRC05:29
*** sdake has joined #openstack-infra05:30
*** baoli has joined #openstack-infra05:38
*** sabeen2 has joined #openstack-infra05:39
*** sabeen1 has quit IRC05:39
*** baoli has quit IRC05:42
*** sdake_ has joined #openstack-infra05:45
*** sdake has quit IRC05:49
*** markvoelker has joined #openstack-infra05:51
*** markvoelker_ has joined #openstack-infra05:53
*** matbu has joined #openstack-infra05:53
*** markvoelker has quit IRC05:56
*** jeadre has quit IRC06:01
*** jeadre has joined #openstack-infra06:02
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** ddieterly has joined #openstack-infra06:07
*** ParsectiX has joined #openstack-infra06:07
*** jeadre has quit IRC06:08
*** jeadre has joined #openstack-infra06:09
*** flepied has quit IRC06:09
*** salv-orlando has joined #openstack-infra06:10
*** ddieterly has quit IRC06:12
*** Sree_ has joined #openstack-infra06:14
*** Sree_ is now known as Guest4172206:14
*** stevemar has quit IRC06:15
*** davideagnello has joined #openstack-infra06:16
*** Sree has quit IRC06:16
*** oomichi has joined #openstack-infra06:18
*** afazekas_ has joined #openstack-infra06:19
*** markvoelker has joined #openstack-infra06:22
*** markvoelker_ has quit IRC06:25
*** yfried|afk has joined #openstack-infra06:26
*** yushiro has quit IRC06:26
*** markvoelker_ has joined #openstack-infra06:27
*** sergmelikyan has joined #openstack-infra06:27
*** armax has quit IRC06:28
*** arxcruz has joined #openstack-infra06:28
*** stevemar has joined #openstack-infra06:29
*** markvoelker has quit IRC06:29
*** davideagnello has quit IRC06:29
*** yfried has joined #openstack-infra06:30
*** yfried|afk has quit IRC06:31
Kennanjhesketh: did we have a bug for that jenkins failure issue?06:31
jheskethKennan: yes, it's related to here
openstackLaunchpad bug 1490380 in neutron "netaddr 0.7.16 causes gate havoc" [Critical,In progress] - Assigned to Kevin Benton (kevinbenton)06:31
jhesketha fix has gone in06:31
*** sergmelikyan has quit IRC06:32
Kennanok jhesketh: seems it is still under review06:33
*** jeadre has quit IRC06:33
*** jeadre has joined #openstack-infra06:34
*** sdake_ is now known as sdake06:36
*** yuanying-alt has joined #openstack-infra06:37
*** nadya has joined #openstack-infra06:37
*** mrunge has joined #openstack-infra06:40
*** flepied has joined #openstack-infra06:40
*** flepied1 has joined #openstack-infra06:41
*** yuanying-alt has quit IRC06:41
*** flepied has quit IRC06:45
*** tiswanso has joined #openstack-infra06:47
*** yushiro has joined #openstack-infra06:48
*** ParsectiX has quit IRC06:51
*** tiswanso has quit IRC06:51
*** ParsectiX has joined #openstack-infra06:52
*** stevemar has quit IRC06:53
*** puranamr has quit IRC06:58
*** Guest41722 has quit IRC06:58
*** Sree has joined #openstack-infra06:59
*** yushiro has quit IRC06:59
*** bexelbie_ has joined #openstack-infra06:59
*** Sukhdev has quit IRC07:00
*** stevemar has joined #openstack-infra07:00
*** Sree has quit IRC07:03
*** nadya has quit IRC07:05
*** achanda has quit IRC07:05
*** ddieterly has joined #openstack-infra07:08
*** peristeri has joined #openstack-infra07:09
*** jtomasek has quit IRC07:11
*** jtomasek has joined #openstack-infra07:11
*** ddieterly has quit IRC07:13
*** sabeen2 has quit IRC07:18
*** jeadre has quit IRC07:18
*** jeadre has joined #openstack-infra07:19
*** zeih has joined #openstack-infra07:19
openstackgerritSachi King proposed openstack-infra/project-config: Add constraints jobs to Jenkins
*** marcusvrn_ has quit IRC07:25
*** tmorin has joined #openstack-infra07:29
*** asettle has quit IRC07:30
*** Ala has joined #openstack-infra07:33
*** kaisers has joined #openstack-infra07:33
*** doude has joined #openstack-infra07:34
*** fhubik has joined #openstack-infra07:36
*** Sree has joined #openstack-infra07:37
*** jeadre has quit IRC07:40
*** zeih_ has joined #openstack-infra07:41
*** jeadre has joined #openstack-infra07:41
*** b10n1k_ has quit IRC07:44
*** zeih has quit IRC07:44
*** dizquierdo has joined #openstack-infra07:44
*** samuelBartel has joined #openstack-infra07:45
*** sshnaidm has joined #openstack-infra07:52
*** hashar has joined #openstack-infra07:55
*** baoli has joined #openstack-infra07:58
gmannoomichi: mtreinish : lifeless : (needed for is failing on neutron kilo job (invalid MAC)08:01
*** jerryz has quit IRC08:02
*** sshnaidm has quit IRC08:02
fboclarkb, Hi, about your discussion on puppet-httpd and http_mod on centos7 please have a look to and I think this is a bit like what we want : avoid a2enmod to be run on centos, as modules are enabled as soon as they are installed.08:03
*** baoli has quit IRC08:03
gmannoomichi: mtreinish : lifeless : which i think requirements updates ( netaddr>=0.7.12,!=0.7.16) needed for kilo branch too but that again need fix08:03
gmannseems like deadlock here ?08:03
*** ifarkas has joined #openstack-infra08:03
gmannsdake: ^^08:04
*** gildub has quit IRC08:08
*** ddieterly has joined #openstack-infra08:09
*** claudiub has joined #openstack-infra08:10
*** hichihara has quit IRC08:13
*** ddieterly has quit IRC08:14
oomichigmann: the latest is released08:14
*** jistr has joined #openstack-infra08:15
oomichigmann: netaddr-0.7.1708:15
*** sshnaidm has joined #openstack-infra08:15
*** kiran-r has joined #openstack-infra08:15
*** derekh has joined #openstack-infra08:15
gmannoomichi: i see, thats cool -
gmannoomichi: I will recheck our fixes08:17
oomichigmann: yeah, cool :)08:18
lifelessgmann: oomichi we should be able to fix kilo first, then roll forward to fix liberty/master08:19
gmannlifeless: tempest test failing on  will fail for kilo too08:20
gmannlifeless: ohk, you mean requirements update. yea should pass with 0.7.17 and then we can update kilo requirement also08:22
*** salv-orlando has quit IRC08:24
*** salv-orlando has joined #openstack-infra08:25
*** rossella_s has quit IRC08:25
*** yuanying-alt has joined #openstack-infra08:25
*** abalutoiu has joined #openstack-infra08:27
*** rossella_s has joined #openstack-infra08:27
*** dizquierdo has quit IRC08:27
*** pblaho has joined #openstack-infra08:28
*** markvoelker_ has quit IRC08:28
*** yuanying-alt has quit IRC08:30
openstackgerritDerek Higgins proposed openstack-infra/project-config: Tripleo CI reshuffle
*** kushal has joined #openstack-infra08:32
*** ihrachys has joined #openstack-infra08:41
*** amotoki has joined #openstack-infra08:41
*** ParsectiX_ has joined #openstack-infra08:41
*** ParsectiX_ has quit IRC08:42
*** dguitarbite has joined #openstack-infra08:42
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository for Fuel ScaleIO plugin integration
*** dingyichen has quit IRC08:47
*** tiswanso has joined #openstack-infra08:48
*** sdake has quit IRC08:50
*** fhubik is now known as fhubik_brb08:51
*** tiswanso has quit IRC08:53
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** e0ne has joined #openstack-infra08:58
*** yushiro has joined #openstack-infra08:59
lifelessgmann: 218736 is abandoned09:00
*** ildikov has joined #openstack-infra09:01
gmannlifeless: yea, now with netaddr-0.7.17 that test should pass without any issue09:02
gmannlifeless: should merge now without any tempest changes09:03
*** rossella_s has quit IRC09:05
openstackgerritMerged openstack-infra/release-tools: Document library release process
*** rossella_s has joined #openstack-infra09:06
*** e0ne has quit IRC09:09
*** fhubik_brb is now known as fhubik09:09
*** e0ne has joined #openstack-infra09:09
*** flip214 has joined #openstack-infra09:09
*** ddieterly has joined #openstack-infra09:10
*** bharathm has quit IRC09:12
*** abregman has quit IRC09:13
*** ddieterly has quit IRC09:14
*** bexelbie_ is now known as bexelbie09:19
derekhIs there anybody else around who can take a look at a reshuffle of tripleo jobs/repos please
*** markvoelker has joined #openstack-infra09:23
*** salv-orlando has quit IRC09:24
*** markvoelker_ has joined #openstack-infra09:26
*** pelix has joined #openstack-infra09:27
marzif__here: please if anyone can +A this review:
*** markvoelker has quit IRC09:30
*** markvoelker_ has quit IRC09:31
*** hashar_ has joined #openstack-infra09:34
*** yushiro has quit IRC09:36
*** ig0r_ has joined #openstack-infra09:37
*** hashar has quit IRC09:37
*** markvoelker has joined #openstack-infra09:37
*** marzif__ has quit IRC09:38
*** marzif__ has joined #openstack-infra09:38
bauzashey folks09:38
bauzascan see logstash's job queue is spiking09:39
*** ig0r__ has quit IRC09:39
bauzasanyone could confirm that logstash is a bit delayed?09:39
*** yfried is now known as yfried|afk09:43
*** dizquierdo has joined #openstack-infra09:45
openstackgerritTobias Henkel proposed openstack-infra/zuul: Support pagination in simplequery for Gerrit >=2.9
*** yfried|afk is now known as yfried09:53
*** vivekd has joined #openstack-infra09:54
*** pc_m has joined #openstack-infra09:57
*** sshnaidm has quit IRC09:57
*** Sree__ has joined #openstack-infra09:58
*** pc_m_ has joined #openstack-infra09:58
*** tnovacik has joined #openstack-infra09:59
*** pc_m_ has quit IRC09:59
*** Sree has quit IRC10:01
*** bharathm has joined #openstack-infra10:01
*** pc_m has quit IRC10:01
*** bharathm has quit IRC10:06
*** sshnaidm has joined #openstack-infra10:10
*** ddieterly has joined #openstack-infra10:10
*** marzif__ has quit IRC10:12
*** pc_m has joined #openstack-infra10:12
*** yuanying-alt has joined #openstack-infra10:14
*** john-davidge has joined #openstack-infra10:15
*** ddieterly has quit IRC10:15
*** john-davidge_ has joined #openstack-infra10:18
openstackgerritRamakrishnan G proposed openstack/diskimage-builder: Add proliant-tools element
*** yuanying-alt has quit IRC10:19
*** fhubik is now known as fhubik_brb10:19
*** john-davidge has quit IRC10:20
*** john-davidge_ is now known as john-davidge10:20
*** Sree__ has quit IRC10:23
*** jeadre has quit IRC10:23
*** jeadre has joined #openstack-infra10:24
openstackgerritRamakrishnan G proposed openstack/diskimage-builder: Add proliant-tools element
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds os-win repository
*** ParsectiX_ has joined #openstack-infra10:37
*** akraminsky has joined #openstack-infra10:46
*** teran has joined #openstack-infra10:46
openstackgerritIhar Hrachyshka proposed openstack/requirements: Add version cap for netaddr < 0.7.16
*** marzif__ has joined #openstack-infra10:47
*** tiswanso has joined #openstack-infra10:48
openstackgerritEmilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: copy dstat logs if exist
*** tiswanso has quit IRC10:53
*** jaypipes has joined #openstack-infra10:58
*** jtomasek has quit IRC10:59
*** bharathm has joined #openstack-infra11:02
*** dguitarbite has quit IRC11:04
*** bharathm has quit IRC11:07
marzif__please if anyone give +A to: we are currently blocked, thanks11:08
*** soren has joined #openstack-infra11:09
*** e0ne has quit IRC11:10
*** ddieterly has joined #openstack-infra11:11
*** ddieterly has quit IRC11:16
*** nadya has joined #openstack-infra11:17
*** dguitarbite has joined #openstack-infra11:20
*** stevemar has quit IRC11:22
*** fhubik_brb is now known as fhubik11:24
sdagueclarkb: so, the DHCPNAK issue is probably not a direct relationship to the failure11:27
*** jcoufal has joined #openstack-infra11:28
sdagueit does show that we are bleeding dhcp requests between test nodes all the time -
sdagueI tracked back through and matched up mac address entries11:29
marzif__sdague, if you have one moment, we need +A as we are blocked currently by
*** wenlock has quit IRC11:34
*** jaypipes has quit IRC11:35
*** jaypipes has joined #openstack-infra11:36
*** ldnunes has joined #openstack-infra11:38
*** markus_z has joined #openstack-infra11:39
*** amotoki has quit IRC11:39
*** amotoki has joined #openstack-infra11:39
openstackgerritMerged openstack-infra/project-config: Splitting stackforge/freezer on multiple repos
*** marzif__ has quit IRC11:42
*** gordc has joined #openstack-infra11:43
*** yfried is now known as yfried|afk11:43
*** fhubik is now known as fhubik_brb11:43
*** trown|outttypeww is now known as trown11:44
daemontool_sdague, ty11:45
*** fhubik_brb is now known as fhubik11:45
*** openstackgerrit has quit IRC11:46
*** jtomasek has joined #openstack-infra11:46
*** yfried|afk is now known as yfried11:47
*** openstackgerrit has joined #openstack-infra11:47
*** k4n0 has joined #openstack-infra11:48
*** markvoelker_ has joined #openstack-infra11:51
*** markvoelker has quit IRC11:51
*** akraminsky has quit IRC11:51
*** markvoelker_ has quit IRC11:55
*** ihrachys has quit IRC11:56
*** ihrachys has joined #openstack-infra11:56
*** jtomasek has quit IRC11:56
*** jtomasek has joined #openstack-infra11:56
*** markvoelker has joined #openstack-infra11:56
*** sergmelikyan has joined #openstack-infra11:58
*** mwagner_lap has quit IRC11:59
*** e0ne has joined #openstack-infra11:59
*** nadya has quit IRC12:01
*** sergmelikyan has quit IRC12:02
*** yuanying-alt has joined #openstack-infra12:03
*** vivekd has quit IRC12:07
*** yuanying-alt has quit IRC12:07
*** dprince has joined #openstack-infra12:08
*** alivigni has joined #openstack-infra12:09
openstackgerritChris Burroughs proposed openstack-infra/jenkins-job-builder: support for the phabricator plugin
*** ddieterly has joined #openstack-infra12:12
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository for Fuel ScaleIO plugin integration
openstackgerrityolanda.robla proposed openstack-infra/puppet-jenkins: Don't configure memory swap bytes on cgroups for Trusty
*** aysyd has joined #openstack-infra12:14
*** ddieterly has quit IRC12:16
*** dkranz has joined #openstack-infra12:17
openstackgerritgordon chung proposed openstack-infra/project-config: Make gate-ceilometer-dsvm-functional-postgresql voting
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient
*** trown is now known as trown|relocating12:19
*** rfolco has joined #openstack-infra12:19
*** ParsectiX_ has quit IRC12:21
*** ParsectiX has quit IRC12:21
*** ParsectiX has joined #openstack-infra12:22
*** cburroughs has joined #openstack-infra12:23
ajo ping12:23
ajocan we prioritize to unbreak python unit tests?12:24
ajokibana link:
ajowe're seeing 1000s of failures due to a netaddr change12:24
ajosdague, anteaya , does it make sense to ping anyone about this?12:25
ajoihrachys ^?12:25
sdagueajo: it needs tests to come back first12:25
ajohmm, I guess it's not that super urgent, I just realized it's only neutron-unit tests12:26
ajohmm, woops sdague , I see12:26
ajowhat happened?12:26
*** hashar_ has quit IRC12:26
ihrachysajo: I guess sdague means we need a +1 report from jenkins first12:27
sdagueihrachys: yes12:27
ihrachysajo: now that it is about to fail on grenade, how about respinning the patch to clear some nits you got?12:28
ajoah, ok, I thought we had a worse issue12:28
ajoihrachys, we could, if we get the +2's again quickly12:28
ajoI didn't want to clear those out12:28
ajoand delay the thing12:28
ihrachysajo: obviously we will get it quick12:29
ajoihrachys, Ok, I will handle the comment about validate_mac12:29
*** baoli has joined #openstack-infra12:29
ihrachysand rosella's12:29
ajothe other changes could deserve more discussion,12:29
ajolet he mech her's12:29
sdaguealso - Bug 1355573 - grenade volume scenarios periodically fail /w SSHTimeout seems to have gotten massively worse in the last 24 hours12:29
openstackbug 1355573 in grenade "grenade volume scenarios periodically fail /w SSHTimeout" [Undecided,New]
*** baoli_ has joined #openstack-infra12:31
*** baoli_ has quit IRC12:31
*** baoli_ has joined #openstack-infra12:32
*** soren has quit IRC12:33
*** k4n0 has quit IRC12:34
*** baoli has quit IRC12:34
*** tiswanso has joined #openstack-infra12:34
*** krtaylor has quit IRC12:34
*** Sree has joined #openstack-infra12:35
*** tiswanso has quit IRC12:39
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: added query for netaddr 0.7.16 mac addr parsing bug
*** annegentle has joined #openstack-infra12:44
openstackgerritwill soula proposed openstack-infra/jenkins-job-builder: Add Ability to use the AWS Cloudformation Plugin
openstackgerritwill soula proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin
*** ddieterly has joined #openstack-infra12:46
*** tlian has joined #openstack-infra12:46
*** krtaylor has joined #openstack-infra12:47
*** ifarkas has quit IRC12:48
*** k4n0 has joined #openstack-infra12:48
*** ifarkas has joined #openstack-infra12:49
*** burgerk has joined #openstack-infra12:50
*** bharathm has joined #openstack-infra12:51
openstackgerritMerged openstack-infra/puppet-accessbot: Basic acceptance tests
*** amuller has joined #openstack-infra12:53
*** bharathm has quit IRC12:56
*** afazekas__ has joined #openstack-infra12:56
*** cbader has joined #openstack-infra12:58
*** afazekas_ has quit IRC12:58
*** hashar has joined #openstack-infra13:01
*** tmorin has quit IRC13:01
*** mjturek1 has joined #openstack-infra13:05
*** rossella_s has quit IRC13:05
*** vivekd has joined #openstack-infra13:06
*** rossella_s has joined #openstack-infra13:06
*** dkranz has quit IRC13:07
*** madong_ has quit IRC13:08
*** jeadre has quit IRC13:08
*** tmorin has joined #openstack-infra13:08
*** ddieterly has quit IRC13:09
*** jeadre has joined #openstack-infra13:09
tmorinlooking for help on an issue with jenkins checks not starting on a change ( )13:10
tmorinanyone ?13:10
*** sabeen1 has joined #openstack-infra13:11
*** vinay_yadhav has joined #openstack-infra13:11
*** kgiusti has joined #openstack-infra13:11
*** rlandy has joined #openstack-infra13:13
*** glauco_ has joined #openstack-infra13:13
*** xyang1 has joined #openstack-infra13:16
*** alivigni has quit IRC13:16
*** jcoufal has quit IRC13:17
*** wenlock has joined #openstack-infra13:18
*** dkranz has joined #openstack-infra13:20
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient
*** e0ne has quit IRC13:22
*** wenlock has quit IRC13:22
*** matrohon has joined #openstack-infra13:23
tmorinping fungi ? (about the jenkins-related issue above)13:23
*** ivar-lazzaro has joined #openstack-infra13:23
*** ivar-lazzaro has quit IRC13:24
openstackgerritMerged openstack-infra/elastic-recheck: added query for netaddr 0.7.16 mac addr parsing bug
*** ivar-lazzaro has joined #openstack-infra13:24
*** glauco_ has quit IRC13:26
*** k4n0 has quit IRC13:27
*** eharney has joined #openstack-infra13:27
openstackgerritMerged openstack-infra/groups: Add ambassador application form
*** e0ne has joined #openstack-infra13:29
*** mtanino has joined #openstack-infra13:30
*** links has quit IRC13:30
*** jcoufal has joined #openstack-infra13:30
*** ayoung has joined #openstack-infra13:31
*** jcoufal has quit IRC13:31
*** thorst1 has joined #openstack-infra13:32
*** jcoufal has joined #openstack-infra13:32
*** burgerk has quit IRC13:34
*** stevemar has joined #openstack-infra13:34
*** trown|relocating is now known as trown13:35
*** superdan is now known as dansmith13:35
*** salv-orlando has joined #openstack-infra13:36
*** dims has joined #openstack-infra13:36
*** hashar_ has joined #openstack-infra13:39
*** doug-fish has joined #openstack-infra13:39
*** stevemar has quit IRC13:39
*** k4n0 has joined #openstack-infra13:40
*** ivar-lazzaro has quit IRC13:41
*** tiswanso has joined #openstack-infra13:41
*** hashar has quit IRC13:41
*** ldnunes has quit IRC13:41
*** soren has joined #openstack-infra13:41
*** ddieterly has joined #openstack-infra13:43
*** k4n0 has quit IRC13:44
tmorinping anteaya ?... I have an issue with jenkins checks not getting started on a change ( ), blocking progress of the said patch13:44
*** k4n0 has joined #openstack-infra13:45
svashekafungi, mordred : hi guys, could you please review , it's been waiting for a long time13:46
*** ldnunes has joined #openstack-infra13:46
openstackgerritPranav Salunke proposed openstack-infra/project-config: Create new project labs
*** Sree has quit IRC13:46
*** Sree has joined #openstack-infra13:47
*** esker has quit IRC13:47
*** esker has joined #openstack-infra13:48
*** doude_ has joined #openstack-infra13:48
*** kiran-r has quit IRC13:48
*** doude has quit IRC13:48
*** doude_ has quit IRC13:49
*** cdoron has joined #openstack-infra13:49
*** annegentle has quit IRC13:49
*** k4n0 has quit IRC13:50
mordredsvasheka: done13:51
mhumordred: hi, I was away on holidays, is phabricator + cauth still a thing, and if so, anything needed on our side to make it work ?13:51
mordredmhu: oh hey!13:51
mordredClint: ^^ mhu is our favorite person13:51
*** Sree has quit IRC13:51
*** mwagner_lap has joined #openstack-infra13:52
mordredmhu: I've been mostly away as well - just getting back in to it this week13:52
*** yuanying-alt has joined #openstack-infra13:52
mhuyay, I am popular ! \o/13:52
svashekamordred, thank you!13:52
mordredmhu: it's definitely still a thing, I believe our next step is "figure out how to run cauth in front of phab"13:52
mordredmhu: which will probably be easier with you :)13:52
mhumordred, alright, happy to help13:53
Clintmordred: woot!13:53
*** glauco has joined #openstack-infra13:54
*** pradk has joined #openstack-infra13:54
pabelangerWanted to see if there was any issue add myself into the trystack-site-core group? Some team mates have some review up and looking for help getting them merged into upstream.13:54
fungitmorin: shows that stackforge/networking-bagpipe-l2 doesn't have any gate pipeline jobs (except for a job which only runs when you're changing requirements lists)13:56
*** glauco has quit IRC13:56
*** yuanying-alt has quit IRC13:56
*** sigmavirus24_awa is now known as sigmavirus2413:56
tmorinfungi: yes, that's true, we haven't enabled unit tests yet (we'll soon enable python-jobs)13:57
*** spzala has joined #openstack-infra13:57
*** k4n0 has joined #openstack-infra13:57
fungitmorin: without a job to run on a change, zuul won't enqueue it into the corresponding pipeline13:57
tmorinfungi: however, that shouldn't block changes being merged right ?   but right now, a workflow+1 does nothing13:57
tmorinfungi: this wasn't true last week13:57
fungitmorin: did the change last week modify a requirements list?13:58
tmorinfungi: e.g. was merged without a jenkins vote13:58
tmorinsorry wrong example13:58
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: add query for misuse of netaddr
openstackgerritMerged openstack-infra/project-config: Add copy-screenshots publisher
openstackgerritRyan Rossiter proposed openstack/requirements: Bump oslo.versionedobjects version to 0.9.0
*** mriedem has joined #openstack-infra13:59
tmorinfungi: a change was merged last week, and yes it did change requirements.txt -- would that block subsequent changes ?13:59
fungitmorin: no, the only jobs your project has in the check and gate pipelines are jobs which specifically limit their running to changes which modify requirements.txt and test-requirements.txt (the requirements check jobs)14:00
tmorinfungi: understood14:01
tmorinfungi: what would cause a workflow+1 to do nothing then ?  ( )14:01
fungitmorin: correct14:02
fungitmorin: without a job to run on a change, zuul won't enqueue it into the corresponding pipeline14:02
fungi(repeating my earlier statement)14:02
*** doude has joined #openstack-infra14:02
*** doude has quit IRC14:02
*** doude has joined #openstack-infra14:02
fungitmorin: i recommend starting with something like adding some unit test jobs, and then modifying tox.ini so that they always pass14:03
*** Kennan has quit IRC14:03
*** Kennan2 has joined #openstack-infra14:03
*** dteselkin has quit IRC14:03
*** asilenkov has quit IRC14:03
fungithen you can introduce a change to your tox.ini later which runs some actual tests in your unit test job14:03
*** deepakcs has quit IRC14:03
tmorinfungi: yes, we plan to do that, but other changes to the project (before we added the check-requirements) were merging fine14:03
*** ParsectiX has quit IRC14:04
openstackgerritClaudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements
fungitmorin: have an example?14:04
*** ayoung has quit IRC14:04
tmorinfungi: oh, I get it14:04
tmorinfungi: we had noop before, but we haven't it now14:05
fungitmorin: we have a noop job which you can run if you don't have any check or gate jobs. that's what you were running previously14:05
*** bnemec has joined #openstack-infra14:05
tmorinfungi: somebody suggested to remove it when we added check-requirements....14:05
* fungi is in a meeting right now or would explain options in more detail14:05
fungitmorin: yes, we don't want anyone running noop if they have other jobs in check or gate pipelines14:06
fungiit causes them to share a gate queue with other projects which only run the noop job14:06
*** asilenkov has joined #openstack-infra14:06
tmorinfungi: yes, but in our case, that blocks any merge from happening right ?14:06
fungimost projects start with non-conditional jobs and don't run into this corner case14:06
*** wfoster has joined #openstack-infra14:07
*** dteselkin has joined #openstack-infra14:07
fungithe requirements jobs are conditional, and the reviewers of the change to your jobs didn't consider that14:07
*** burgerk has joined #openstack-infra14:07
tmorinfungi: does it makes sense to re-add noop-job until we enable python-jobs ?14:07
tmorinfungi: ok, I understand the reason for the mistake14:08
*** dkranz has quit IRC14:08
fungitmorin: only if you remove the requirements job template when switching back to the noop template14:08
*** afazekas__ has quit IRC14:08
*** jroll has quit IRC14:09
*** jroll has joined #openstack-infra14:09
*** markvoelker has quit IRC14:09
tmorinfungi: you mean I can't have both the noop-jobs and the check-requirements job in zuul ? (even temporarily until we add python-jobs)14:10
fungitmorin: yes, we don't want anyone running noop if they have other jobs in check or gate pipelines14:10
fungiit causes them to share a gate queue with other projects which only run the noop job14:10
tmorinfungi: ok, understood14:11
*** eharney has quit IRC14:11
*** boris-42 has joined #openstack-infra14:11
tmorinfungi: then I have an incentive to not delay enabling python-jobs, which is not a bad thing ! ;)14:11
*** zz_dimtruck is now known as dimtruck14:11
fungitmorin: silver lining!14:11
tmorinfungi: ;-)   thanks a lot for your clear explanations14:12
pabelangerany infra core mind +2 its adding a .gitreview files into trystack-site14:12
fungihtm... jenkins01 is so starved for memory i can't run `free` (and can't even ssh in without many attempts)14:14
anteayafungi: :(14:15
*** wenlock has joined #openstack-infra14:17
*** mpaolino has joined #openstack-infra14:17
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: update 1490374 for tempest hits as well
openstackgerritMerged openstack-infra/system-config: Upgrade etherpad-dev to Ubuntu Trusty
openstackgerritBrianna Poulos proposed openstack/requirements: Add Castellan to global requirements
*** annegentle has joined #openstack-infra14:19
*** ihrachys has quit IRC14:20
*** ildikov has quit IRC14:20
*** ihrachys has joined #openstack-infra14:21
*** zeih_ has quit IRC14:22
*** tonytan4ever has joined #openstack-infra14:22
*** r-daneel has joined #openstack-infra14:22
*** cdoron has quit IRC14:24
*** dkranz has joined #openstack-infra14:24
*** thiagop has joined #openstack-infra14:24
sdaguefungi: when you get a chance, can you promote this test skip -
sdaguethat's causing quite a lot of carnage atm, and we need breathing room to figure it out14:25
*** vinay_yadhav has quit IRC14:26
*** vivekd has quit IRC14:27
*** smoser has quit IRC14:27
*** dkranz has quit IRC14:28
fungisdague: done14:28
*** baoli_ has quit IRC14:29
openstackgerritMerged openstack-infra/elastic-recheck: add query for misuse of netaddr
*** baoli has joined #openstack-infra14:30
*** jswarren has joined #openstack-infra14:30
*** jswarren_ has joined #openstack-infra14:30
openstackgerritVictor Ryzhenkin proposed openstack/requirements: Add murano-specs repository to projects.txt
*** abalutoiu has quit IRC14:31
openstackgerritJohn Trowbridge proposed openstack/diskimage-builder: Move service enable to install
*** dkranz has joined #openstack-infra14:32
*** claudiub has quit IRC14:32
*** alivigni has joined #openstack-infra14:34
*** jswarren_ has quit IRC14:35
openstackgerritVictor Ryzhenkin proposed openstack/requirements: Add murano-specs repository to projects.txt
*** ildikov has joined #openstack-infra14:36
sdaguefungi: thank yuo14:37
*** peristeri has quit IRC14:38
*** ayoung has joined #openstack-infra14:38
*** mriedem is now known as mriedem_away14:39
*** bharathm has joined #openstack-infra14:40
clarkbsdague so >1 VM fight for the same ip?14:41
*** dkranz has quit IRC14:43
*** jaosorior has joined #openstack-infra14:44
openstackgerritJohn Trowbridge proposed openstack/diskimage-builder: Move service enable to install
*** bharathm has quit IRC14:45
sdagueclarkb: nope14:46
sdagueit's other test nodes14:46
clarkbbut we firewall14:46
sdagueif you look at that search by MAC you'll find it's on 2 jobs at the same time14:46
*** MarkAtwood has joined #openstack-infra14:47
sdagueclarkb: apparently pooly14:47
clarkbI wonder if nova/neutron just wipe out our default deny14:47
clarkboh except its l2 broadcast14:48
sdagueclarkb: could be, they do assume they own the iptables14:48
clarkbhrm ebtables may be needed14:48
*** dkranz has joined #openstack-infra14:48
sdaguelet me get a good example14:48
*** eharney has joined #openstack-infra14:48
fungioh, that's sinister14:48
*** bmjen has joined #openstack-infra14:48
*** stevemar has joined #openstack-infra14:48
mtreinishyikes, that's something14:49
*** achanda has joined #openstack-infra14:49
fungii guess we can just block on the host interfaces at least and fix that14:50
clarkbwe can use last octet of test node to set second octet for test vm range14:50
clarkbthat should isolate on /24s14:50
*** sputnik13 has joined #openstack-infra14:50
sdagueclarkb: well, except, it shouldn't matter14:50
sdaguebecause we're doing the right thing and not handing that address to that node14:50
sdagueand, our default lease to cirros is 1d14:50
*** stevemar has quit IRC14:50
*** wenlock1 has joined #openstack-infra14:51
clarkbya but it is extra noise14:51
sdagueso, from the Nova side, it's handled correctly14:51
*** dims has quit IRC14:51
sdaguesure, so the other option is *always* run multi host14:51
sdaguebecaus nova-net then protects against this14:51
sdagueand makes dnsmasq only listen on internal interface14:52
*** sputnik13 has quit IRC14:52
sdagueso there is a DHCPNAK on
*** sputnik13 has joined #openstack-infra14:53
sdagueat exactly the same time the same MAC address is getting an ACK from it's test node -
sdaguedifferent patch different test jobs14:53
*** markvoelker has joined #openstack-infra14:53
sdaguebut time corolated exactly14:53
*** Ala has quit IRC14:53
sdagueI got suspicious when I found out how much we DHCPNAK but succeed in runs14:54
*** markvoelker has quit IRC14:54
sdagueand eventually dug through enough logs to realize the mac address in question wasn't one that existed in the job at all14:54
*** markvoelker has joined #openstack-infra14:55
*** hdd has joined #openstack-infra14:55
*** samuelBartel has quit IRC14:56
openstackgerritJohn Trowbridge proposed openstack/diskimage-builder: [ironic-agent] Move service enable to install
*** spzala has quit IRC14:57
*** jistr is now known as jistr|call14:57
sdaguealso, in the land of promoting things we need: and - in that order. Which is stable/kilo and master patches to neutron to fix the netaddr issue with their l3 agent14:57
sdaguegrenade w/ neutron is currently 100% fail14:57
sdagueso blocking grenade, devstack, tempest in addition to neutron14:58
fungisdague: i guess as soon as 218871,3 finishes gate jobs, i'll pump those through after it14:58
sdaguefungi: cool, the kilo patch has to go first, so that the master one will pass grenade14:58
sdaguethe master patch is currently jenkins -1 because of that14:58
fungialso i may just need to reboot jenkins01 straight up. not even enough memory to fork a ps15:00
sdaguefungi: yeh, I found it was super slow this weekend15:00
fungithough on a positive note, the server's so bad off that nodepool hasn't been able to add any slaves to it for a while15:00
fungiso i don't think this will interrupt any jobs at least15:00
sdagueis there some kind of monit definition that could be used to auto kill things?15:01
sdaguejenkins seems to go south at regular enough a clip15:01
*** baoli has quit IRC15:01
fungifor jenkins? probably not. under normal circumstances we've been putting them into a graceful shutdown (can be done with an authenticated api call) and then waiting for all running jobs to complete and deleting any ready nodes from them while that happens, then stopping the service and editing the global configuration to clear out remaining slaves15:02
fungii'd be hesitant to automate that15:02
*** baoli has joined #openstack-infra15:02
fungihowever, there's a recent commit that fixes a jenkins bug which may be related to the behavior clarkb reported, so upgrading is something we should look into soon15:03
*** dims has joined #openstack-infra15:03
fungisince it might make this stop being necessary again15:03
sdaguewould be nice15:03
*** dizquierdo has quit IRC15:03
clarkbI did respond to the bug to clarify again that we dont use a cloud plugin...15:03
fungiyeah, i saw15:04
clarkbbut ya maybe we can get a build with that fix and try it real soon now15:04
fungiokay, jenkins01 is rebooting now15:04
*** bnemec has quit IRC15:04
*** achanda has quit IRC15:05
*** flepied has joined #openstack-infra15:05
*** eantyshev has joined #openstack-infra15:05
fungii caught it before it came up enough to start running any jobs, stopped the service and cleaned the remaining slaves out of its config15:06
clarkbI do enjoy that the response to people thinking you use cloud plugins is that you should expect such broken behavior15:06
*** gyee has joined #openstack-infra15:07
fungialso that if you're not using long-lived slaves you must by default be using one of those broken cloud plugins15:07
fungipretty sure they're going to come back and say we're not using jenkins slaves in the way that they're meant to be used, should use a cloud plugin instead15:08
*** flepied1 has quit IRC15:08
*** dkranz has quit IRC15:08
derekhIs there anybody else around who can take a look at a reshuffle of tripleo jobs/repos please
*** bnemec has joined #openstack-infra15:09
*** sigmavirus24 is now known as sigmavirus24_awa15:09
*** yfried has quit IRC15:11
*** tjones has joined #openstack-infra15:11
*** dkranz has joined #openstack-infra15:11
*** larainema has joined #openstack-infra15:11
openstackgerritMerged openstack-infra/elastic-recheck: update 1490374 for tempest hits as well
*** tjones has left #openstack-infra15:14
*** wznoinsk1 is now known as wznoinsk15:15
*** stevemar has joined #openstack-infra15:16
*** sigmavirus24_awa is now known as sigmavirus2415:17
*** woodster_ has joined #openstack-infra15:18
*** jtomasek has quit IRC15:20
*** stevemar has quit IRC15:21
funginodepool seems to have ceased booting new nodes. looking into it now15:21
*** zeih has joined #openstack-infra15:22
*** thedodd has joined #openstack-infra15:23
anteayafungi: :|15:23
*** claudiub has joined #openstack-infra15:23
openstackgerritEvgeny Antyshev proposed openstack-infra/devstack-gate: Third-party rhel-based oses in devstack-gate
*** FallenPegasus has joined #openstack-infra15:24
*** wshao_ has joined #openstack-infra15:24
*** ivar-lazzaro has joined #openstack-infra15:24
flip214Hi. Looking at the drbd tempest runs I can see that *all* failed during some 6-hour window today. was that some general problem, or do I start do dig deeper?15:25
flip214picture link would be
flip214sorry for the long line, but it's easier to see that way15:26
*** jaypipes has quit IRC15:26
*** vtapia has joined #openstack-infra15:26
flip214hmmm, perhaps that's better:
fungilast mention of demand from gearman was at 14:51 utc, which roughly corresponds to when it looks like nodepool may have ceased building new nodes15:27
*** zeih has quit IRC15:27
openstackgerritMerged openstack/requirements: netaddr!=0.7.16 due to MAC validation regression
*** ivar-lazzaro has quit IRC15:27
clarkbflip214: if you use neutron it is ucrrently broken by a new netaddr release15:27
sdaguefungi / clarkb - the tempest patch merged15:27
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory
*** ivar-lazzaro has joined #openstack-infra15:28
sdagueso if we could get a promote on and - in that order15:28
clarkbflip214: if you don't use neutron then I think you may need to do more investgating15:28
clarkbsdague: once we figure out why we aren't building nodes we can do that15:28
sdagueclarkb: ok15:28
*** jsavak has joined #openstack-infra15:28
flip214clarkb: thanks for the hint, "AssertionError: False is not true : Invalid mac address detected."15:29
*** k4n0 has quit IRC15:29
fungirax-ord: Forbidden: Quota exceeded for ram: Requested 8192, but already used 507904 of 512000 ram15:29
*** flepied has quit IRC15:29
fungithough that's probably unrelated15:29
anteayafungi: still not what we like to hear15:30
anteayamorning zaro15:30
*** tiswanso has quit IRC15:30
clarkbfungi: the gearman connection appears to be up and functioning, both sides report it ot be established15:30
*** tiswanso has joined #openstack-infra15:31
*** david-ly_ has joined #openstack-infra15:31
clarkbfungi: we did only calcuate demand 10 times during the 1400 hour15:32
*** david-ly_ is now known as david-lyle_15:33
clarkbwe hould be doing it every few seconds iirc15:33
fungiall the traceback noise in the debug log from broken tripleo-test-cloud-hp1 is tough to wade through15:33
*** harlowja_at_home has joined #openstack-infra15:33
*** david-lyle has quit IRC15:33
clarkbfungi: I think we should stack dump then lookto see what the main thrad is doing as that is where we calculate iirc15:33
sdagueflip214: Bug 1490380 - netaddr 0.7.16 causes gate havoc15:34
openstackbug 1490380 in neutron "netaddr 0.7.16 causes gate havoc" [Critical,In progress] - Assigned to Miguel Angel Ajo (mangelajo)15:34
fungiagreed, getting one now15:34
sdagueflip214: - you can see how we got wrecked by that as well15:34
*** fedexo has joined #openstack-infra15:34
fungiclarkb: SIGUSR2 YEAH?15:34
fungier, capslockfail15:34
*** david-lyle_ is now known as david-lyle15:35
Clintmore like capslock success15:35
*** puranamr has joined #openstack-infra15:35
fungiit's the toggle of AWESOME15:35
flip214sdague: thanks, right.15:35
flip214we're running in the openstack CI, so "your" problems are "our" problems as well ;/15:35
flip214thanks for all the hard work, BTW15:36
clarkbflip214: I think so but let me check15:36
fungiclarkb: yeah, that was it15:36
clarkber fungi ^ and yes confirmed15:36
sdagueflip214: so, if you use neutron, you need to wait for another set of patches to come in15:36
clarkbpabelanger: the centos7 image build issue apperas to possibly be a difference between rax image sand hpcloud images :/15:37
*** ifarkas has quit IRC15:37
flip214it already started working a few hours ago.15:37
clarkbpabelanger: they build fine on rax but not hpcloud so that is the next place to start looking15:37
*** fhubik has quit IRC15:37
sdagueflip214: ok, yeh, then that was just the invalid macaddr thing15:37
sdaguethat was 0.7.16 having a big bug15:37
mordredclarkb: snapshot or dib?15:37
clarkbmordred: snapshot15:37
sdaguethere is still a behavior change in 0.7.17 that there are patches to fix15:37
mordredclarkb: lovely15:37
clarkbmordred: there is no different between the providers with dib15:37
mordredclarkb: I know - but I've long since stopped asusming15:38
fungiclarkb: 2015-08-31 15:35:43,851 DEBUG nodepool.stack_dump15:38
fungiclarkb: and i did a second one at 2015-08-31 15:36:47,214 DEBUG nodepool.stack_dump15:38
fungifor comparison purposes15:38
mordredclarkb: when you upload dibs manually - what  have you been doing to get it to work consistenly. anything in particular? or just tons of retries?15:38
*** jtomasek has joined #openstack-infra15:39
clarkbfungi: I see the issue15:39
fungiclarkb: Thread: Gearman client connect (140579170150144) seems to be spinning in waiter.acquire()15:39
clarkbfungi: I don't think thats the problem Thread: NodePool (140581185824512) is the main thread15:40
clarkbfungi: and its in reconfigure managers and its sitting on a jenkins open15:40
clarkbwe might need to make that smarter considering our jenkins woes and fal late rather than early on that15:40
fungioh... i wonder whether jenkins01 tanked it15:40
clarkbya I think it did15:40
fungiokay, so any objection to me restarting nodepoold now?15:40
clarkbno let me paste this thread stack so we can debug it after but don't need you to wait15:41
fungiokay, nodepoold restarted15:41
*** glauco has joined #openstack-infra15:42
fungiit's booting a ton of nodes15:42
clarkbwe either need a timeout there or need to defer that action to later15:42
fungithe test nodes graph already indicates the change. i'll go ahead and promote those changes sdague mentioned now15:42
*** glauco has quit IRC15:43
clarkbfungi: but run loop is basically update config then check for needed nodes then launch nodes15:43
*** glauco has joined #openstack-infra15:43
clarkbso if update config hangs on jenkins we stop checking demand15:43
clarkbwe have already calculated demand more in the last couple minutes than we did in the whole 1400 hour15:44
TheJuliaSpamapS: greghaynes: wrt building your own IPA images, 218879 is required to fix the dib element15:44
*** Piet has joined #openstack-infra15:44
clarkbfungi: I am going to investigate the simple act of adding a timeout there15:46
fungiclarkb: thanks, sounds promising anyway15:46
*** soren has quit IRC15:47
dimsfungi: clarkb: anyone report "Intel PCI CI" out of control yet?
pabelangerclarkb: okay, thanks for the heads up15:49
clarkbfungi: looks like its doable so once I get breakfast I can whip something up for that then investigate how I broke centos7 image builds on hpcloud15:49
dimslogging failures every few mins15:49
fungidims: nope, you're the first caller15:49
dimsccarmack reported it on #openstack-qa15:49
*** ccarmack has joined #openstack-infra15:49
*** amotoki has quit IRC15:50
dimsfungi: looks like it's hitting other reviews too :(
fungianteaya: feel up for writing to the tpci-ann ml about this while i deactivate the account? example
fungidims: yep15:51
ccarmackHi, can someone help me with Intel PCI CI?  I uploaded a patch set, and now that CI is going crazy with failures.
anteayafungi: yes gerrit id 1575115:51
*** Sukhdev has joined #openstack-infra15:51
*** jistr|call is now known as jistr15:53
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory
fungii've deactivated their account, flushed gerrit's caches and am seeing if i can find and disconnect any open connections they currently have15:54
*** glauco has quit IRC15:56
*** glauco has joined #openstack-infra15:56
*** mrmartin has joined #openstack-infra15:56
*** stevemar has joined #openstack-infra15:57
anteayafungi: thank you15:57
anteayaI will send the email15:57
anteayaccarmack: thank you for speaking up15:57
fungii see two open connections in gerrit from the intelpcici username, from different source addresses15:59
ccarmackanteaya:  no problem, I see that dims had the same problem, and now the emails seemed to have stopped15:59
*** sigmavirus24 is now known as sigmavirus24_awa15:59
anteayaccarmack: great, thank you for that feedback15:59
*** jsavak has quit IRC15:59
*** annegentle has quit IRC15:59
*** wshao_ has quit IRC15:59
*** jtomasek has quit IRC15:59
fungihrm, no close-connection api command in 2.916:00
*** hdd has quit IRC16:00
*** markvoelker_ has joined #openstack-infra16:00
anteayaoh zaro backported it16:00
anteayaare we on 2.9?16:00
*** markvan_ is now known as markvan16:00
*** jsavak has joined #openstack-infra16:00
anteayaour gerrit 2.9 has zaro's backport I think16:00
*** sshnaidm has quit IRC16:00
fungier, 2.8 i meant16:01
*** puranamr has quit IRC16:01
anteayayeah, not in 2.816:01
anteayais is a 2.11 feature16:01
fungiyeah, looking to see if there's some other way to terminate specific connections in gerrit16:01
anteayabut can be backported to 2.10 and 2.916:01
*** markvoel_ has joined #openstack-infra16:01
anteayafungi: go you16:01
*** sridhar_ram has joined #openstack-infra16:01
*** sergmelikyan has joined #openstack-infra16:01
fungimay just have to wait it out, or perhaps restart gerrit16:01
*** sigmavirus24_awa is now known as sigmavirus2416:01
clarkband we cant 2.9 2.10 or 2.11 until jgit works again16:02
*** bexelbie has quit IRC16:03
fungiwell, anyway it looks like intelpcici hasn't left any new review comments since 1600 utc, so several minutes now16:03
*** markvoelker has quit IRC16:03
*** dannywilson has joined #openstack-infra16:04
fungiactually looks like 15:53 was the most recent16:04
fungiso about the time i deactivated them16:04
*** ivar-lazzaro has quit IRC16:04
*** baoli has quit IRC16:04
clarkbfungi for ssh slaves looks like latest release does include the fix. maybe we should put 04 in shutdown mode? (thats the one running newer than default already)16:04
*** trown is now known as trown|lunch16:04
sdaguefungi: thanks for promoting those patches16:04
*** baoli has joined #openstack-infra16:05
sdaguelets see what the dust settles / looks like after all of this16:05
anteayafungi: thanks16:05
*** markvoelker_ has quit IRC16:05
anteayaclarkb: yeah jgit :(16:05
*** baoli has quit IRC16:05
anteayaccarmack: ^^16:06
*** dkranz has quit IRC16:06
*** markus_z has quit IRC16:06
*** hdd has joined #openstack-infra16:07
*** akraminsky has joined #openstack-infra16:07
*** mriedem_away is now known as mriedem16:08
ccarmackanteaya:  sorry, could you explain "please address your system"?  Is that a message to the intel pci owner?16:08
*** hashar_ has quit IRC16:08
*** matrohon has quit IRC16:08
anteayaccarmack: yes16:08
ccarmackok, cool16:08
anteayait means fix yourself16:09
*** flashgordon is now known as jogo16:09
*** dkranz has joined #openstack-infra16:10
fungiheh... "go fix yourself"16:10
fungiclarkb: yeah, that sounds like a great idea16:10
fungiwant me to?16:10
*** sridhar_ram1 has joined #openstack-infra16:10
anteayafungi: should I use that next time?16:10
anteayaI'm collecting phrases16:10
anteayaI don't seem to have hit on the winning combination yet16:11
SpamapSTheJulia: ACK, thanks16:11
clarkbfungi: I just about have a jenkins api timeout patch ready have to make it pep8 happy16:11
*** flepied has joined #openstack-infra16:12
clarkbthis line is one character too long :/16:12
*** bharathm has joined #openstack-infra16:12
*** sridhar_ram has quit IRC16:12
fungiclarkb: awesome. i'll start quiescing jenkins04 while you adjust your terminal width16:12
*** kiran-r has joined #openstack-infra16:13
*** glauco has quit IRC16:13
fungijenkins04 is in prepare for shutdown now, eta ~1.1 hours i thinkl16:13
*** jistr has quit IRC16:13
clarkbfungi: once its ready you can just upgrade via the gui to get the latest and any dep changes if necessary16:14
clarkbfungi: itwill show you what it upgraded in that case whcih we can then more automagically apply to the other masters16:14
fungiclarkb: yep, since there's a release with it. will do16:14
*** spzala has joined #openstack-infra16:15
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient
*** glauco has joined #openstack-infra16:15
*** armax has joined #openstack-infra16:15
*** fedexo has quit IRC16:15
*** glauco has quit IRC16:16
*** mpaolino has quit IRC16:16
*** bharathm has quit IRC16:16
*** tmorin has left #openstack-infra16:17
*** vtapia has quit IRC16:17
openstackgerritFabien Boucher proposed openstack-infra/puppet-httpd: Only run http_mod for Debian os family for mod/*.pp
*** vivekd has joined #openstack-infra16:18
*** tiswanso has quit IRC16:19
openstackgerritFabien Boucher proposed openstack-infra/puppet-httpd: Only run http_mod for Debian os family for mod/*.pp
*** e0ne has quit IRC16:20
*** glauco has joined #openstack-infra16:20
openstackgerritMerged openstack-infra/groups: Install ruby-bundler during bootstrapping
mmmporki just added some folks to the review, but if anyone has a chance, I'd love if someone would review my nodepool testing doc changes -
*** sridhar_ram1 is now known as sridhar_ram16:21
clarkbfungi: and now pypi is being cranky at me installing deps ... anyways its being worked on16:21
*** jaypipes has joined #openstack-infra16:22
*** skylerberg has joined #openstack-infra16:22
*** tiswanso has joined #openstack-infra16:23
*** glauco has quit IRC16:23
*** sergmelikyan has quit IRC16:23
*** zeih has joined #openstack-infra16:23
anteayaclarkb: :(16:24
*** Swami has joined #openstack-infra16:24
*** glauco_ has joined #openstack-infra16:24
*** salv-orl_ has joined #openstack-infra16:25
*** unicell has quit IRC16:25
*** sergmelikyan has joined #openstack-infra16:26
greghaynesTheJulia: Cool, I think I got past all my PEBKAC, now on to infra-cloud specific issues16:26
SpamapSgreghaynes: o/16:27
greghaynesSpamapS: One thing im running in to now is that glean seems to be too willing to overwrite our eni confic with dhcp16:27
*** harlowja_at_home has quit IRC16:28
*** zeih has quit IRC16:28
*** salv-orlando has quit IRC16:28
SpamapSgreghaynes: so if we're using inventory_dhcp, we should not add network info16:28
SpamapSI didn't think of that before, but it seems imperative.16:28
greghaynesSpamapS: Yep, I have bifrost doing that - IIRC glean assumes no config = you want dhcp16:28
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: update signature for another expression
SpamapSyeah lets fix that16:28
SpamapSthats WAY too presumptuous16:29
SpamapSare you sure we're not just accidentally including dhcp-all-interfaces or something?16:29
SpamapSwhich actually respects existing configs IIRC16:29
greghaynesSpamapS: not 100%, I need to find it in the glean code real quick...16:29
*** tqtran has joined #openstack-infra16:30
mordredgreghaynes, SpamapS: so - the reason for that16:30
SpamapSI think glean should just leave any interface already configured alone.16:30
greghaynesyea, thatll break our current HP setup16:31
greghaynesi think16:31
*** markvoel_ has quit IRC16:31
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: update signature for another expression
*** ociuhandu has joined #openstack-infra16:31
*** ashleighfarnham has joined #openstack-infra16:31
mordredwe can't make an image that has dhcp interfaces configured and then also tell glean to ignore configured interfaces16:31
*** ashleighfarnham has quit IRC16:31
mordredbecause we have one cloud where eth0 is static and one where it's dhcp16:31
openstackgerritDavid Lyle proposed openstack/requirements: Changing the bounds for Django
*** ashleighfarnham has joined #openstack-infra16:31
mordredand in one of those config-drive does not include any network info16:31
*** markvoelker has joined #openstack-infra16:31
greghaynesmordred: what if we change it to not overwrite in that instance, only to write out an interface which doesnt exist16:32
openstackgerritDavid Lyle proposed openstack/requirements: Bumping django_openstack_auth
mordrednot overwriting I think is a fine idea16:32
mordredbecuase our current workflow is ...16:32
mordredoh, wait16:32
*** markvoelker_ has joined #openstack-infra16:32
mordredI tihnk the reason we weren't doing that is that udev wouldn't make the thing in the first place if there wasn't a file at least to start with16:32
*** sridhar_ram has quit IRC16:32
mordredor something wouldn't get the event triggered16:32
mordredI _fully_ support this being better16:33
*** alexsyip has joined #openstack-infra16:33
mordredjust trying to vomit the reasons I remember for why it is the way it is16:33
greghaynesyea, I remember us adding this for a reason16:33
mordredgreghaynes: what's the tl;dr on the current problem?16:33
mordredgreghaynes: what's inventory_dhcp?16:33
greghaynesmordred: Were baking bridge + vlan eni config into our infra-cloud images so they boot and dhcp on correct interfaces16:33
*** lifeless has quit IRC16:34
mordredrather than passing that info via config drive16:34
*** vtapia has joined #openstack-infra16:34
greghaynesmordred: its 'your hosts get ip from dhcp'16:34
*** ramashri has joined #openstack-infra16:34
greghaynesyes - because even with the vlan config-drive patch we need to then alter the interface post-glean so its on a bridge, not just a vlan16:34
mordredthat's why I was originally suggesting finishing teaching glean about the dhcp and vlan options that config drive can pass, and then teaching bifrost to pass in vlan info with "please use dhcp" in the config drive stanza16:34
greghayneswhich would mean some puppet that operates with networking down16:35
SpamapSoh, well that seems like a rather smooth way forward16:35
*** annegentle has joined #openstack-infra16:35
SpamapSexcept, we want to DHCP on the bridge16:36
SpamapSnot the interface16:36
greghaynesyes, exactly16:36
*** markvoelker has quit IRC16:36
SpamapSgreghaynes: were we struggling with glean's vlan stuff?16:36
SpamapSgreghaynes: because maybe we are violating layers here16:37
greghaynesSpamapS: no, I got the bifrost patch done16:37
mordredSpamapS: glean doesn't grok bridges at all16:37
mordredSpamapS: yeah16:37
SpamapSgreghaynes: maybe we should just let glean setup vlan25, and then just always run post-glean16:37
*** markvoelker_ has quit IRC16:37
jklarehi again, anybody got a moment to help me with a periodic job that does not run at all?16:37
mordredlike, have glean set up eth0 - and then have post-glean move the interface into the bridge16:37
SpamapSso add to the image a post-up.d that moves the network info to the bridge16:37
*** Hal has joined #openstack-infra16:37
mordredSpamapS: ++16:38
mordredI think that's the right separation of concerns16:38
mordredSpamapS: while we're gleaning:
mordredSpamapS: I'd love to land that and cut a new glean16:38
SpamapSgreghaynes: perhaps chalk this up to another educational failure? ;-)16:38
*** geoffarnold has joined #openstack-infra16:39
fungijklare: what periodic job? is it filtered (to a specific branch or certain set of files)? if so, that's the most common cause16:39
greghaynesok, I wanted to do this because it is very low effort and I think itll 'just work'... how about I deploy crinkle's two hosts like this real quick and you can work on the post-up bit?16:39
*** sergmelikyan has quit IRC16:39
greghaynesand then ill poke at bifrost + vlan patch16:39
*** Hal has quit IRC16:41
*** lifeless has joined #openstack-infra16:41
greghaynesSpamapS: other thought - with the glean + vlan what subnet do we want the ironic agent to run over?16:41
greghaynesstill untagged?16:41
SpamapSgreghaynes: yeah lets get the hosts that help people move forward in parallel up for sure16:42
SpamapSgreghaynes: that seems like the simplest thing yeah16:42
greghaynesI guess it has to otherwise we have to edit the ramdisk16:42
clarkbmmmpork: reviewed16:42
greghaynesok, well then that was a lot of the work I was doing so not a big waste ;)16:42
SpamapSunless we want to start adding glean to IPA's16:42
SpamapSwhich I don't think we do! ;)16:42
jklarefungi yeah, acutally it is, but i do not understand why it is getting filtered (let me give you the link....16:43
krotscheckAnyone here have opinions on languge style guides?16:43
fungijklare: the periodic pipeline is changeless and branchless, so adding any filters to the job will cause it not to be selected16:43
*** Hal has joined #openstack-infra16:43
greghaynesSpamapS: TheJulia ok, one patch I need to write up for bifrost - in a lot of places its using ansible's default network interface to determine the controller IP, I think that needs to be exposed as a setting so we can force it to use the 10.x address rather than 15.x16:43
mmmporkclarkb: thanks! i'll look over the changes and submit revisions!16:43
krotscheckIf so, go look at these and weigh in :),n,z16:43
mmmporki figured there'd be a few problems ;D16:43
*** arxcruz has quit IRC16:44
greghaynesSpamapS: TheJulia Right now I have that hacked in, but that needs to happen with either config_drive or inventory_dhcp16:44
*** jsavak has quit IRC16:44
clarkbmmmpork: I think the biggest one is dont use git review -x and use -d instead16:44
*** jsavak has joined #openstack-infra16:44
clarkbmmmpork: -x is just problematic if you don't understand what it is doing so I don't like putting it in docs as a thing people will blindly copy pasta16:44
jklarefungi ah, so i have to figure out a way to not filter this job... but what if it can only run for newer branches?16:44
fungijklare: the way we usually solve that is to have branch-specific job names for the periodic pipeline (such as gate-openstack-chef-repo-chef-rake-integration-kilo and gate-openstack-chef-repo-chef-rake-integration-master) by using a branch-designator parameter in the job-template to vary its name16:45
jklarefungi ok, i will try this and put in a patch later16:45
greghaynesSpamapS: I'm guessing that is what you meant by 'split up bifrost networking between deploy and agent networks'?16:46
fungijklare: basically, the zuul periodic pipelines, because they're not tied to proposed changes, lack context for things like branches so you have to specify the branch to run within the job definition16:46
jklarefungi i will add you as a reviewer if thats ok?16:46
mmmporkclarkb: thanks! i'm glad that the mistakes were things i brought over from the README and not my actual things haha16:46
fungijklare: sure, that's fine, though i think most of the project-config-core reviewers are familiar with the concept16:46
mmmporkso thanks for taking the time to look at that, i'll make those fixes and resubmit16:46
*** davideagnello has joined #openstack-infra16:47
*** jerryz has joined #openstack-infra16:47
SpamapSgreghaynes: Yes16:47
SpamapSgreghaynes: in places it is trying to have them separate, but it's not complete.16:48
openstackgerritClaudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements
greghaynesah, ok16:48
*** ayoung has quit IRC16:48
*** davideagnello has quit IRC16:49
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds os-win repository
SpamapSgreghaynes: so yeah, get crinkle some boxes to test the openstack-itself puppet on. And I'll poke at the glean vlan patch and maybe a post-up.d that will move vlan25's network info to a bridge. Or did you already have the bridge script?16:50
*** davideagnello has joined #openstack-infra16:50
greghaynesSpamapS: I dont have a bridge script, no16:50
*** yamahata has joined #openstack-infra16:50
*** tqtran has quit IRC16:51
greghaynesSpamapS: If you want to poke at the vlan patch - I am cycling on node 2e251e9d-0b00-477c-9069-f9dbe966b5e3 - you could cycle on any of the others16:51
openstackgerritClark Boylan proposed openstack-infra/nodepool: Add a Jenkins API timeout.
clarkbfungi: ^ note that is completely untested by any of our testing :/16:52
*** sdake_ has joined #openstack-infra16:52
*** jsavak has quit IRC16:53
*** jsavak has joined #openstack-infra16:53
*** wenlock_ has joined #openstack-infra16:54
clarkbgah that needs voluptuous updates too16:55
*** dims has quit IRC16:55
openstackgerritClark Boylan proposed openstack-infra/nodepool: Add a Jenkins API timeout.
*** salv-orl_ has quit IRC16:56
docaedovoluptuous updates are the best kind16:56
zaroianw: any reason this hasn't merged?
zaro* merged/approved16:58
*** wenlock_ has quit IRC16:59
*** dizquierdo has joined #openstack-infra17:00
*** puranamr has joined #openstack-infra17:01
derekhAre their other issues with the new netaddr package ? we're seeing this on tripleo
openstackLaunchpad bug 1490639 in tripleo "Floating IP's no longer working" [Undecided,New]17:01
*** earlephilhower has joined #openstack-infra17:02
*** dannywilson has quit IRC17:02
*** bhunter71 has joined #openstack-infra17:02
derekhnevermind, I see neutron is on it17:03
SpamapSgreghaynes: +1 thanks17:03
ihrachysderekh: it's just fixed in neutron17:04
derekhihrachys: yup, thanks, I just noticed it after I posted my message17:05
*** ihrachys has quit IRC17:05
*** derekh has quit IRC17:05
openstackgerritJan Klare proposed openstack-infra/project-config: add new periodic job for chef integration testing
*** rossella_s has quit IRC17:05
*** trown|lunch is now known as trown17:06
*** rossella_s has joined #openstack-infra17:06
*** sridhar_ram has joined #openstack-infra17:09
*** kiran-r has quit IRC17:10
*** julim has joined #openstack-infra17:11
fungiNakato: if you need any cooperation from individual project-teams (particularly neutron!) on the tox/unit test constraints work, would be a great place to chime in17:11
*** tonytan4ever has quit IRC17:12
*** unicell has joined #openstack-infra17:12
openstackgerritAugustina Ragwitz proposed openstack-infra/nodepool: Improve testing and initial installation documentation
mmmporkclarkb: i made the corrections you suggested! thanks so much17:13
*** ybathia has joined #openstack-infra17:14
jklarefungi is there any documentation on these 'branch designators' in job templates?17:14
sdake_gmann what can I do for you17:14
greghaynesclarkb: Hey - so you said growroot worked and tests passed, are we just waiting on more reliable uploads for ubuntu-trusty?17:15
clarkbjklare: no and they are poorly named should just be called 'suffix' or similar17:15
clarkbjklare: they are mostly used to duplicate jobs with uniue names so that we can change behavior in zuul17:15
clarkbjklare: non voting, specific branch, test against source of another project, etc17:15
clarkbgreghaynes: we are17:15
greghaynesclarkb: hrm, ok. I wonder if we should try another image type too now?17:16
clarkbgreghaynes: centos7 maye17:16
clarkbor f22 depending on how brave you are17:16
greghaynestwo of my favorite things17:17
*** sergmelikyan has joined #openstack-infra17:17
*** achanda has joined #openstack-infra17:18
fungijklare: clarkb: though in the current case they are aptly named since the desire seems to be to designate the branch on you want to run the job when in the periodic pipeline17:19
fungi(which was also the original use for that parameter, hence its name)17:20
*** madhu_ak has joined #openstack-infra17:20
*** tnovacik has quit IRC17:22
*** vivekd has quit IRC17:23
sdagueit appears that rax has added some really slow nodes17:23
*** mjturek1 has quit IRC17:23
*** zeih has joined #openstack-infra17:24
sdaguehmmm... actually, it's being under a lot of load ?17:25
sdaguefungi / clarkb - so workspace setup is now often taking 40 minutes? -
*** e0ne has joined #openstack-infra17:26
*** hdd has quit IRC17:27
*** zeih has quit IRC17:28
greghaynes a lot of time on that line... what is that doing?17:28
greghaynesjust a super slow clone?17:29
*** dims has joined #openstack-infra17:29
*** amuller is now known as amuller_afk17:29
fungisdague: got so bad it was unresponsive for 20 minutes17:29
sdague2015-08-31 16:48:13.542 | + git_fetch_at_ref openstack/heat refs/zuul/master/Zcd98780d74ad49518c70397c8e2c3912 - took 90 seconds, for instance17:29
sdaguefungi: ok, yeh, that's triggering a ton of fails on jobs from timeouts17:29
sdaguehowever, I don't think it's just the last 20 minutes17:30
*** dims_ has joined #openstack-infra17:30
jeblairwe may need to clean out the refs17:30
*** dims has quit IRC17:30
*** hdd has joined #openstack-infra17:30
sdague - started hitting again 4 days ago17:30
sdagueout of the blue17:30
*** mjturek1 has joined #openstack-infra17:30
fungicompare to for zm0117:31
sdagueright, so zm02 is way slower anyway17:31
*** ildikov has quit IRC17:31
sdagueas zm01 is rarely getting above loadavg 217:31
*** larainema has quit IRC17:32
sdagueand zm02 is averaging load avg 7 over that time period17:32
jeblairthey have approx the same number of refs in nova17:33
*** sdake has joined #openstack-infra17:33
sdaguejeblair: how about heat?17:34
sdaguethe nova fetch was actually super speedy17:34
clarkbit does the remote branch listing thing to prune stale branches iirc17:35
*** sthillma has joined #openstack-infra17:35
clarkbthat is slow when you have a bunch of refs17:35
*** baoli has joined #openstack-infra17:35
*** jaosorior has quit IRC17:35
jeblairsdague: about the same too (about 25% of nova)17:35
*** stevemar has quit IRC17:35
sdagueyeh, I don't know. Any chance it's just a noisy neighbor problem?17:36
*** e0ne has quit IRC17:36
*** tonytan4ever has joined #openstack-infra17:36
jeblairsdague: that's possible :/17:36
*** hdd has quit IRC17:36
*** hdd has joined #openstack-infra17:37
*** pvaneck has joined #openstack-infra17:37
sdaguebecause, it's definitely gotten a ton worse over the last 4 days17:37
*** baoli has quit IRC17:37
sdaguethis typically ends up triggering failures in the jobs because they run out of time17:37
*** sdake_ has quit IRC17:37
sdaguelike large-ops, which I think is 90 minutes total timeout17:37
sdaguebut when we spend 40 minutes in setup workspace, you're toast17:37
*** baoli has joined #openstack-infra17:38
jeblairthere's a definite memory/swap spike at the same time; unsure whether that's cause or effect17:38
*** stevemar has joined #openstack-infra17:38
*** e0ne has joined #openstack-infra17:39
openstackgerritAugustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting
*** bharathm has joined #openstack-infra17:41
mmmporkgreghaynes: ^^17:41
*** stevemar has quit IRC17:42
*** sputnik13 has quit IRC17:43
*** ig0r__ has joined #openstack-infra17:43
openstackgerritSean Dague proposed openstack-infra/devstack-gate: warn when workspace setup takes a long time
*** ashleighfarnham has quit IRC17:44
sdagueok, I'm proposing we add a timing warning there as well that we can directly get a signature on17:44
sdaguethe devstack one is secondary17:45
sdagueand they are slow for different reasons17:45
sdagueso we should warn and track them separately17:45
openstackgerritChristian Berendt proposed openstack-infra/groups: Add a basic Vagrantfile for a development environment
clarkbthe times there don't line up for the centos7 switch fwiw17:45
clarkbcentos7 swtich happened early 29th UTC17:46
*** ig0r_ has quit IRC17:46
*** ivar-lazzaro has joined #openstack-infra17:46
clarkbalso I have a test change up that trims down the PROJECT list to make this less bad17:47
*** mjturek1 has quit IRC17:47
clarkbit will take a lot of work to do on the JJB side though I think17:47
pvaneckjeblair: Hi, when you get a chance, could you review ?17:49
*** ashleighfarnham has joined #openstack-infra17:50
madhu_akdougwig: could you please have a look at this patch:
*** MIDENN_ has joined #openstack-infra17:50
jeblairpvaneck: ack17:51
*** doug-fish has quit IRC17:51
*** doug-fish has joined #openstack-infra17:51
*** Sukhdev has quit IRC17:51
*** FallenPegasus has quit IRC17:53
*** mjturek1 has joined #openstack-infra17:53
jeblairfungi, sdague: the spike on zm02 has a good timing correlation with the several hundred nods launched when fungi restarted nodepool17:56
sdaguejeblair: ok, except the failure window here is not limitted to just that spike17:57
*** shayneburgess has left #openstack-infra17:59
*** sputnik13 has joined #openstack-infra18:00
*** FallenPegasus has joined #openstack-infra18:00
*** jeadre has quit IRC18:01
fungiyeah, zm03 for example saw a spike too, but its load average only jumped to 30 not 14018:01
*** jeadre has joined #openstack-infra18:01
fungisuggesting that zm02 is suffering some other performance degredation, perhaps18:02
*** stevemar has joined #openstack-infra18:03
*** salv-orlando has joined #openstack-infra18:04
*** shayneburgess has joined #openstack-infra18:05
*** abdelwas has joined #openstack-infra18:06
*** FallenPegasus has quit IRC18:06
*** davideagnello has quit IRC18:06
jeblairsdague: looking at logstash links for the "slow node" query, the jobs are not simple jobs.  they are largely multinode or ironic-pxe_ssh18:06
jeblairit may be that "merger load spike" and "appearance of jobs triggering slow node query" are different issues18:07
clarkbmultinode with the serial devstacking may trip that more18:08
sdaguejeblair: yeh, unfortunately the timeouts that seem to be hurting us are the largeops jobs18:08
sdaguegives us the timings on the workspace create18:08
sdaguewhich is what will be impacted by the zm02 slowness18:09
*** sergmelikyan has quit IRC18:09
sdaguethe devstack slow warning isn't really a great indicator18:09
*** salv-orlando has quit IRC18:09
jeblairsdague: oh, are you saying you have noticed in increased incidence of large-ops failing in this manner?18:09
*** dizquierdo has quit IRC18:09
jeblairi thought the only trend we had was the slow-query18:10
sdaguejeblair: yes, sorry, that's why I went down this rabbit hole18:10
sdaguelarge-ops failures18:10
jeblairsdague: have any more handy?18:11
sdaguenot atm18:11
sdaguewas chasing cinder unit test fails18:12
openstackgerritChristian Berendt proposed openstack-infra/groups: Add a basic Vagrantfile for a development environment
fungiclarkb: jenkins04 is idle now. i'll give upgrading the plugin a go18:13
clarkbfungi: thanks let me know if  Ican help18:13
jeblairfungi, clarkb: oh, what plugin?18:13
clarkbupstream puppet wildfly isn't a huge fan of my update to make upgrades possible :/ I have responded and hopeflly they turn around18:14
clarkbjeblair: ssh-slaves18:14
clarkbjeblair: there was a recent (this month) update to the plugin to fix a deadlock situation which we think may be our problem18:14
jeblairclarkb: is it sposed to fix the leaks?18:14
clarkbjeblair: it is supposed to fix a deadlock, not really sure if its ourdeadlock18:14
fungipabelanger: ianw: i just saw someone mention anecdotally on the distutils-sig ml in a packaging thread that "dnf 1.1.0 just fixed package download caching"18:14
clarkbbut seems worht a shot18:14
asselin_clarkb, fungi let me know if I can help test on my system as well18:14
*** FallenPegasus has joined #openstack-infra18:15
fungijeblair: theory being that we delete the slave before the cleanup finishes in jenkins, and it leaves a stuck thread behind trying to ssh to the slave to complete teh cleanup18:15
clarkbasselin_: we hit it regularly enough taht we should be able to test it effectively on our master(s) but feel free to upgrade ssh-slaves plugin to latest and see if it fixes on your end18:15
jeblairfungi: nice18:15
fungijeblair: the patch in question implements a timeout for that operation, allowing the thread to terminate gracefully18:15
pabelangerfungi: ORLY?18:16
pabelangerchecking it out now18:16
*** pelix has quit IRC18:17
*** markvoelker has joined #openstack-infra18:17
*** hashar has joined #openstack-infra18:18
jeblairsdague: typo in 21896618:18
*** julim has quit IRC18:18
clarkbfungi: the thing to keep an eye on is the thread count in melody and specifically see if the forremoting thread count skyrockets18:19
clarkb(that is a case insensitive search string)18:19
*** hdd has quit IRC18:20
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: add signature for cinder pyopenssl issue
*** sergmelikyan has joined #openstack-infra18:20
pabelangerfungi: ianw: looks like they added back the --downloadonly flag18:20
*** ildikov has joined #openstack-infra18:20
pabelanger bug 1048433 in dnf "RFE --downloadonly" [Unspecified,New] - Assigned to packaging-team-maint18:20
pabelangergoing to give it a test now18:21
*** markvoelker has quit IRC18:21
*** julim has joined #openstack-infra18:21
*** b10n1k_ has joined #openstack-infra18:21
*** FallenPegasus has quit IRC18:21
sdaguejeblair: looking18:21
fungithe road to happiness is paved with small victories18:21
*** jtomasek has joined #openstack-infra18:22
fungi(or something)18:22
openstackgerritSean Dague proposed openstack-infra/devstack-gate: warn when workspace setup takes a long time
sdaguejeblair: good catch, thanks18:23
fungiclarkb: looks like the ssh slaves plugin was the only one it downloaded when updating18:23
*** hashar has quit IRC18:23
clarkbfungi: great, I know in the last upgrade it updated another plugin18:24
*** zeih has joined #openstack-infra18:25
*** davideagnello has joined #openstack-infra18:25
*** spzala has quit IRC18:28
*** MarkAtwood has quit IRC18:28
*** dtantsur|afk is now known as dtantsur18:29
*** zeih has quit IRC18:29
*** FallenPegasus has joined #openstack-infra18:30
fungiclarkb: ssh slaves plugin 1.10 was what you were expecting, right?18:31
clarkbfungi: ys18:32
clarkbfungi: it appears to have released right after that bugfix and according to changelog includes the deadlock fix18:32
fungiawesome. fingers crossed18:32
fungiuncrossed now. made typing too hard18:32
*** gyee has quit IRC18:33
fungithe clouds favor us. we're getting pretty decent node utilization for a change18:33
openstackgerritAugustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting
clarkbfungi: right now 04 has 8 forremoting threads18:34
clarkbfungi: I think <100 is normal and a few thousand is definitely broken18:35
*** mjturek1 has quit IRC18:36
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: Add python-gnocchiclient
fungiclarkb: cool. probably have to give it a few days for confirmation unfortunately18:36
fungibut here's hoping18:36
*** sputnik13 has quit IRC18:37
*** FallenPegasus has quit IRC18:37
*** tonytan4ever has quit IRC18:38
*** tonytan4ever has joined #openstack-infra18:38
openstackgerritChristian Berendt proposed openstack-infra/groups: Add a basic Vagrantfile for a development environment
*** tonytan4ever has quit IRC18:38
*** ayoung has joined #openstack-infra18:38
fungimy dyslexia saw "add a balsamic vinaigrette file"18:39
jeblairfungi: hungry?18:39
clarkbpour over your salad18:39
*** salv-orlando has joined #openstack-infra18:39
anteayaI'm actually making a salad18:41
anteayablackbean and pink grapefruit18:41
*** mrmartin has quit IRC18:42
fungithat sounds like a most excellent salad18:43
*** mjturek1 has joined #openstack-infra18:43
fungii do like fruit and bean salads18:43
*** pcm_ has joined #openstack-infra18:43
fungia generally under-appreciated category of saladness18:43
anteayawith walnuts18:45
*** pc_m has quit IRC18:45
anteayaand fresh basil if it is around18:45
clarkbfritos, beans and salsa18:45
*** FallenPegasus has joined #openstack-infra18:46
anteayaha ha ha18:47
anteayathere's that too18:47
*** juggler has quit IRC18:47
pabelangerfungi: okay, so there is a fix for --downloadonly, but not merged into dnf yet18:47
fungipabelanger: does it look like its future is promising, or are the primary dnf devs balking at it?18:48
pabelangerfungi: unsure18:48
pabelangergoing to try and loop myself into the discussions18:48
*** ig0r__ has quit IRC18:48
fungipabelanger: at any rate, --downloadonly may be separate from fixing download caching18:48
clarkbpabelanger: looking at puppet logs nothing about firewalld is logged18:49
clarkbpabelanger: so I wonder if facter is giving us different data and making things go wonky18:49
fungipabelanger: since ianw indicated that dnf's current package caching is useless/nonexistent18:49
fungi(or was, perhaps until 1.1.0)18:49
*** jeadre has quit IRC18:49
*** jeadre has joined #openstack-infra18:50
clarkbpabelanger: but it does claim to install the package, the two rules files (ipv6 and ipv4) then it tries to "start" the service to apply them and that fails18:50
*** FallenPegasus has quit IRC18:50
*** mjturek2 has joined #openstack-infra18:52
glauco_is there a major concern on migrating to use puppetlab official vcsrepo?
pabelangerfungi: Ya. We had some thought about using squid, but unsure if that is the right approach either18:52
glauco_we noticed that openstack version is quite outdated and we would like to use the depth param to speed up some configurations18:53
pabelangerclarkb: Odd. I don't have an HP account otherwise, I'd try and poke around too18:53
*** mjturek1 has quit IRC18:53
clarkbpabelanger: ya I intend on booting a node and poking around18:53
*** stevemar has quit IRC18:54
clarkbglauco_: it didn't work way back when and mordred made changes that were not compatible with upstream to make it work iirc18:54
clarkbglauco_: in general I advise against shallow clones as they come with a large number of caveats and they are broken on ubuntu trusty for example18:54
clarkbglauco_: its possible that the upstream module would work today, would need to test it18:55
mordredclarkb: aroo?18:55
clarkbmordred: puppet vcsrepo18:55
clarkbmordred: glauco_ is wondering if we can use upstream again18:55
mordredah. yeah. it's designed around a different use case18:55
glauco_clarkb: for some context, we noticed that zuul installation is taking a long time because it is checking out the whole twitter bootstrap repo18:55
*** edmondsw_ has joined #openstack-infra18:56
glauco_would using a tarball generated by github be a better alternative to shallow git repos?18:56
*** edmondsw_ has quit IRC18:56
mordredthe upstream one did not do what we considered to be the right thing related to wanting ot always have master, a tag or a revision in place18:56
nibalizerglauco_: i'd like to see us get to a modern vcsrepo18:56
*** edmondsw has joined #openstack-infra18:56
mordredand goes out of its way to not overwrite a repo state if a repo is a state other than what's declared18:56
nibalizerI agree with clark that shallow clones save you run-time at the cost of weird-stuff-happened-weridly-debug-time18:56
mordredwhich is useful for laptops18:56
mordredbut pretty bad for servers18:56
fungiperhaps the openstackinfra/zuul module should not continuously deploy bootstrap from git? we must be doing that for some reason but i don't know what the reason is18:57
*** edmondsw has quit IRC18:57
mordredwe also wanted to be able to specify a branch and have it keep it at tip of that branch18:57
*** edmondsw has joined #openstack-infra18:57
mordredto not report change if it had just fetched but didn't actually update the working tree18:57
nibalizerThe git module understands our needs and is git focused and simple18:57
mordredif upstream can do all of those things, I'm great with using it18:57
funginibalizer: yeah, back when we had more uses of hg and bzr than we do now18:58
openstackgerritMerged openstack/requirements: Bump oslo.versionedobjects version to 0.9.0
*** mrunge has quit IRC18:58
*** amuller_afk is now known as amuller18:58
*** sthillma_ has joined #openstack-infra18:58
*** ociuhandu has quit IRC18:58
nibalizerso if we were gonna do a big change i'd advocate for the git module, or maybe patching our vscrepo changes into the upstream vcsrepo18:58
nibalizerbut it also seems to work as is so18:58
dims_pcm_: since it got merged in global requirements, the proposal bot has already filed reviews for other projects that use netaddr (,n,z)18:59
*** sabeen1 has quit IRC18:59
glauco_ok, we'll check how is the support for upstream vcsrepo later on. thanks for sharing the concerns18:59
*** FallenPegasus has joined #openstack-infra18:59
*** sthillma has quit IRC19:01
*** sthillma_ is now known as sthillma19:01
mordrednibalizer: ++ to git - a lot of the problems we patched around had to do with the difficulty of finding the "right" semantics to express across the different VCS systems19:01
*** spzala has joined #openstack-infra19:01
mordredwhich, it turns out, is VERY hard19:01
mordredand I give props to upstream for getting it as close as they did19:02
*** dtantsur is now known as dtantsur|afk19:03
*** sdake has quit IRC19:03
*** jsavak has quit IRC19:05
openstackgerritMerged openstack-infra/puppet-gerrit: Add acceptance tests for puppet-gerrit
*** yfried has joined #openstack-infra19:06
*** FallenPegasus has quit IRC19:06
*** jsavak has joined #openstack-infra19:06
*** hashar has joined #openstack-infra19:07
*** Rockyg has joined #openstack-infra19:08
clarkbyolanda: nibalizer ^ that change failed both of its rspec jobs19:09
*** sergmelikyan has quit IRC19:09
clarkbyolanda: nibalizer should we be merging tests that dont pass?19:09
*** sdake has joined #openstack-infra19:10
*** sdake has quit IRC19:10
*** sdake has joined #openstack-infra19:10
*** ZZelle has joined #openstack-infra19:13
*** fitoduarte has joined #openstack-infra19:14
*** mjturek1 has joined #openstack-infra19:14
openstackgerritMerged openstack-infra/elastic-recheck: update signature for another expression
*** FallenPegasus has joined #openstack-infra19:15
*** mjturek2 has quit IRC19:15
rm_workHey, possible to get this workflowed today? Running out of time for testing for Liberty. :(
crinkleclarkb: probably not when the patch is for adding tests...19:17
*** hdd has joined #openstack-infra19:18
openstackgerritJohn Trowbridge proposed openstack/diskimage-builder: [ironic-agent] Use svc-map for enabling agent
*** FallenPegasus has quit IRC19:20
mordredclarkb: heads up - glanceclient 1.0 breaks shade19:22
mordredclarkb: it broke jlk doing something else - I'm investgating - but for now, let's not upgrade glanceclient on nodepool :)19:23
clarkbok and are fun ones19:24
*** baoli has quit IRC19:24
asselin_oh no... :(19:24
clarkbthats two differnet hosts in ord getting the same IP at the same time19:24
clarkblifeless: ^19:24
clarkbI suppose we can only file a bug against rax for that?19:24
*** baoli has joined #openstack-infra19:24
clarkband hope they can figure out what nova/neutron did19:25
*** mrmartin has joined #openstack-infra19:25
clarkbsuch cloud many IPs wow19:25
sdaguemordred: glanceclient 1.0 broken cinder unit tests, but only because they were implicitly using it's pyopenssl dependency19:25
greghaynesI wonder if cloud-utils does the dhcp-like thing where it tries to ping the IP before assigning it19:25
*** zeih has joined #openstack-infra19:25
anteayarm_work: done, nice work19:25
greghayneser, cloud-init19:25
sdagueclarkb: it's neutron with a custom backend there19:25
sdaguegreghaynes: no, cloud-init doesn't care19:26
sdaguecloud-init does what you told it19:26
clarkband in this case its their weird agent setting the IP19:26
greghaynesoh even better19:26
mordredsdague, clarkb: apparently there is now a problem with the "insecure" argument19:26
clarkbsdague: so not sure how we would fingerprint that other than look for same IP in use at the same time with some fuzzyness19:26
clarkbsdague: because we do safely reuse IPs over time19:26
glauco_clarkb: gerrit-puppet-beaker-rspec-dsvm-trusty fails because beaker sees a pending test as a failure19:26
PietWe posted a draft version of the cloud personas to invision19:27
PietFeel free to comment -
sdagueclarkb: so... does this mean that we've got 2 ssh client connections running 2 at the same time?19:27
clarkbsdague: yes19:27
rm_workanteaya: thanks19:27
clarkbsdague: and as far as nodepool is concerned they are different hosts19:27
sdagueso, could we stamp a file in d-g19:27
sdaguewhen it starts19:27
clarkbwe can probably defend against this ins nodepool by treating a dup IP in the db as an error19:27
sdagueand look for it when another one starts19:27
clarkbI may work on that patch today19:27
sdagueyeh, or that, in nodepool19:28
sdagueanyway, get defensive about it19:28
nibalizerclarkb: argh dangit, I thought those were the upgrade jobs that never pass19:28
*** FallenPegasus has joined #openstack-infra19:29
glauco_clarkb: gerrit-puppet-beaker-rspec-dsvm-centos7 fails to apply the existing puppet module.19:29
clarkbglauco_: regardless if your intent is to add tests they should make the job pass19:29
clarkbglauco_: otherwise its just noise19:29
clarkbglauco_: thats ok we expect them to pass on at least one of them right?19:29
nibalizerglauco_: ya why isn't htat passing on trusty19:29
nibalizerFailure/Error: apply_manifest(default_puppet_module, catch_failures: true) it's not a pending that is failing19:30
nibalizerhowever there isn't a failure in the last 10 lines of output19:30
*** zeih has quit IRC19:30
glauco_clarkb: sure! We could either approach it as: let's make this module idempotent or we could remove that test19:30
openstackgerritMerged openstack-infra/project-config: Add experimental job for Barbican to test updated gate
glauco_nibalizer: do you mean on trusty or centos7?19:31
glauco_on trusty the test that fails is the pending one19:31
madhu_akcan anyone workflow this patch?
glauco_on centos7 what fails is the 'apply' of the manifest19:31
clarkbrm_work: now that I am not debugging nodepool crazyness, what is the liberty concern? infra doesn't follow that schedule so we can get that experiemtanl test in any time19:32
*** sthillma_ has joined #openstack-infra19:32
clarkbrm_work: if however you are trying to test that particular thing before liberty cuts that different (but considering its experimental I am not sure if that is urgent)19:32
sdagueclarkb: so before I decide I need to start drinking, because my well pump adventure is going to have a day 2 (or 3 based on counting), blocking random inbound dhcp requests to nodes....19:32
rm_workclarkb: right but i need the experimental job so i can properly test a change in a project that DOES follow the freezes :P19:32
rm_workclarkb: but it's fine, anteaya +W'd it19:33
*** mrmartin has quit IRC19:33
rm_workclarkb: yeah, the latter19:33
mtreinishsdague: you can't drink and talk about blocking random inbound dhcp requests to nodes? They seem to go hand in hand :)19:33
glauco_clarkb: nibalizer: would you prefer removing that pending test?19:33
clarkbsdague: I realized that my octet idea won't work for the same reason its happening. l2 broadcast19:33
*** FallenPegasus has quit IRC19:33
clarkbsdague: so ebtables necessary or listening only on the private addr19:34
clarkbglauco_: I would prefer that we only merge test changes that pass some sort of testing19:34
clarkbglauco_: however its best to do that I defer to you and nibalizer and yolanda, I just don't want to see "add tests" and the tests fail19:34
*** sthillma has quit IRC19:34
*** sthillma_ is now known as sthillma19:34
clarkbsdague: or accept it as noise> would make it hard to debug real dhcp issues if/when they happen19:35
sdagueclarkb: so, I'm wondering if - is also double node use19:35
clarkbsdague: the other wierd dhcp case is multinode but it uses nova multihost so would not be affected19:35
pcm_hi! does any know what the magical incantation is for python-neutronclient to specify a list as an attribute?19:35
clarkbsdague: oh maybe I can dig through logs for that one too19:35
*** davideag_ has joined #openstack-infra19:35
sdaguebecause that log shows pbr being installed to 1.6.019:36
*** davideagnello has quit IRC19:37
openstackgerritAssaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron + functional bug 1478847
openstackbug 1478847 in neutron "test_udp_{input|output}.* fails with " AssertionError: RuntimeError not raised."" [High,In progress] - Assigned to Jakub Libosvar (libosvar)19:37
sdaguebut pbr freeze shows it at 0.11.019:37
clarkbsdague: will be easier to check once logstash catches up tags:"console" AND message:"inet" is the query19:37
*** salv-orlando has quit IRC19:37
clarkbbut let me dig through the logs and see if  Ican find another ansible like lifeless did19:38
*** ramashri has quit IRC19:40
clarkbsdague: there are two worlddump files, can that happen during a single test run?19:41
*** sharikapongubala has joined #openstack-infra19:41
*** jeadre has quit IRC19:41
anteayapcm_: oh that would be a neutron question I am thinking19:41
*** jeadre has joined #openstack-infra19:42
*** FallenPegasus has joined #openstack-infra19:42
*** bknudson has joined #openstack-infra19:42
pcm_anteaya: yeah but really applies to any client. If I have python-fooclient   bar-create name cidrs=?19:43
sdagueclarkb: yes19:43
*** hdd has quit IRC19:43
sdaguethere can be up to 6 I think19:44
sdagueI'm dumping at success points to try to build up data for the ping goes away issue that I don't understand19:44
*** hdd has joined #openstack-infra19:44
pcm_anteaya: Any cli client that needs to provide an attibute that is a list (e.g. converted into JSON {'cidr': ['', ''] ...}19:44
clarkbsdague: gotcha19:45
pcm_anteaya: I can't seem to figure out how to specify the command line representation for the list.19:45
clarkbsdague: so without more data on whta the other thing may be its hard to know if this was dup host again, but logstash should be able to tell uys when it catches up19:45
sdagueit seemed better to sort an iptables issue by seeing the tables when it worked19:45
sdagueand see if there was anything different when they did not19:45
clarkbhrm I should recheck the nodepool logs though because for the instance I did find nodepool wasn't really aware19:46
*** andersonvom has joined #openstack-infra19:46
anteayapcm_: let's move to -dev this isn't really an -infra question19:46
*** thorst1 has quit IRC19:47
*** andersonvom has quit IRC19:47
clarkbthe mystery gets more mysterious19:47
clarkbaccording to nodepool we booted and deleted this node yesterday19:47
fungi(you've probably already seen it, but HA!)19:48
clarkbI know what happened19:48
clarkbfungi: when you restarted jenkins01 the slave from the 30th was still in there19:48
clarkbfungi: then jenkins came back up and ssh'd to that IP address which was the new slave from today19:48
*** FallenPegasus has quit IRC19:49
clarkbfungi: it then ran a "second" job on the old host when in reality it was a second job on the new host with the shared IP19:49
clarkbthats magical19:49
fungiclarkb: wait, what?!?19:49
clarkbfungi: did you clera out the slave list on jenkins01 when you restarted it?19:49
*** yfried has quit IRC19:49
fungimaybe it happened in the split second before i stopped the service19:49
fungibut yes, i did19:49
*** Sukhdev has joined #openstack-infra19:49
clarkboh wait no because the timestamps don't match up for the restart19:50
*** salv-orlando has joined #openstack-infra19:50
fungii kept hammering ssh until i got in and immediately did a service stop jenkins, confirmed it wasn't running, then cleaned up the config.xml and started it again19:50
clarkbya I think I am wrong because you restarted well after these jobs ran19:50
openstackgerritMonty Taylor proposed openstack-infra/shade: Move glanceclient to new common interface
mordredSpamapS, Shrews: when you get sec ^^19:51
glauco_nibalizer: where are we defining beaker-rspec-* checks and gates? I can't find it anywhere other than project-config/zuul/layout19:51
sdagueclarkb: so a ton of nodes started failing on the pbr contextual conflict issue19:51
mordredSpamapS, Shrews: that does not work yet - there are more interface changes19:51
mordredI just wanted to loop you in that glanceclient is different now19:52
*** davideagnello has joined #openstack-infra19:52
glauco_nibalizer: I would like to see the command it is executing right now. Just noticed that puppet-gerrit will not going to work on centos because of the java_home settings19:52
*** DericHorn-HP has joined #openstack-infra19:52
mordredfungi, jhesketh, pleia2, SergeyLukjanov, jeblair, clarkb: nobody upgrade glanceclient on nodepool. it'll break hard19:52
clarkbso from the logs what I can tell happened is yesterday we made 4636852, and deleted it that day. Then today we made 4656491, a jenkins master was able to ssh into 4656491 thinking it was 4636852, then we ran two jobs on the same host because two different slaves were both ssh'd into it19:53
glauco_nibalizer: does deleting the nodeset 'nodepool-centos7.yml' from under nodesets folder makes it skip centos testing?19:53
clarkbglauco_: no19:53
*** davideag_ has quit IRC19:53
clarkbglauco_: it will run the job and fail, its fine if centos7 is known to fail for now19:53
*** jeadre has quit IRC19:53
clarkbglauco_: as long as those tests pass somewhere19:53
*** jeadre has joined #openstack-infra19:54
clarkbah more info19:54
clarkb4636852 didn't run a job19:54
*** stevemar has joined #openstack-infra19:54
clarkbso it was never offlined19:54
clarkbfungi: I have a hunch this is just jenkins derping, we try to delete the node and it doesn't actually delete19:54
clarkbfungi: then later when jenkins gets enough cpu cycles that aren't doing too many threads things it ssh's back into hosts19:55
clarkbfungi: and if they have never run a job before well its totally cool to run a job now19:55
clarkbso hopefully that plugin update fixes this :/19:56
clarkbsdague: I would be surprised if the pbr thing is this19:56
sdagueclarkb: ok, the pbr thing is spiking19:56
sdagueand it's really confusing19:56
sdaguebecause we install pbr 1.6.0 twice in most of those runs19:56
sdagueand it says it's 0.11.0 at the end19:57
clarkbsdague: if a project has a direct 0.11.0 depends it may be sucking it down19:57
*** bswartz has quit IRC19:57
sdagueand not ever tell us?19:57
clarkbsdague: if its via a lib dep install as a transitive thing it wouldn't surprise me19:58
clarkbpip logs is where I would look next19:58
mordredI was just about to suggest those19:58
clarkbwe don't seem to grab them on grenade runs19:58
sdagueit's not just grenade runs19:59
openstackgerritJim Rollenhagen proposed openstack/requirements: Add ironic-lib at initial release
sdaguethough those do get hit a bunch19:59
jrolldhellmann: ^ corresponding g-r if you're curious19:59
*** stevemar has quit IRC19:59
annegentlemordred: around?19:59
*** teran has quit IRC20:00
clarkbfungi: I am tempted to just go ahead and update the plugin across the board in the off chance it just fixes it and if it doesn't then we are no worse off (assuming its not riddled with even worse bugs too)20:00
mordredannegentle: yes - oh bother - is that today?20:01
*** david-lyle has quit IRC20:01
* anteaya hands mordred a pot of honey20:01
mordredannegentle: my mom just got in to town and I'm juggling her and my sanity20:01
openstackgerritClaudiu Belu proposed openstack/requirements: WIP: Removes posix-ipc from global-requirements
annegentlemordred: yup!20:01
mordredannegentle: can we push to tomorrow?? (/me hides and apologizes profusely)20:02
anteayadoes you mom weigh about the same as the other two oranges?20:02
annegentlemordred: oh sure20:02
annegentlemordred: no prob20:02
mordredannegentle: THANK YOU!20:03
annegentlemordred: 'course we won't conflict with TC meeting :)20:03
*** jeadre has quit IRC20:04
glauco_nibalizer: clarkb the test seems to be failing due the apache race condition. this patch should fix the race could we have the workflow triggered for this one?20:04
*** jeadre has joined #openstack-infra20:05
greghaynesSpamapS: Did you have a patch or somesuch for your bifrost network splitting out?20:05
greghaynesSpamapS: My changes are all one offs on the bastion, about to work on formulating them into a patch20:05
*** markvoelker has joined #openstack-infra20:06
*** david-lyle has joined #openstack-infra20:06
openstackgerritMerged openstack-infra/elastic-recheck: add signature for cinder pyopenssl issue
*** glauco_ has quit IRC20:07
*** kgiusti has left #openstack-infra20:10
*** stevemar has joined #openstack-infra20:10
*** ddecapit has joined #openstack-infra20:11
*** FallenPegasus has joined #openstack-infra20:11
*** andersonvom has joined #openstack-infra20:11
*** thedodd has quit IRC20:11
*** andersonvom has quit IRC20:11
*** dims has joined #openstack-infra20:11
*** dims has quit IRC20:12
*** thedodd has joined #openstack-infra20:12
*** cburroughs has quit IRC20:13
*** ddecapit_ has joined #openstack-infra20:14
*** thorst1 has joined #openstack-infra20:14
*** puranamr has quit IRC20:14
*** jeadre has quit IRC20:14
*** dims_ has quit IRC20:14
*** puranamr has joined #openstack-infra20:15
*** dprince has quit IRC20:16
clarkbmadhu_ak: done20:16
*** FallenPegasus has quit IRC20:16
*** mrmartin has joined #openstack-infra20:16
*** thedodd has quit IRC20:16
*** ramashri has joined #openstack-infra20:16
madhu_akclarkb: thanks20:16
*** sergmelikyan has joined #openstack-infra20:17
*** ddecapit has quit IRC20:17
openstackgerritMonty Taylor proposed openstack-infra/shade: Move glanceclient to new common interface
mordreddguerri`, SpamapS, Shrews, clarkb: ^^ that works for me now20:18
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: clarify duties in library release process
*** tiswanso has quit IRC20:20
*** jeadre has joined #openstack-infra20:20
*** HeOS has joined #openstack-infra20:20
*** tiswanso has joined #openstack-infra20:20
*** mrmartin has quit IRC20:21
*** glauco has joined #openstack-infra20:21
fungimordred: thanks for the glanceclient heads up20:22
fungimordred: should we stop puppet there/.20:22
clarkbfungi: possibly, any nodepool merges will upgrade it I think20:23
fungii'll do that as a prophylactic measure20:23
fungiclarkb: agree on upgrading the plugin everywhere. it doesn't seem to have broken jenkins04 yet, so presumably at least no worse than the current version we have on other masters20:24
fungii can cycle through more of them gracefully over the course of my evening20:24
*** austin81 has joined #openstack-infra20:24
clarkbfungi: great let me know if you need help20:25
openstackgerritMerged openstack-infra/project-config: Make lbaasv2 scenario API job non-voting
*** FallenPegasus has joined #openstack-infra20:25
fungi#status info puppet agent temporarily disabled on to avoid accidental upgrade to python-glanceclient 1.0.020:26
openstackstatusfungi: unknown command20:26
clarkbsdague: for the 1900ish grenade run I on;y see one job matching that IP and its in logstash now20:26
lifelessfungi: lol, had not20:26
*** zeih has joined #openstack-infra20:26
mordredfungi: yeah20:26
lifelessclarkb: syslog records ansible runs20:26
mordredfungi: also - that change (and the stack under it) we should land and release20:26
lifelessclarkb: it showed the wrong job name in the second ansible invocation20:26
fungi#status log puppet agent temporarily disabled on to avoid accidental upgrade to python-glanceclient 1.0.020:27
openstackstatusfungi: finished logging20:27
lifelessclarkb: oh ugh IP conflict? noice.20:27
mordredfungi: (although, actually, lemme rebase it so it's not held up on the neutorn change)20:27
clarkblifeless: maybe? I think it may have actually been stale jenkins config then jenkins ssh'd to the same slave twice thinking it was two different machines20:27
fungimordred: confirming, 219005?20:27
openstackgerritMonty Taylor proposed openstack-infra/shade: Move glanceclient to new common interface
mordredfungi: there you go - rebased20:28
lifelessclarkb: so we've had three jobs with the same looks-like-a-race symptom20:28
lifelessclarkb: I haven't dug into the other two to see if its the same20:28
clarkblifeless: I think its related to an issue with jenkins that we have suffered through for a while20:29
clarkbwe are testing a potential fix now20:29
lifelessclarkb: cool. thanks !20:29
glaucoCan we get this review workflowed? Right now the acceptance tests are failing due to this apache race condition.20:29
*** FallenPegasus has quit IRC20:30
*** zeih has quit IRC20:31
clarkbglauco: I think we are still discussing how we want to fix it on the mailing list20:31
clarkband jeblair in particular is not a fan of that method20:31
*** wenlock_ has joined #openstack-infra20:31
fungimordred: removing more lines than you add in shade means that the bug it represents is slightly fixed!20:31
mordredfungi: yes!20:32
*** jsavak has quit IRC20:32
mordredfungi: honestly, as much as it's a yuck for a second - it's GREAT that it's aligning20:32
clarkbmordred: and by aligning you mean using the sessions from keystone?20:32
fungimordred: yep, this is a very good thing. shade shrinkage ftw20:33
*** jsavak has joined #openstack-infra20:33
fungiclarkb: yep20:33
mordredclarkb: yes. and common constructor args and patterns20:33
*** matrohon has joined #openstack-infra20:33
fungimordred: you should have a chart of shade loc over time, with the loc axis labeled "how broken is openstack interoperability?"20:34
mordredfungi: hahaha20:34
clarkbmordred: why is service_name None in the test?20:34
mordredclarkb: becuase it defaults to none20:34
mordredclarkb: you only need to pass something if you override it20:34
openstackgerritAugustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting
clarkbok if tests work then it should merge20:35
*** wenlock_ has quit IRC20:35
fungii've got jenkins06 and 07 quiescing in preparation for ssh slaves plugin upgrade20:36
*** achanda has quit IRC20:38
*** FallenPegasus has joined #openstack-infra20:39
*** amitgandhinz has joined #openstack-infra20:40
*** achanda has joined #openstack-infra20:41
*** achanda has quit IRC20:41
*** achanda has joined #openstack-infra20:41
*** kushal has quit IRC20:42
*** dprince has joined #openstack-infra20:42
*** jsavak has quit IRC20:43
openstackgerritAugustina Ragwitz proposed openstack/diskimage-builder: Update dpkg elements README from free text to table formatting
*** gyee has joined #openstack-infra20:43
*** FallenPegasus has quit IRC20:43
*** alivigni has quit IRC20:44
*** puranamr has quit IRC20:46
*** puranamr has joined #openstack-infra20:46
clarkbfungi: still looks good fwiw20:48
*** tiswanso has quit IRC20:49
*** tiswanso has joined #openstack-infra20:50
*** regXboi has joined #openstack-infra20:51
*** baoli has quit IRC20:51
clarkbso I think thats the issue, need keystonemiddleware to fix20:52
*** FallenPegasus has joined #openstack-infra20:52
clarkbnow to check on centos7 things20:53
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions
*** Swami has quit IRC20:53
*** ildikov has quit IRC20:54
*** xyang1 has quit IRC20:55
*** yamahata has quit IRC20:55
*** mwagner_lap has quit IRC20:56
*** teran has joined #openstack-infra20:56
*** markvoelker has quit IRC20:56
*** rfolco has quit IRC20:57
*** FallenPegasus has quit IRC20:57
*** yamahata has joined #openstack-infra20:57
*** dims has joined #openstack-infra20:57
*** ihrachys has joined #openstack-infra20:57
*** jpeeler has quit IRC21:00
*** ccarmack has left #openstack-infra21:00
*** xyang1 has joined #openstack-infra21:01
*** esker has quit IRC21:01
*** hichihara has joined #openstack-infra21:01
*** shashank_hegde has joined #openstack-infra21:01
greghaynesSpamapS: is a straightforwardish way to I think get what we want21:02
greghaynesSpamapS: not as pretty as it could be, but its probably a good start21:02
greghaynesTheJulia: ^21:02
*** jpeeler has joined #openstack-infra21:03
clarkbupdate on hpcloud centos7, there is no facter installed?21:03
*** jpeeler has quit IRC21:03
*** jpeeler has joined #openstack-infra21:03
clarkbhrm maybe on that one host, thats weird21:04
*** glauco__ has joined #openstack-infra21:04
clarkbdefinitely no facter on the two hpcloud-b1 images I checked21:04
clarkbb3 is fine though, so weird21:04
*** annegentle has quit IRC21:04
*** dizquierdo has joined #openstack-infra21:05
fungidifferent image ages there?21:05
clarkbfungi: ya looking at that next21:05
*** rossella_s has quit IRC21:05
*** glauco has quit IRC21:06
*** FallenPegasus has joined #openstack-infra21:06
clarkbtwo days newer in b121:06
openstackgerritMatt Riedemann proposed openstack/requirements: Block sqlalchemy-migrate 0.9.8, cap at 0.10.0
mriedemdhellmann: sdague: mtreinish: ^21:06
*** rossella_s has joined #openstack-infra21:06
openstackgerritMerged openstack-infra/puppet-gerrit: Fix race condition in httpd install and config
*** stevemar has quit IRC21:08
ryanpetrelloI'm literally banging my head against this atm21:09
ryanpetrello(the pbr issue)21:09
clarkbpabelanger: firewalld is not installed on the centos7 image in hpcloud21:09
ryanpetrellohave spent the last 30 minutes trying to figure out wtf I'm doing wrong :)21:10
clarkbryanpetrello: keystonemiddleware needs a new release that isn't broken21:10
clarkband or we need to exclude it from testing21:10
*** regXboi has quit IRC21:10
ryanpetrelloyep, right now it's totally breaking kilo devstack for me21:10
*** FallenPegasus has quit IRC21:10
*** doug-fish has quit IRC21:11
pabelangerclarkb: Right, so it looks like we are trying to control a service we are not properly managing.21:12
pabelangerguess we need to add some logic for firewalld correctly21:13
clarkbpabelanger: well its not trivial to do it properly with puppet, we can install it everywhere which is silly since we want it off21:13
clarkbbut we can't stop it unless its installed21:13
clarkband we can't uninstall it until after its stopped21:13
clarkbwe may want to move the logic out of puppet and into or something21:13
clarkbor wirte a fact to check if it is there and do things based on that21:14
*** julim has quit IRC21:14
pabelangerI can try sometime locally tomorrow21:14
clarkbthis is tricky, is there any way to tell puppet its ok if this resource fails?21:14
clarkbwe might get around this with some exec hacks21:15
*** rmoats has joined #openstack-infra21:15
pabelangerRight, I was thinking of some exec hack21:15
clarkbexec systemctl stop firwalld onlyif ps -elf | grep -q firewalld21:15
*** ddecapit_ has quit IRC21:15
clarkbpabelanger: ^ something like that? then have a package ensure absent on firewalld require that exec?21:15
*** puranamr has quit IRC21:16
clarkbpabelanger: I'm guessing that firewalld became a default in a more recent point release of centos721:16
clarkband that hpcloud still lags on the base image point release21:16
*** thcipriani has left #openstack-infra21:16
*** puranamr has joined #openstack-infra21:16
clarkbI am going to throw up that patch really quick and we can argue over how terrible it is :)21:16
*** Hal has quit IRC21:17
*** dprince has quit IRC21:18
*** tiswanso has quit IRC21:19
*** esker has joined #openstack-infra21:19
openstackgerritMatt Riedemann proposed openstack-infra/infra-manual: Add warnings about tagging a release
*** FallenPegasus has joined #openstack-infra21:20
*** mayurig has joined #openstack-infra21:20
*** trown is now known as trown|outttypeww21:20
openstackgerritClark Boylan proposed openstack-infra/puppet-iptables: Really stop using firewalld
clarkbpabelanger: ^ I am gonna WIP that until I can test it21:23
*** glauco has joined #openstack-infra21:23
*** esker has quit IRC21:24
*** tmcpeak has joined #openstack-infra21:24
*** esker has joined #openstack-infra21:24
*** FallenPegasus has quit IRC21:24
*** john-davidge has quit IRC21:25
*** burgerk has quit IRC21:26
tmcpeakhey all - I'm working on some automation for Bandit.  We currently have some automation which can produce a list of projects like ['anchor', 'barbican'] etc.  What's the best way to get the corresponding repo for that project?21:27
*** glauco__ has quit IRC21:27
tmcpeakinitial unsophisticated approach would be: try openstack/ and then try stackforge/, but that seems low tech21:27
tmcpeakwas thinking you guys might have something better :)21:27
*** dizquierdo has quit IRC21:27
*** ivar-lazzaro has quit IRC21:28
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for sqla-migrate 0.9.8 bug 1490740
*** Rockyg has quit IRC21:29
openstackbug 1490740 in OpenStack-Gate "sqlalchemy-migrate 0.9.8 release breaks stable/kilo because of capped pbr" [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)21:29
clarkbtmcpeak: gerrit project list | grep ?21:29
clarkbtmcpeak: the suffix is unique21:29
*** dkranz has quit IRC21:30
*** mayurig has quit IRC21:30
tmcpeakclarkb: ahh ok cool21:30
*** mayurig has joined #openstack-infra21:31
clarkbtmcpeak: however, maybe if you explained the use case a bit more there is a better way21:31
clarkbdo you need to retrieve the source to run bandit against?21:32
clarkbmaybe you can just hardcode repo paths for each name in bandit?21:32
clarkbthat seems like it has inverted the relationship between linter and project though21:33
tmcpeakclarkb: yeah so we currently have a tool which parses the Jenkins Jobs and Zuul Layout to determine which projects are using Bandit gates.  What we'd like to do is build additional automation that pulls the project source for each project that does use Bandit, and runs that proejct's Bandit tox setup21:33
*** austin81 has left #openstack-infra21:33
*** FallenPegasus has joined #openstack-infra21:33
tmcpeakclarkb: we'd also like to avoid hardcoding anything.  The idea is that the list is automatically generated.  The reason we'd like to get it is so that we make sure Bandit changes don't break any projects which are currently using it in a gate21:33
clarkbtmcpeak: thats an odd expectation for a linter21:34
fungii was hoping jenkins06 and 07 would quiesce before i ran out to dinner, but they still need some time. i'll continue with them when i get back21:34
clarkbtmcpeak: since in theory as you add new rulkes you will be breaking projects21:34
* fungi will bbiaw21:34
clarkbtmcpeak: then at the beginning of each cycle we fix per project and bump the bandit version21:34
clarkbfungi: np21:34
tmcpeakclarkb: yeah, but in this case when we "add new rules" we are technically exposing new security issues.  Would be nice to at least provide some fore-warning21:34
*** e0ne has quit IRC21:34
tmcpeakclarkb: also our current setup is for projects to not pin to a specific version of Bandit21:35
tmcpeakwe'd like to keep it that way :)21:35
*** annegentle has joined #openstack-infra21:35
clarkbok, that is really different than the way I tried to have you all put it in place21:35
*** ddieterly has quit IRC21:36
clarkbI'm not sure how you can add more restrictive rules without pinning21:36
clarkbhow do you handle something like nova failing a new rule you want to add?21:36
tmcpeakclarkb: we are doing it with profiles.  Basically projects maintain their own "profille" which specifically lists the tests they use21:36
*** ivar-lazzaro has joined #openstack-infra21:36
tmcpeakclarkb: in that case we provide new tests, but it's up to the projects to decide when to add the rules to their profiles.  We also provide an updated profile with "all tests" so projects can pick and choose21:37
clarkbI see so similar to using version pins but with more selectivity in functionality21:37
*** annegent_ has joined #openstack-infra21:38
clarkbin that case asking gerrit is going to be the simplest way to get the canonical string I think however we try to remove gerrit from the test pipeline to avoid ddosing it21:38
*** FallenPegasus has quit IRC21:38
tmcpeakclarkb: yeah it ends up being pretty similar21:38
tmcpeakclarkb: what do you think about this?
*** matrohon has quit IRC21:39
clarkbtmcpeak: some projects in that file are still in stackforge21:39
clarkbas there is lag time to moving them in gerrit as it requires a gerrit downtime21:39
tmcpeakclarkb: I think that's ok, as long as the projects we need are in there21:40
*** annegentle has quit IRC21:40
clarkbtmcpeak: they would be but they won't tell you what the canonical path is21:40
*** claudiub has quit IRC21:40
tmcpeakcan I just take what's listed in repo and append to the git URL?21:44
tmcpeakclarkb: ^21:44
*** jcoufal has quit IRC21:45
*** regXboi has joined #openstack-infra21:45
clarkbtmcpeak: you can if you don't mind the race that exists between us updating gerrit and that repo being updated21:46
clarkbtmcpeak: my point is that it isn't perfect due to that administrative task in gerrit requiring downtime21:46
*** yolanda has quit IRC21:47
tmcpeakclarkb: ok cool, I think we can live with it21:47
tmcpeakI'll start with this approach and if we have headaches I'll rethink :)21:47
*** FallenPegasus has joined #openstack-infra21:47
*** regXboi has quit IRC21:48
tmcpeakclarkb: thanks for the advice21:48
*** rlandy has quit IRC21:49
clarkbtmcpeak: also note that there is a potential race between changes to bandit and the projects if you aren't cogating21:49
*** Sukhdev has quit IRC21:49
clarkbbasically bandit could approve a change that doesn't break nova now, but at the same time nova has approved a change that would break21:49
*** rmoats has quit IRC21:49
*** cbader has quit IRC21:49
clarkbthen depending on the order of merging you can end up in a state where bandit no longer works against nova21:50
*** yolanda has joined #openstack-infra21:50
tmcpeakclarkb: yeah, I think that could happen regardless.  This automation is just to help out with what I'm mostly doing by hand to try to make sure this doesn't happen21:50
*** rlandy has joined #openstack-infra21:50
clarkbtmcpeak: it cannot happen if you cogate21:50
clarkbbut there are drawbacks to that21:51
tmcpeakit sounds complicated...21:51
clarkbit just requires you to run the same test against nova too21:51
clarkbthen zuul does what it does21:51
*** Sukhdev has joined #openstack-infra21:51
*** FallenPegasus has quit IRC21:51
clarkbthe drawbacks there are bandit has to go into the giant gate queue with everything else21:52
tmcpeakthen Bandit could literally block Nova updates?21:52
clarkbyes thats what zuul does21:52
clarkbin general we have gone with using releases with linters instead though21:52
*** ivar-lazzaro has quit IRC21:52
clarkbbecause hodling everyone to a changing set of rules is hard whereas updating every 6 months is consumable and doesn't have these issues21:53
*** jtomasek has quit IRC21:54
tmcpeakyeah, we've wanted to avoid that to this point, in part because we're still making overall Bandit enhancements (not new tests) that we'd like projects to consume faster21:54
*** FallenPegasus has joined #openstack-infra21:54
tmcpeakclarkb: we're doing our security midcycle this week though, I'll kick it around with some of the other folks21:55
tmcpeakmaybe somebody wants to set up a cogate PoC :)21:55
tmcpeaksounds like something bknudson might be into21:56
bknudsonjust run bandit against keystone? that doesn't sound that bad.21:57
bknudsoneven running it against the other projects doesn't sound bad21:57
clarkbthe big drawback i really for bandit itself where you have to entire the giant queue21:57
tmcpeakthat is… also not great :)21:57
openstackgerritAdam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config
*** sabeen has joined #openstack-infra21:58
rm_workSo I have this updated gate job, and I have a change in the project repo that will make it actually work -- which one should i set as "depends-on" for which21:58
clarkbrm_work: can you use specific examples?21:59
rm_workerr sorry, the review that just posted21:59
*** FallenPegasus has quit IRC21:59
*** wenlock has quit IRC21:59
rm_workand is in the project repo21:59
clarkbrm_work: 167885 should depend on 217330 since there is no way to test them together until 217330 merges22:00
openstackgerritMerged openstack-infra/shade: Move glanceclient to new common interface
clarkbmordred: ^22:00
openstackgerritMerged openstack-infra/elastic-recheck: Add query for sqla-migrate 0.9.8 bug 1490740
openstackbug 1490740 in OpenStack-Gate "sqlalchemy-migrate 0.9.8 release breaks stable/kilo because of capped pbr" [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)22:01
rm_workclarkb: they are being tested together via an experimental job already22:01
rm_work(the thing we were discussing briefly earlier)22:01
clarkbrm_work: sure, but there is no job that will take the contents of those two changes and run them together dynamically22:01
rm_worknot dynamically... but they ARE being run together22:01
clarkbso you should have the project depend on the code that it needs to run22:01
clarkbin this case 167885 requires 217330 to function properly22:02
mriedemfungi: we should talk in irc next time there is a gate explosion :)22:02
*** sharikapongubala has quit IRC22:02
openstackgerritAdam Harwell proposed openstack-infra/project-config: Update Barbican Gate to use new Devstack config
*** sharikapongubala has joined #openstack-infra22:03
*** tmcpeak has quit IRC22:04
rm_workclarkb: so then with 167885 depending on 217330(gate) in the project we can +2/+W the change and it will wait patiently for whenever the project-config change is merged?22:04
clarkbrm_work: yup22:04
rm_workcool, thanks22:04
*** sergmelikyan has quit IRC22:04
*** sergmelikyan has joined #openstack-infra22:04
*** thedodd has joined #openstack-infra22:05
*** thiagop has quit IRC22:06
*** sergmelikyan has quit IRC22:07
*** eharney has quit IRC22:07
clarkbok so nova clones do take about ~550mb on centos722:07
*** sergmelikyan has joined #openstack-infra22:08
clarkbwhcih I believe is a slight improvement but not as major as I original thought22:08
*** Adri2000 has quit IRC22:08
*** dimtruck is now known as zz_dimtruck22:11
*** markvoelker has joined #openstack-infra22:11
*** ivar-lazzaro has joined #openstack-infra22:12
*** sergmelikyan has quit IRC22:12
*** sigmavirus24 is now known as sigmavirus24_awa22:15
*** markvoelker has quit IRC22:16
*** amuller has quit IRC22:16
*** asettle has joined #openstack-infra22:18
*** sergmelikyan has joined #openstack-infra22:18
*** sergmelikyan has quit IRC22:19
*** ivar-lazzaro has quit IRC22:21
*** sergmelikyan has joined #openstack-infra22:21
*** Piet has quit IRC22:22
*** dguerri` is now known as dguerri22:27
*** zeih has joined #openstack-infra22:27
*** edmondsw has quit IRC22:28
*** tlian2 has joined #openstack-infra22:28
*** spzala has quit IRC22:29
*** tlian has quit IRC22:30
*** dguerri is now known as dguerri`22:31
*** amitgandhinz has quit IRC22:31
*** jaypipes has quit IRC22:32
*** annegent_ has quit IRC22:32
*** zeih has quit IRC22:32
*** annegentle has joined #openstack-infra22:32
*** pradk has quit IRC22:33
*** pcm_ has quit IRC22:33
*** ociuhandu has joined #openstack-infra22:33
*** thorst1 has quit IRC22:35
*** mjturek1 has quit IRC22:35
*** btully has quit IRC22:36
armaxclarkb: ping22:38
*** hashar has quit IRC22:38
armaxregarding the keystonemiddleware pbr conundrum22:38
armaxyou aware of a fix?22:38
clarkbI am not22:39
*** sridhar_ram has quit IRC22:39
armaxclarkb: yeah, I am looking but I cannot find anything so far22:39
lifelessarmax: pbr conundrum ?22:41
armaxlifeless: I was simply referring to this error ContextualVersionConflict: (pbr 0.11.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('pbr<2.0,>=1.3'), set(['sqlalchemy-migrate']))22:41
armaxcaused by keystonemiddleware apparently22:42
anteayaI have not gone for a walk all day and will be going now22:42
lifelessarmax: in devstack?22:42
armaxlifeless: nothing specific to pbr per se22:42
clarkblifeless on the older branches/grenade I think22:42
clarkbbecause no constraints22:43
lifelessso grenade would show that if the new side didn't use constraints22:43
lifelessbut we made it use constraints22:43
lifelessarmax: is this in a log, or a personal machine22:43
armaxno in the gate22:43
*** MarkAtwood has joined #openstack-infra22:43
*** resker has joined #openstack-infra22:43
clarkbits the old sidr22:43
lifelessarmax: what job ? Got a log for me ?22:43
armaxlifeless: one example here22:43
armaxanother one here:22:44
mtreinishlifeless: it's kilo22:44
mtreinisharmax: it was caused by the sqlalchemy migrate release22:44
lifelessso kilo has <1.0 on pbr nearly universally still22:44
mtreinisharmax: mriedem has a g-r cap going through the gate now22:44
mtreinishthat'll fix it22:44
lifelessmtreinish: thanks22:44
armaxmtreinish: that’s awesome22:44
* lifeless stops poking22:44
armaxmtreinish: link?22:45
mtreinisharmax: there is also a ML thread about it22:45
openstackgerritJerry Zhao proposed openstack-infra/project-config: Add networking-fortinet project to neutron
armaxmtreinish: thanks, there’s been too much events going on the last few hours22:45
armaxmtreinish: it’s been hard to catch up22:45
armaxmtreinish: thanks22:45
clarkbis that the same rhing?22:46
clarkbjust transitive problem with middleare?22:46
*** esker has quit IRC22:47
mriedemclarkb: it's the same thing22:48
*** resker has quit IRC22:48
*** esker has joined #openstack-infra22:48
*** Piet has joined #openstack-infra22:52
*** ihrachys has quit IRC22:53
*** hdd has quit IRC23:00
*** mwagner_lap has joined #openstack-infra23:01
*** gordc has quit IRC23:02
*** angela-s has joined #openstack-infra23:03
*** stevemar has joined #openstack-infra23:04
*** annegentle has quit IRC23:04
Nakatolifeless: went back through constraints discussions and caught something I missed.  I've updated the changes, what do you think?23:04
*** ashleighfarnham has quit IRC23:04
*** wenlock1 has quit IRC23:07
*** tmcpeak has joined #openstack-infra23:09
*** ramashri has quit IRC23:11
*** thedodd has quit IRC23:12
kfox1111what distro's running on?23:13
*** tmcpeak has left #openstack-infra23:13
lifelessNakato: whats the change ?23:17
Nakatolifeless: in tox.ini the default is to use upper-constraints.txt from git, meaning nothing special to run tox -epyNN-constraints, and thus could be easily added as a default run option.23:17
*** angela-s has left #openstack-infra23:17
kfox1111is it ubuntu?23:17
Nakatolifeless: and exporting a new varible in run-tox.sh23:17
*** ddieterly has joined #openstack-infra23:20
*** Zhongjun has joined #openstack-infra23:21
pvaneckkfox1111: seems so
kfox1111ok. thanks. :)23:23
*** glauco has quit IRC23:27
*** zeih has joined #openstack-infra23:28
lifelessNakato: I like.23:31
lifelesssdague: ^ should address most of your concerns I think23:31
lifelesssdague: in that its Just Works for devs23:31
*** rlandy has quit IRC23:33
*** zeih has quit IRC23:33
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions
*** ZZelle has quit IRC23:35
NakatoThanks fungi23:35
*** spzala has joined #openstack-infra23:36
*** stevemar has quit IRC23:38
*** ramashri has joined #openstack-infra23:40
gmannsdake: that was regarding netaddr thing, now all set :)23:41
sdakenot sure how i can help with that :)23:42
sdakemaybey ou mean sdague?23:42
gmannsdake: ahh, i misread author  on :)23:44
*** sergmelikyan has quit IRC23:44
*** yushiro has joined #openstack-infra23:47
yushirogood morning infra-team,23:47
clarkbgood morning23:48
yushiroclarkb: morning :)23:48
*** teran has quit IRC23:48
yushiroWould you please review my patch about project-config ?23:49
clarkbarmax: mestery dougwig ^ is that something you all want to ACK first?23:51
* armax looks23:51
clarkbnot quite sure what the relationship between neutron and networking-fujitsu is but figure I can ask23:51
*** Sukhdev has quit IRC23:51
yushirosorry.  I'll ask them first.23:51
yushiroclarkb: Thank you.23:52
*** btully has joined #openstack-infra23:52
*** ramashri has quit IRC23:53
*** sridhar_ram has joined #openstack-infra23:53
*** DericHorn-HP has quit IRC23:54
*** dingyichen has joined #openstack-infra23:55
*** spzala has quit IRC23:55
*** ramashri has joined #openstack-infra23:56
*** btully has quit IRC23:57
openstackgerritKevin Fox proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip
*** achanda_ has joined #openstack-infra23:58
*** mayurig has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!