Tuesday, 2016-05-17

*** jamesmcarthur has joined #openstack-infra00:00
jeblairpabelanger: yes, though that's for the 'jenkins' user.  i guess they are not the same.  :)00:00
pabelangerjeblair: we should be able to land our zuul-worker element after diskimage-builder has been related00:01
*** mixos has joined #openstack-infra00:03
jeblairnibalizer: hiera group overlap is starting to be more complex; i'm about to add the jenkins ssh private key to a third hiera group (which means we have 3 copies of it)00:03
openstackgerritKhai Do proposed openstack-infra/system-config: Change commit message length validation to 72 chars  https://review.openstack.org/31619700:03
jeblairnibalizer: well, correction, it will be in 2 groups and 1 fqdn (so still 3 places)00:03
*** tosky has quit IRC00:05
openstackgerritMerged openstack-infra/system-config: Revert "Set region_name for tripleo-test-cloud-rh1"  https://review.openstack.org/31712900:05
*** r-mibu has joined #openstack-infra00:05
openstackgerritJames E. Blair proposed openstack-infra/system-config: Use the jenkins ssh private key for the zuul launcher  https://review.openstack.org/31714600:05
jeblairthat is in hiera now ^00:05
pabelangerjeblair: I believe you could have also used: $openstack_project::jenkins_ssh_key00:07
*** Qiming has joined #openstack-infra00:07
pabelangerjeblair: err, nevermind00:07
jeblairpabelanger: that's the public key i believe00:07
pabelangerjeblair: yes, I fat fingered that00:07
*** vkramskikh_ has quit IRC00:07
*** jamesmcarthur has quit IRC00:08
*** jamesmcarthur has joined #openstack-infra00:08
jeblaironce more00:08
pabelangerjeblair: just confiming, did you bring jenkins07.o.o offline?00:09
jeblairpabelanger: not i00:09
pabelangerokay, must be down then00:09
*** ddieterly has joined #openstack-infra00:11
*** ddieterly is now known as ddieterly[away]00:11
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure commitmessage.maxLineLength  https://review.openstack.org/31617500:11
*** johnny__ has quit IRC00:14
*** jamesmcarthur has quit IRC00:14
*** jamesmcarthur has joined #openstack-infra00:15
*** IlyaG has joined #openstack-infra00:15
jeblairTASK [zuul_runner] *************************************************************00:16
jeblairfatal: [node]: FAILED! => {"ansible_job_id": "18234392.1443", "changed": false, "failed": true, "finished": 1, "msg": "Exit code 1", "rc": 1}00:16
jeblairthat's not the most helpful thing.00:16
Shrewsjeblair: we should add -vvv to ansible-playbook00:17
jeblairgood idea00:17
jeblairi think we're going to need to add something to nodepool to tell it to only allocate one node at a time to a target00:17
jeblairotherwise, iterating on this will be very disruptive00:18
pabelangershould telnet be working?00:19
*** ddieterly[away] is now known as ddieterly00:19
jeblairpabelanger: it fails quickly00:19
pabelangerwonder if it is a path issue on the far side00:20
pabelangeror something like that00:20
pabelangerthere were some successful jobs: http://paste.openstack.org/raw/497283/00:22
pabelangerso \o/ to that00:22
*** xarses has quit IRC00:22
*** doug-fish has quit IRC00:23
*** IlyaG has quit IRC00:23
*** jamesmcarthur has quit IRC00:23
*** kzaitsev_mb has quit IRC00:25
*** sdake_ has joined #openstack-infra00:27
*** yuanying has joined #openstack-infra00:27
*** mtanino_ has joined #openstack-infra00:27
openstackgerritPaul Belanger proposed openstack-infra/zuul: Set default_flow_style=False for readability  https://review.openstack.org/31714900:28
pabelangerjeblair: Shrews: ^ cosmetic change for ansible playbooks00:29
*** mtanino has quit IRC00:29
*** xarses has joined #openstack-infra00:29
*** jamesmcarthur has joined #openstack-infra00:29
*** yuanying has quit IRC00:30
*** sdake has quit IRC00:30
*** Qiming has quit IRC00:31
*** rhochmuth has joined #openstack-infra00:33
*** amitgandhinz has joined #openstack-infra00:34
*** ewilson has joined #openstack-infra00:35
*** jamielennox|away is now known as jamielennox00:37
*** rhochmuth has quit IRC00:38
*** baoli has quit IRC00:39
*** sarob has joined #openstack-infra00:39
*** signed8bit_Zzz is now known as signed8bit00:40
*** yuanying has joined #openstack-infra00:40
*** rhallisey has joined #openstack-infra00:42
*** sflanigan has quit IRC00:43
*** _sarob has joined #openstack-infra00:43
*** rhochmuth has joined #openstack-infra00:43
*** jamesmcarthur has quit IRC00:44
*** sflanigan has joined #openstack-infra00:44
*** sflanigan has joined #openstack-infra00:44
*** baoli has joined #openstack-infra00:45
openstackgerritMerged openstack-infra/project-config: Revert "Setup region-name for tripleo-test-cloud-rh1"  https://review.openstack.org/31712800:47
*** mriedem has joined #openstack-infra00:47
*** sarob has quit IRC00:47
*** ddieterly is now known as ddieterly[away]00:50
*** kzaitsev_mb has joined #openstack-infra00:50
*** _sarob has quit IRC00:51
*** dimtruck is now known as zz_dimtruck00:52
*** gyee has quit IRC00:52
*** jamesmcarthur has joined #openstack-infra00:53
*** ddieterly[away] is now known as ddieterly00:53
*** IlyaG has joined #openstack-infra00:54
*** Liuqing has joined #openstack-infra00:56
*** ewilson has quit IRC00:56
*** ewilson has joined #openstack-infra00:57
*** rhochmuth has quit IRC00:58
*** jamesmcarthur has quit IRC00:59
*** shashank_hegde has quit IRC00:59
*** ewilson has quit IRC01:02
*** Jeffrey4l has joined #openstack-infra01:02
*** bpokorny_ has joined #openstack-infra01:03
openstackgerritMerged openstack-infra/irc-meetings: Adding additional chairs to IRC meeting  https://review.openstack.org/31712001:04
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add nodepool_ssh_public_key to nodepool.o.o  https://review.openstack.org/31619101:05
openstackgerritPaul Belanger proposed openstack-infra/puppet-openstackci: Add nodepool_ssh_public_key  https://review.openstack.org/31618801:05
*** Liuqing has quit IRC01:06
*** bpokorny has quit IRC01:07
*** Sukhdev has quit IRC01:07
pabelangerokay, zuul-worker DIB element works, tested locally.01:07
pabelangercalling it for the day01:07
*** ewilson has joined #openstack-infra01:08
*** jamesmcarthur has joined #openstack-infra01:08
*** bpokorny_ has quit IRC01:08
*** jamesmcarthur has quit IRC01:09
openstackgerritMerged openstack/diskimage-builder: Revert "Revert "Properly fail/trap in eval_run_d"" and fix PIPESTATUS  https://review.openstack.org/31713701:09
*** jamesmcarthur has joined #openstack-infra01:12
*** Liuqing has joined #openstack-infra01:14
*** rhochmuth has joined #openstack-infra01:14
*** rhochmuth has quit IRC01:14
*** jamesmcarthur has quit IRC01:15
ianwgreghaynes / bkero : ok, 1.16.1 pushed ^01:15
*** kzaitsev_mb has quit IRC01:16
*** baoli has quit IRC01:17
*** ewilson has quit IRC01:17
*** zhurong has joined #openstack-infra01:17
*** cody-somerville has joined #openstack-infra01:18
*** ewilson has joined #openstack-infra01:18
*** kien-ha has joined #openstack-infra01:18
openstackgerritMerged openstack-infra/system-config: Use the jenkins ssh private key for the zuul launcher  https://review.openstack.org/31714601:21
*** Qiming has joined #openstack-infra01:21
jheskethpabelanger: I'm a bit confused by https://review.openstack.org/#/c/316187/1 if you have a moment to help explain it01:24
*** jerryz has quit IRC01:25
*** Apoorva_ has joined #openstack-infra01:27
*** rhochmuth has joined #openstack-infra01:27
*** krtaylor has joined #openstack-infra01:28
*** baoli has joined #openstack-infra01:28
*** ddieterly has quit IRC01:28
*** itisha has quit IRC01:29
*** krtaylor has quit IRC01:30
*** yuanying has quit IRC01:30
*** Apoorva has quit IRC01:31
*** Apoorva_ has quit IRC01:32
*** baoli has quit IRC01:36
*** thorst has quit IRC01:37
*** thorst has joined #openstack-infra01:38
*** ccamacho has quit IRC01:39
*** rhochmuth has quit IRC01:39
*** ewilson has quit IRC01:40
*** rhochmuth has joined #openstack-infra01:41
*** rhallisey has quit IRC01:41
*** ccamacho has joined #openstack-infra01:42
*** Sukhdev has joined #openstack-infra01:42
*** rhochmuth has quit IRC01:44
*** Sukhdev has quit IRC01:44
*** jpr has joined #openstack-infra01:48
*** rfolco has quit IRC01:52
*** zz_dimtruck is now known as dimtruck01:53
*** rhochmuth has joined #openstack-infra01:58
*** tphummel has quit IRC01:59
*** jamielennox is now known as jamielennox|away01:59
*** mtanino has joined #openstack-infra02:00
*** Julien-zte has joined #openstack-infra02:01
*** mtanino_ has quit IRC02:02
*** baoli has joined #openstack-infra02:02
*** mtanino has quit IRC02:06
*** thorst has quit IRC02:06
*** rhochmuth has quit IRC02:07
*** thorst has joined #openstack-infra02:07
*** sridhar_ram has quit IRC02:07
*** yamahata has quit IRC02:07
*** baoli has quit IRC02:08
*** csaba has quit IRC02:10
*** gildub has quit IRC02:13
*** kzaitsev_mb has joined #openstack-infra02:13
*** jpr has quit IRC02:13
*** gothicmindfood has quit IRC02:14
*** thorst has quit IRC02:15
*** kzaitsev_mb has quit IRC02:18
*** gothicmindfood has joined #openstack-infra02:21
*** yuanying has joined #openstack-infra02:24
*** csaba has joined #openstack-infra02:24
*** mriedem has quit IRC02:28
*** Julien-zte has quit IRC02:29
*** bhunter71 has quit IRC02:29
*** lakshmiS has quit IRC02:31
*** Wei_Liu has joined #openstack-infra02:36
*** yamahata has joined #openstack-infra02:37
*** yamahata has quit IRC02:38
*** yamahata has joined #openstack-infra02:39
Wei_Liuhi, I am working on a local CI system, I could not find the "Label Verified -1/+1" on my local gerrit server, the version of gerrit is 2.10-rc0, How can I config this verified -1/+1?02:40
*** yamamoto has quit IRC02:45
*** fawadkhaliq has joined #openstack-infra02:46
*** amitgandhinz has quit IRC02:51
*** rhochmuth has joined #openstack-infra02:52
*** yamamoto has joined #openstack-infra02:52
*** kien-ha has quit IRC02:53
*** yuanying has quit IRC02:54
*** rhochmuth has quit IRC02:57
*** woodster_ has quit IRC02:58
*** duobei has joined #openstack-infra02:59
*** yamahata has quit IRC03:00
*** shashank_hegde has joined #openstack-infra03:05
*** fawadkhaliq has quit IRC03:09
*** thorst has joined #openstack-infra03:13
*** kzaitsev_mb has joined #openstack-infra03:14
*** signed8bit has quit IRC03:17
*** psachin has joined #openstack-infra03:17
*** kzaitsev_mb has quit IRC03:19
*** thorst has quit IRC03:20
*** armax has quit IRC03:20
*** links has joined #openstack-infra03:31
*** rhochmuth has joined #openstack-infra03:33
*** shashank_hegde has quit IRC03:34
*** sdake has joined #openstack-infra03:36
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: DO NOT REVIEW: This is for verifying CI with local.conf  https://review.openstack.org/24354503:37
*** yamahata has joined #openstack-infra03:38
*** dimtruck is now known as zz_dimtruck03:38
*** sdake_ has quit IRC03:38
*** rhochmuth has quit IRC03:40
zaroWei_Liu: look here, https://gerrit-review.googlesource.com/Documentation/config-labels.html#label_Verified03:42
zaroWei_Liu: or you can have gerrit create for you when you create the gerrit site with 'java -jar gerrit.war init -d my_site'03:43
*** rhochmuth has joined #openstack-infra03:47
*** yuanying has joined #openstack-infra03:48
*** esikachev has joined #openstack-infra03:50
Wei_Liuzaro: get it, thanks a lot03:50
*** duobei has quit IRC03:52
*** esikachev has quit IRC03:54
*** jamielennox|away is now known as jamielennox03:55
*** yamahata has quit IRC03:55
*** fawadkhaliq has joined #openstack-infra03:56
*** yamahata has joined #openstack-infra03:56
*** yamamoto has quit IRC03:58
*** psachin has quit IRC04:02
*** camunoz has quit IRC04:02
*** winggundamth has quit IRC04:04
*** duobei has joined #openstack-infra04:08
*** abregman has quit IRC04:09
openstackgerritKhai Do proposed openstack-infra/project-config: add a build job for Gerrit importer plugin  https://review.openstack.org/31722304:10
*** bpokorny has joined #openstack-infra04:11
*** lakshmiS has joined #openstack-infra04:12
*** winggundamth has joined #openstack-infra04:13
openstackgerritKhai Do proposed openstack-infra/project-config: add build job for Gerrit delete-project plugin  https://review.openstack.org/31722604:14
*** kzaitsev_mb has joined #openstack-infra04:15
*** salv-orlando has joined #openstack-infra04:15
*** psachin has joined #openstack-infra04:17
*** thorst has joined #openstack-infra04:18
openstackgerritKhai Do proposed openstack-infra/project-config: remove build job for javamelody plugin for Gerrit 2.8  https://review.openstack.org/31723004:18
*** salv-orl_ has quit IRC04:18
*** rhochmuth has quit IRC04:18
*** sdake_ has joined #openstack-infra04:19
*** bpokorny has quit IRC04:20
*** sdake has quit IRC04:20
*** kzaitsev_mb has quit IRC04:20
*** duobei has quit IRC04:22
*** yuanying has quit IRC04:22
*** yuanying has joined #openstack-infra04:23
*** thorst has quit IRC04:25
*** gildub has joined #openstack-infra04:26
*** yuanying has quit IRC04:27
*** harlowja_at_home has joined #openstack-infra04:29
openstackgerritDong Ma proposed openstack-infra/subunit2sql: Fix db API get_ids_for_all_tests  https://review.openstack.org/31032304:39
*** yamamoto has joined #openstack-infra04:39
*** tojuvone has quit IRC04:40
openstackgerritRyo Tagami proposed openstack-infra/jenkins-job-builder: Add support for Jenkins OpenStack Cloud Plugin  https://review.openstack.org/30694804:42
*** lakshmiS has quit IRC04:45
*** yuanying has joined #openstack-infra04:46
*** IlyaG_ has joined #openstack-infra04:50
*** tojuvone has joined #openstack-infra04:52
*** sree has joined #openstack-infra04:53
*** sree has quit IRC04:53
*** sree has joined #openstack-infra04:53
*** IlyaG has quit IRC04:54
*** shashank_hegde has joined #openstack-infra04:56
*** flepied has quit IRC04:56
*** sdake_ has quit IRC04:56
*** javeriak has joined #openstack-infra04:59
*** asselin has joined #openstack-infra05:03
*** jcoufal has joined #openstack-infra05:04
*** asselin__ has quit IRC05:06
*** maishsk_ has quit IRC05:07
*** yuanying has quit IRC05:08
*** e0ne has joined #openstack-infra05:09
*** e0ne has quit IRC05:11
*** Sukhdev has joined #openstack-infra05:17
*** kzaitsev_mb has joined #openstack-infra05:17
*** nelsnels_ has quit IRC05:18
*** savihou has joined #openstack-infra05:19
*** harlowja_at_home has quit IRC05:20
*** kzaitsev_mb has quit IRC05:22
*** nelsnelson has joined #openstack-infra05:22
*** thorst has joined #openstack-infra05:23
*** thorst has quit IRC05:30
*** shashank_hegde has quit IRC05:32
*** javeriak has quit IRC05:34
*** hparekh has quit IRC05:39
*** javeriak has joined #openstack-infra05:39
*** hparekh_ has joined #openstack-infra05:39
*** hparekh_ has quit IRC05:39
*** hparekh has joined #openstack-infra05:39
*** jaosorior has joined #openstack-infra05:43
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Add PS4 to show file/function/line  https://review.openstack.org/31726005:44
*** duobei has joined #openstack-infra05:49
*** shashank_hegde has joined #openstack-infra05:49
*** ramishra has quit IRC05:49
*** ramishra has joined #openstack-infra05:50
*** esikachev has joined #openstack-infra05:50
*** javeriak has quit IRC05:52
*** rlandy has quit IRC05:53
*** nadya has joined #openstack-infra05:53
*** esikachev has quit IRC05:55
*** savihou has quit IRC05:58
*** javeriak has joined #openstack-infra05:58
*** rcernin has joined #openstack-infra06:03
*** Jeffrey4l has quit IRC06:04
*** abregman has joined #openstack-infra06:05
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/31658406:07
*** nijaba has quit IRC06:07
openstackgerritMurad Awawdeh proposed openstack-infra/project-config: Basic Jenkins jobs  https://review.openstack.org/31644306:09
*** abregman has quit IRC06:10
*** savihou has joined #openstack-infra06:11
*** abregman has joined #openstack-infra06:12
openstackgerritNuman Siddique proposed openstack-infra/project-config: Enable functional test job in openstack ci for networking-ovn  https://review.openstack.org/31698306:13
*** rcernin has quit IRC06:18
*** savihou has quit IRC06:18
*** rcernin has joined #openstack-infra06:21
*** savihou has joined #openstack-infra06:22
*** shashank_hegde has quit IRC06:23
*** camunoz has joined #openstack-infra06:26
*** timrc has quit IRC06:26
*** thorst has joined #openstack-infra06:28
*** timrc has joined #openstack-infra06:29
*** Sukhdev has quit IRC06:29
*** nadya has quit IRC06:31
*** Liuqing has quit IRC06:33
*** sree has quit IRC06:33
*** changzhi has joined #openstack-infra06:33
*** thorst has quit IRC06:35
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Add PS4 to show file/function/line  https://review.openstack.org/31726006:39
*** andreas_s has joined #openstack-infra06:40
openstackgerritDong Ma proposed openstack-infra/subunit2sql: Fix db API get_ids_for_all_tests  https://review.openstack.org/31032306:40
*** jamielennox is now known as jamielennox|away06:41
*** javeriak has quit IRC06:44
*** akaszuba has joined #openstack-infra06:44
*** jtomasek_ has quit IRC06:50
*** jyuso1 has joined #openstack-infra06:51
openstackgerritMerged openstack-infra/project-config: fuel-devops: Add pylint&coverage jobs  https://review.openstack.org/31594806:51
*** flepied has joined #openstack-infra06:51
*** flepied has quit IRC06:56
*** jaosorior has quit IRC06:56
*** jaosorior has joined #openstack-infra06:57
*** duobei has quit IRC06:57
*** rhochmuth has joined #openstack-infra06:58
*** mixos has quit IRC06:59
*** rhochmuth has quit IRC07:02
*** duobei has joined #openstack-infra07:05
*** jtomasek_ has joined #openstack-infra07:06
*** javeriak has joined #openstack-infra07:08
*** zeih has joined #openstack-infra07:12
*** jtomasek_ has quit IRC07:12
*** Liuqing has joined #openstack-infra07:12
*** zeih has quit IRC07:17
*** arxcruz has joined #openstack-infra07:18
*** kzaitsev_mb has joined #openstack-infra07:18
openstackgerritYAMAMOTO Takashi proposed openstack-infra/irc-meetings: Make networking-midonet meeting every two weeks  https://review.openstack.org/31728207:20
*** shashank_hegde has joined #openstack-infra07:22
*** vincentll has joined #openstack-infra07:22
*** Bhargava has joined #openstack-infra07:22
*** fawadkhaliq has quit IRC07:22
*** fawadkhaliq has joined #openstack-infra07:23
*** Liuqing has quit IRC07:23
*** kzaitsev_mb has quit IRC07:24
*** duobei has quit IRC07:26
*** Hal has joined #openstack-infra07:27
*** Hal is now known as Guest6097107:28
*** |-paul-| has joined #openstack-infra07:30
*** mikelk has joined #openstack-infra07:30
*** ifarkas has joined #openstack-infra07:32
*** esikachev has joined #openstack-infra07:32
*** matrohon has joined #openstack-infra07:33
*** thorst has joined #openstack-infra07:33
*** jlanoux has joined #openstack-infra07:34
*** zeih has joined #openstack-infra07:39
*** thorst has quit IRC07:40
*** zeih has quit IRC07:44
*** nijaba has joined #openstack-infra07:46
*** savihou has quit IRC07:48
*** savihou has joined #openstack-infra07:48
*** Bhargava has left #openstack-infra07:50
*** nmagnezi has joined #openstack-infra07:50
*** duobei has joined #openstack-infra07:58
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** rossella_s has joined #openstack-infra08:00
*** shardy has joined #openstack-infra08:03
*** jtomasek has joined #openstack-infra08:04
*** pilgrimstack has joined #openstack-infra08:04
*** savihou has quit IRC08:10
*** jang has joined #openstack-infra08:10
*** savihou has joined #openstack-infra08:10
*** yaume has joined #openstack-infra08:11
yolandagood morning08:12
*** dmk0202 has joined #openstack-infra08:12
*** Jeffrey4l has joined #openstack-infra08:13
*** _degorenko|afk is now known as degorenko08:18
*** acabot has quit IRC08:18
*** jistr has joined #openstack-infra08:19
*** kzaitsev_mb has joined #openstack-infra08:20
openstackgerritMerged openstack-infra/project-config: generate_puppetfile: add missing $DIR  https://review.openstack.org/31659808:21
*** acabot has joined #openstack-infra08:22
openstackgerritMerged openstack-infra/project-config: Don't run dsvm on ironic api-ref changes  https://review.openstack.org/31678408:22
*** shashank_hegde has quit IRC08:23
*** hashar has joined #openstack-infra08:23
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add infracloud east to puppetmaster clouds  https://review.openstack.org/28292008:23
*** hashar has quit IRC08:23
*** hashar has joined #openstack-infra08:23
*** asettle has joined #openstack-infra08:24
*** kzaitsev_mb has quit IRC08:25
openstackgerrityolanda.robla proposed openstack-infra/shade: Add bay pod integration for shade+magnum  https://review.openstack.org/31665008:29
*** Guest60971 has quit IRC08:29
*** e0ne has joined #openstack-infra08:30
*** chem has joined #openstack-infra08:32
*** docaedo2 has quit IRC08:36
*** kzaitsev_mb has joined #openstack-infra08:37
*** thorst has joined #openstack-infra08:38
*** derekh has joined #openstack-infra08:39
*** fawadkhaliq has quit IRC08:41
*** fawadkhaliq has joined #openstack-infra08:42
*** kzaitsev_mb has quit IRC08:42
*** jcoufal has quit IRC08:44
*** Hal has joined #openstack-infra08:45
*** thorst has quit IRC08:45
*** Hal is now known as Guest3645908:45
*** kzaitsev_mb has joined #openstack-infra08:46
*** savihou has quit IRC08:47
*** savihou has joined #openstack-infra08:48
*** eyalb has joined #openstack-infra08:49
*** eyalb has quit IRC08:50
*** Jeffrey4l has quit IRC08:51
*** changzhi has quit IRC08:51
*** changzhi has joined #openstack-infra08:51
*** salv-orlando has quit IRC08:54
*** chem` has joined #openstack-infra08:55
*** chem has quit IRC08:57
*** maishsk has joined #openstack-infra09:01
*** javeriak has quit IRC09:04
*** javeriak has joined #openstack-infra09:04
*** maishsk_ has joined #openstack-infra09:05
*** maishsk has quit IRC09:05
*** maishsk_ is now known as maishsk09:05
*** javeriak has quit IRC09:06
*** javeriak has joined #openstack-infra09:07
*** jordanP has joined #openstack-infra09:10
*** kzaitsev_mb has quit IRC09:11
*** sambetts|afk is now known as sambetts09:12
*** vkramskikh_ has joined #openstack-infra09:15
*** rhochmuth has joined #openstack-infra09:16
*** rhochmuth has quit IRC09:16
KiallHey  - Q - is infra planning a move to xenial? and any idea on timeline? Looking for access to a xenial version of a package09:16
openstackgerritEthan Lynn proposed openstack-infra/project-config: Add tempest tests for senlin API  https://review.openstack.org/31654409:16
*** electrofelix has joined #openstack-infra09:17
openstackgerritSam Betts proposed openstack-infra/elastic-recheck: Add query for nova resource tracker/scheduler race  https://review.openstack.org/31636409:17
*** dtantsur|afk is now known as dtantsur09:21
*** maishsk has quit IRC09:21
*** sshnaidm has quit IRC09:22
wznoinskhi infra cores09:23
Kiallwznoinsk: all the jenkins UI's were firewalled off after some sort of jenkins security issue...09:23
wznoinskKiall: thanks09:25
*** sree has joined #openstack-infra09:27
*** kushal has joined #openstack-infra09:28
*** jcoufal has joined #openstack-infra09:30
*** dmk0202 has quit IRC09:31
*** sree_ has joined #openstack-infra09:31
*** sree_ is now known as Guest1745609:31
*** zqfan has quit IRC09:33
*** sree has quit IRC09:33
*** ihrachys has joined #openstack-infra09:34
*** amotoki has joined #openstack-infra09:35
*** Regalla has joined #openstack-infra09:39
*** maishsk has joined #openstack-infra09:41
*** matrohon has quit IRC09:43
*** Guest17456 has quit IRC09:46
*** zhurong has quit IRC09:51
*** nadya has joined #openstack-infra09:54
*** niska has quit IRC09:56
*** duobei has quit IRC10:02
*** yamahata has quit IRC10:02
*** duobei has joined #openstack-infra10:03
*** nmagnezi has quit IRC10:04
*** nmagnezi has joined #openstack-infra10:05
penguinologHi! Could anybody help us with patch: https://review.openstack.org/#/c/281824/18 - it has +W, it has not predicates, and it's not merged (and I don't see any pending jobs on zuul for it)10:08
*** Na3iL has joined #openstack-infra10:09
*** jed56 has quit IRC10:09
*** nmagnezi has quit IRC10:09
*** salv-orl_ has joined #openstack-infra10:13
*** salv-orl_ has quit IRC10:13
*** salv-orl_ has joined #openstack-infra10:13
*** Jeffrey4l has joined #openstack-infra10:15
*** sshnaidm has joined #openstack-infra10:15
*** tosky has joined #openstack-infra10:15
*** kzaitsev_mb has joined #openstack-infra10:15
openstackgerritYAMAMOTO Takashi proposed openstack-infra/irc-meetings: Add Neutron Common Classifier meeting  https://review.openstack.org/31683010:16
*** sdague has joined #openstack-infra10:18
sdaguehmmm... did gerrit just fall over?10:20
sdagueoh, I guess the 500 error was intermiten10:21
*** zeih has joined #openstack-infra10:23
*** rossella_s has quit IRC10:23
*** rossella_s has joined #openstack-infra10:24
penguinologfor our patch: recheck/republish not helps: pending jobs not arrives10:24
*** Qiming has quit IRC10:25
*** e0ne has quit IRC10:25
*** e0ne has joined #openstack-infra10:26
*** zeih has quit IRC10:27
*** jlanoux_ has joined #openstack-infra10:28
*** jlanoux has quit IRC10:30
*** niska has joined #openstack-infra10:34
*** yamamoto has quit IRC10:36
*** changzhi has quit IRC10:36
*** maishsk has quit IRC10:38
*** duobei has quit IRC10:39
*** jang has quit IRC10:41
*** daemontool has quit IRC10:41
*** niska has quit IRC10:43
*** nmagnezi has joined #openstack-infra10:43
*** daemontool has joined #openstack-infra10:46
*** jcoufal has quit IRC10:48
*** jordanP has quit IRC10:49
*** zeih has joined #openstack-infra10:50
*** jamielennox|away is now known as jamielennox10:53
*** jlanoux has joined #openstack-infra10:54
*** zeih has quit IRC10:54
*** dtantsur is now known as dtantsur|bbl10:55
*** jlanoux_ has quit IRC10:57
*** zeih has joined #openstack-infra10:57
*** fawadkhaliq has quit IRC10:59
Wei_Liuhi , how can I update project.config of All-Projects, I did not find edit config button in project's general page. the gerrit version is 2.10-rc1010:59
*** niska has joined #openstack-infra11:01
*** nadya has quit IRC11:01
*** Regalla has quit IRC11:01
*** Jeffrey4l has quit IRC11:02
*** zeih has quit IRC11:02
*** penguinolog has quit IRC11:06
*** lucasagomes is now known as lucas-hungry11:07
*** jcoufal has joined #openstack-infra11:09
*** maishsk has joined #openstack-infra11:09
*** amrith has quit IRC11:09
*** kushal has quit IRC11:10
*** amrith has joined #openstack-infra11:12
*** YorikSar has quit IRC11:13
*** maishsk has quit IRC11:14
*** Jeffrey4l has joined #openstack-infra11:14
*** YorikSar has joined #openstack-infra11:14
*** maishsk has joined #openstack-infra11:15
*** esikachev has quit IRC11:16
*** rhochmuth has joined #openstack-infra11:17
sshnaidmclarkb, hi11:18
*** Qiming has joined #openstack-infra11:20
*** rhochmuth has quit IRC11:22
*** ldnunes has joined #openstack-infra11:23
*** jistr has quit IRC11:24
*** jcoufal has quit IRC11:24
*** rhallisey has joined #openstack-infra11:26
*** hparekh has quit IRC11:27
Wei_Liuzaro, I did not find "edit config" button in project's general page of my gerrit web, the gerrit version is 2.10-rc0 , is there any other way to edit project.config?11:29
*** esikachev has joined #openstack-infra11:29
*** fawadkhaliq has joined #openstack-infra11:30
*** fawadkhaliq has quit IRC11:30
*** Na3iL has quit IRC11:30
*** penguinolog has joined #openstack-infra11:31
*** nadya has joined #openstack-infra11:31
*** javeriak has quit IRC11:32
*** salv-orl_ has quit IRC11:32
*** salv-orlando has joined #openstack-infra11:35
*** hparekh has joined #openstack-infra11:36
*** hparekh has joined #openstack-infra11:36
*** gordc has joined #openstack-infra11:36
*** Jeffrey4l has quit IRC11:37
*** Qiming has quit IRC11:37
openstackgerritMerged openstack-infra/puppet-jenkins: Fix incorrect directory permissions in jenkinsuser.pp  https://review.openstack.org/31547911:39
*** Na3iL has joined #openstack-infra11:39
*** Qiming has joined #openstack-infra11:40
*** rhochmuth has joined #openstack-infra11:40
*** jed56 has joined #openstack-infra11:42
*** Qiming_ has joined #openstack-infra11:43
*** rhochmuth has quit IRC11:44
*** rhallisey has quit IRC11:45
*** dguitarbite has joined #openstack-infra11:46
dguitarbitehello, can someone take a look at this review? https://review.openstack.org/#/c/314998/11:46
*** Qiming has quit IRC11:46
*** rhallisey has joined #openstack-infra11:46
*** rfolco has joined #openstack-infra11:46
KiallAny project-config core's about for a hopefully quick review? I'd like to be around as it merges to watch for any issues :) https://review.openstack.org/#/c/317029/11:47
EmilienMpabelanger: do we have plans to have mirrors of UCA ?11:49
*** maishsk has quit IRC11:49
*** Na3iL has quit IRC11:49
*** hparekh has quit IRC11:50
*** Jeffrey4l has joined #openstack-infra11:51
*** javeriak has joined #openstack-infra11:51
*** yamamoto has joined #openstack-infra11:51
*** maishsk has joined #openstack-infra11:51
*** dmk0202 has joined #openstack-infra11:52
penguinologhttps://review.openstack.org/#/c/281824/ - no jobs on zuul after rebase, no recheck working. Could anybody help?11:52
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Prepare files for testing zuulv25  https://review.openstack.org/31675311:53
*** yamamoto has quit IRC11:54
*** Qiming_ has quit IRC11:54
*** thorst has joined #openstack-infra11:55
*** Qiming has joined #openstack-infra11:57
*** nadya has quit IRC11:58
*** hparekh has joined #openstack-infra11:58
*** Qiming has quit IRC11:58
yolandapenguinolog, seems you hit a circular dependency12:00
yolandaException: Dependency cycle detected: 281824 in [u'281824', u'296457']12:00
*** jistr has joined #openstack-infra12:00
yolandatry to check the whole dependency chain and amend it12:01
*** rhochmuth has joined #openstack-infra12:01
*** jcoufal has joined #openstack-infra12:02
*** weshay has joined #openstack-infra12:03
vgridnevhello folks, need review and probably fast-approve for change: https://review.openstack.org/#/c/315767/12:04
*** Qiming has joined #openstack-infra12:04
penguinologyolanda, how can our team see this exception, if job could not be started?12:05
*** julim has joined #openstack-infra12:05
*** gildub has quit IRC12:05
*** rhochmuth has quit IRC12:05
*** aysyd has joined #openstack-infra12:06
*** Qiming has quit IRC12:06
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] make grenade job temporary non-voting  https://review.openstack.org/31576712:06
*** duobei has joined #openstack-infra12:07
*** Qiming has joined #openstack-infra12:07
*** javeriak has quit IRC12:08
*** dprince has joined #openstack-infra12:08
*** esikachev has quit IRC12:09
*** trown|outtypewww is now known as trown12:10
*** anteaya has joined #openstack-infra12:10
*** hparekh has quit IRC12:10
*** hparekh has joined #openstack-infra12:11
*** rodrigods has quit IRC12:11
*** rodrigods has joined #openstack-infra12:11
*** lucas-hungry is now known as lucasagomes12:13
*** rhochmuth has joined #openstack-infra12:14
*** duobei has quit IRC12:15
openstackgerritThierry Carrez proposed openstack-infra/irc-meetings: Cleanup old Trainers and Evoque meetings  https://review.openstack.org/31742012:15
*** raildo-afk is now known as raildo12:15
*** bdemers has quit IRC12:15
*** Qiming has quit IRC12:16
*** Qiming has joined #openstack-infra12:16
*** sshnaidm has quit IRC12:17
*** hparekh has quit IRC12:18
*** nadya has joined #openstack-infra12:18
*** rhochmuth has quit IRC12:18
*** yamamoto has joined #openstack-infra12:19
*** javeriak has joined #openstack-infra12:20
*** ayoung has quit IRC12:23
pabelangerEmilienM: nothing at the moment12:25
*** krtaylor has joined #openstack-infra12:25
EmilienMpabelanger: ok, we also have timeouts, etc on their repos :(12:25
pabelangerEmilienM: Ya, possible we can add it, just not on my list ATM12:25
EmilienMcool no problem, let's first finish the centos thing, I backported the puppet-openstack_extras patch in stable/mitaka so we can test your puppet fact patch in our stable CI12:26
*** jpr has joined #openstack-infra12:26
*** rlandy has joined #openstack-infra12:26
yolandapenguinolog, i saw that on zuul logs, it's not visible from outside12:27
EmilienMpabelanger: can I steal https://review.openstack.org/#/c/316376/ in stable/* ?12:27
*** amitgandhinz has joined #openstack-infra12:27
pabelangerEmilienM: not stealing, feel free to use12:28
*** amitgandhinz has quit IRC12:29
*** amitgandhinz has joined #openstack-infra12:29
*** sshnaidm has joined #openstack-infra12:33
*** Jeffrey4l has quit IRC12:34
*** javeriak has quit IRC12:34
*** baoli has joined #openstack-infra12:35
*** hashar has quit IRC12:36
*** rfolco has quit IRC12:36
*** baoli_ has joined #openstack-infra12:36
*** javeriak has joined #openstack-infra12:36
*** rfolco has joined #openstack-infra12:36
*** hparekh has joined #openstack-infra12:37
*** esikachev has joined #openstack-infra12:39
*** baoli has quit IRC12:40
*** javeriak has quit IRC12:40
*** jpr has quit IRC12:41
openstackgerritMerged openstack-infra/project-config: Make gate-manila-tempest-minimal-dsvm-lvm job voting  https://review.openstack.org/31671612:41
*** _prad_ has joined #openstack-infra12:42
KiallAny project-config core's about for a hopefully quick review? Hate to be that guy, but I'd like to be around as it merges to watch for any issues :) https://review.openstack.org/#/c/317029/12:44
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/31658412:44
*** dtantsur|bbl is now known as dtantsur12:45
openstackgerritMerged openstack-infra/project-config: remove build job for javamelody plugin for Gerrit 2.8  https://review.openstack.org/31723012:45
*** hashar has joined #openstack-infra12:47
openstackgerritMerged openstack-infra/project-config: add build job for Gerrit delete-project plugin  https://review.openstack.org/31722612:48
*** edmondsw has joined #openstack-infra12:48
openstackgerritMerged openstack-infra/project-config: Add neutron-dynamic-routing to #-neutron  https://review.openstack.org/31659512:48
*** rhochmuth has joined #openstack-infra12:49
*** vincentll has quit IRC12:49
*** salv-orlando has quit IRC12:49
openstackgerritAnita Kuno proposed openstack-infra/irc-meetings: Remove the Tuesday 0800 third party meeting.  https://review.openstack.org/31744412:51
*** bhunter71 has joined #openstack-infra12:52
anteayalennyb: ^^12:52
*** javeriak has joined #openstack-infra12:54
*** salv-orlando has joined #openstack-infra12:55
*** matrohon has joined #openstack-infra12:55
*** psilvad has joined #openstack-infra12:57
*** maishsk has quit IRC12:59
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Running tests  https://review.openstack.org/31643612:59
openstackgerritPaul Belanger proposed openstack-infra/project-config: Use regionOne for tripleo-test-cloud-rh1  https://review.openstack.org/31744713:00
*** amrith is now known as _amrith_13:00
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: implement breaker-jobs-nv on xenial  https://review.openstack.org/31744913:01
EmilienMnibalizer: fyi ^ this is a first iteration of running Puppet beaker jobs on Xenial using exprimental queue.13:01
openstackgerritPaul Belanger proposed openstack-infra/system-config: Use regionOne for tripleo-test-cloud-rh1  https://review.openstack.org/31745013:01
*** xyang1 has joined #openstack-infra13:02
*** vincentll has joined #openstack-infra13:03
*** psilvad has quit IRC13:04
sshnaidmdoes anybody succeed to log in into wiki? https://wiki.openstack.org13:04
*** doug-fish has joined #openstack-infra13:06
*** psilvad has joined #openstack-infra13:06
*** ewilson has joined #openstack-infra13:07
*** eharney has joined #openstack-infra13:07
pabelangersshnaidm: yes, but you need to have an existing user account. We've disabled new user creation as a result of being spammed13:07
*** Goneri has joined #openstack-infra13:08
nibalizerEmilienM: nice!13:08
sshnaidmpabelanger, I tried to log in with launchpad account ad it failed. What can I do then?13:08
pabelangersshnaidm: sadly, not much at the moment. We are hoping to upgrade the wiki in the coming week to a newer version of software.  If that goes well we should be able to open account creation again.13:09
sshnaidmpabelanger, thanks for clarifying this13:10
anteayalennyb: thank you13:10
*** janonymous has joined #openstack-infra13:11
*** psilvad has quit IRC13:11
pabelangersshnaidm: Yes, it is not the best policy. Apologies for the inconvenience13:12
*** tpsilva has joined #openstack-infra13:12
phschwartzI am trying to use infra's ansible-puppet and seem to get an error http://paste.openstack.org/show/497336/  any thoughts?13:12
*** psilvad has joined #openstack-infra13:13
*** mriedem has joined #openstack-infra13:14
yolandaphschwartz, that shows that zuul.test cannot be resolved properly13:14
vgridnevhello folks, could you please take a look on this one: https://review.openstack.org/#/c/315767/13:14
*** dstufft has quit IRC13:14
*** pwgravel has joined #openstack-infra13:14
nibalizerphschwartz: that looks like puppet agent instead of puppet apply13:14
nibalizerpuppet apply would not request a certificiate13:15
*** dstufft has joined #openstack-infra13:15
*** rvasilets__ has joined #openstack-infra13:16
anteayanibalizer: morning13:17
anteayanibalizer: how far did we get standing up a storyboard-dev server and what can I do to help?13:17
nibalizeranteaya: hi13:17
nibalizeri think we're ready to run launch_node13:18
anteayahey there13:18
anteayanibalizer: yay!13:18
anteayathank you so much13:18
*** jordanP has joined #openstack-infra13:18
nibalizeri haven't done it yet though, and today/tomorrow/thursday i am very busy internally13:18
anteayanibalizer: I understand13:18
*** psilvad has quit IRC13:18
anteayanibalizer: is it okay with you if I can find someone else willing to push the button?13:18
anteayanibalizer: anything they should know before they do, if I can find someone?13:19
nibalizerso either someone else can pick it up, or I can pick it up once I'm done with this thing I'm doing13:19
nibalizeranteaya: yea someone else is fine13:19
anteayanibalizer: yup, thank you13:19
anteayanibalizer: appreciate your help getting this far13:19
nibalizeri think clark wants feedback on his patches to launch_node13:19
anteayagood luck wiht the internal thing13:19
nibalizerso it was requested to try that out13:19
anteayanibalizer: awesome thanks, I'll look up those patches and start staring at them13:19
*** _ari_ has joined #openstack-infra13:21
*** vincentll has quit IRC13:22
anteayaturns out Zara and SotK were able to find me a place to sit at codethink, relocating13:22
*** bhunter71 has quit IRC13:24
*** psachin has quit IRC13:24
*** dtantsur has quit IRC13:27
*** anteaya has quit IRC13:27
phschwartzyolanda: I think nibalizer is right, there is no puppet master so it shouldn't do a puppt agent. Also in my env the host zuul.test is resolvable.13:27
*** javeriak has quit IRC13:28
*** nwkarsten has joined #openstack-infra13:28
*** dtantsur has joined #openstack-infra13:30
openstackgerritMerged openstack-infra/project-config: Add Designate+Tempest on Mitaka jobs  https://review.openstack.org/31702913:32
*** matt-borland has joined #openstack-infra13:32
Kiallthanks nibalizer + yolanda re ^ :)13:32
Kiallfingers crossed for no explosions once it's pushes ot13:32
*** openstackgerrit has quit IRC13:32
yolandaKiall, you don't look very safe...13:33
*** openstackgerrit has joined #openstack-infra13:33
*** doug-fish has quit IRC13:33
KiallI'm wearing my cape, I'll be OK ;)13:33
nwkarstenI'm running into an issue in my personal system. I'm trying to set up zuul and I'm getting this error in the logs http://paste.openstack.org/show/497272/ I'm using puppet to configure and the SSH that I've passed in are RSA format. Can anyone help?13:34
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Report overcloud delete fail  https://review.openstack.org/31746413:35
*** yamamoto has quit IRC13:36
*** links has quit IRC13:36
*** thiagop has joined #openstack-infra13:36
*** vincentll has joined #openstack-infra13:36
*** rkukura has quit IRC13:37
phschwartzyolanda: Any thoughts why without a puppetmaster set and passed to the role that it would be attempting to do a puppet agent?13:37
phschwartzmordred: ^ any thoughts, I know your familar with ansible-puppet13:37
*** kgiusti has joined #openstack-infra13:38
*** inc0 has joined #openstack-infra13:39
*** bdemers has joined #openstack-infra13:40
Shrewsphschwartz: mordred is vacationing in a land far, far away13:40
yolandaphschwartz, looks as you are not passing the manifest setting13:40
yolandais no manifest is passed, ansible-puppet runs in agent mode13:41
*** jdob has left #openstack-infra13:41
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to scan project repo for new bug fixes in master  https://review.openstack.org/29642013:42
pabelangeryolanda: mind reviewing https://review.openstack.org/#/c/317450/ and https://review.openstack.org/#/c/317447/13:43
pabelangereven testing if possible, pretty sure I have it right this time13:44
*** zz_dimtruck is now known as dimtruck13:44
yolandait's regionOne, with lower r ? i'm used to RegionOne instead13:44
*** sdake has joined #openstack-infra13:44
funginwkarsten: what version of paramiko do you have installed? i know that 2.x doesn't work for zuul yet, but don't recall how it manifests13:46
nibalizerphschwartz: pass in a manifest13:47
pabelangeryolanda: regionOne13:48
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs of specified importance  https://review.openstack.org/29642113:48
pabelangeryolanda: RegionOne will not work13:48
yolandaok +213:48
pabelangeryolanda: cool13:48
*** sdake_ has joined #openstack-infra13:48
*** psachin has joined #openstack-infra13:48
*** sdake has quit IRC13:49
*** winggundamth has quit IRC13:49
pabelangerfungi: do you mind reviewing 317450 and 317447. I have a copy of keystone endpoint-list I can share if needed too.13:50
fungipabelanger: the one with "nova" as the region name got reverted i guess?13:51
fungii'm assuming that's what i saw skimming scrollback13:52
*** winggundamth has joined #openstack-infra13:52
*** jaosorior has quit IRC13:52
pabelangerfungi: yes, I mistakenly testing it locally. It was actually the availability zone13:52
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs with specific tag  https://review.openstack.org/29645413:52
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs of specified importance  https://review.openstack.org/29642113:52
*** dimtruck is now known as zz_dimtruck13:53
*** sigmavirus24_awa is now known as sigmavirus2413:54
*** doug-fish has joined #openstack-infra13:54
nwkarstenfungi: version 1.17.013:54
*** binbincong has quit IRC13:55
*** penguinolog has quit IRC13:55
*** tosky has quit IRC13:56
*** njohnsto- is now known as njohnston--13:56
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Add tool that annotates bug numbers with details from LP  https://review.openstack.org/29647013:56
*** doug-fis_ has joined #openstack-infra13:56
funginwkarsten: okay, so that's likely not the cause. i believe "SSHException: not a valid EC private key file" is really just a misleading error for the remote server not authorizing your key for access13:56
funginwkarsten: looks like it's filed as a long-standing issue with paramiko https://github.com/paramiko/paramiko/issues/38713:57
*** zz_dimtruck is now known as dimtruck13:57
*** jswarren has joined #openstack-infra13:58
*** tosky has joined #openstack-infra13:58
*** bdemers has quit IRC13:59
funginwkarsten: what gerrit is it connecting to? review.openstack.org or one you operate?13:59
*** vincentll has quit IRC13:59
*** doug-fish has quit IRC13:59
hasharhello folks!14:00
hasharany idea what is the issue tracker for the Jenkins Gearman plugin that Khai maintains? I am lost between Storyboard, launchpad and Jenkins jira :D14:00
*** binbincong has joined #openstack-infra14:01
*** doug-fis_ has quit IRC14:01
lucasagomeshi all, if you have some time, mind taking a look at https://review.openstack.org/#/c/315646/ ? The dependency patch for the change is already merged. Thank you14:01
fungihashar: http://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/README.rst#n30 says it's https://storyboard.openstack.org/#!/project/70614:01
jlvillalttx Can you look at https://review.openstack.org/#/c/305528/ ?14:02
jlvillalttx We currently don't have a multiple of 4 meetings on http://eavesdrop.openstack.org/ so we are losing meetings on the index14:02
nwkarstenfungi: review.openstack.org14:03
funginwkarsten: and what username is zuul configured to authenticate to gerrit as?14:03
jlvillalttx So right now XenApi and Zagar don't make the index list14:03
*** jordanP has quit IRC14:03
ttxheh, nice14:03
ttxjlvillal: looking14:03
jlvillalttx thanks14:03
*** abregman has quit IRC14:04
*** med_ has joined #openstack-infra14:04
*** med_ has quit IRC14:04
*** med_ has joined #openstack-infra14:04
*** doug-fis_ has joined #openstack-infra14:04
ttxjlvillal: we'll need a release to get that used14:04
*** rhochmuth has quit IRC14:05
ttxonce it merges14:05
*** Julien-zte has joined #openstack-infra14:05
jlvillalttx, Ah, okay. /me didn't know that.14:05
nwkarstenfungi: my teammates14:05
ttxthe irc-meetings post-jobs instalkl yaml2ical from pypi14:05
ttxjlvillal: I'll try to push a release shortly after that14:05
jlvillalttx, Thank you :)14:05
funginwkarsten: mainly asking so that i can confirm the ssh public key associated with it in gerrit's backend db14:06
funginwkarsten: and also so that i can check the logs in gerrit to see what the details of the authentication failure actually are14:06
nwkarstenfungi: philip-schwartz14:06
funginwkarsten: got it, checking14:07
jheskethjeblair: I'm off in a minute, but I've been working on adding functional tests to the ansible-launcher if you're interested: https://review.openstack.org/#/q/topic:zuulv25-tests14:07
funginwkarsten: there are currently 5 different ssh public keys associated with that account. do you know which one it's supposed to be?14:08
jeblairjhesketh: neat.  people keep adding those to requirements.txt though, and i don't think they belong there14:09
*** daemontool has quit IRC14:09
jeblairjhesketh: (they are not requirements for zuul, and i do not want anyone else to use the v25 launcher)14:09
phschwartzfungi: hold a sec. Why are you looking at my public keys?14:09
*** jpr has joined #openstack-infra14:09
fungiphschwartz: nwkarsten says he has a third-party ci using them to authenticate to gerrit14:10
nwkarstenfungi: nope, but i set it up to use not one of those. i'm going to try setting it up to use my own gerrit user14:10
hasharfungi: found it. apparently that is Jenkins upstream tracker from https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin  :D14:10
hasharfungi: and storyboard as well bah14:10
jheskethjeblair: so if we don't want the launcher to be usable by anybody, then sure, we can probably ignore them... but I think we should possibly reconsider how v3 is approached14:10
jeblairjhesketh: what do you mean?14:11
funginwkarsten: yeah, that's probably a better solution. there is at least one zuul successfully authenticating as philip-schwartz in gerrit from what i can see, but i also see one (presumably yours) returning AUTH FAILURE FROM no-matching-key14:11
jheskethjeblair: given the work here and some of the other outstanding v2 patches, I'm starting to think we should merge what makes sense in v2 and start a new v3 branch. Then cherrypick off the current v3 branch the config/layout changes and refactors etc.14:12
phschwartzfungi: I think the issue might be he isn't using the key we have in our hiera config, let me check with him.14:12
jeblairjhesketh: the work here does not align with v3 at all -- it's a dead-end.14:12
fungiphschwartz: yeah, please coordinate with nwkarsten before i troubleshoot further. it looks like a simple case of username/ssh key mismatch in gerrit14:13
jheskethjeblair: the alternative of trying to sync v2 with v3 I think will be a lot of work... or if we just ignore the ansible-launcher and no longer merge v2 onto v3 we lose a lot of history14:13
jeblairjhesketh: yes, we ignore the ansible launcher14:13
jheskethjeblair: hmm, but it could align I guess was my thinking14:13
jeblairjhesketh: it's a merge commit where the v3 branch wins in any conflicts14:13
*** maishsk has joined #openstack-infra14:13
jheskethwe could iterate on the ansible launcher (as I imagine we'll need to while we're running it anyway) rather than needing a big cut over14:13
jeblairjhesketh: i should have called it something else.  we can rename the file before we merge it into v314:13
*** tonytan4ever has joined #openstack-infra14:14
jeblairjhesketh: it's not the ansible launching that means we need a cutover, it's the comptelely different way of defining jobs, which is the main feature of v3, which is what we ought to be spending our time working on14:14
jeblairjhesketh: i really do not want v25 to be a distraction14:14
jeblairif we spend much more time on it, i'm just going to abort it and move back to working on v314:15
jeblairi think we can learn some stuff about using ansible which will be useful in v314:15
jeblairbut it is really not any kind of a base to build v3 off of14:15
jeblairit is simply a stop-gap14:15
jheskethsure, I agree the big part is the job definitions etc. However won't we want to use a large part of the ansible-launcher in v3? (I mean parts of it already exist there, but would it make more sense to be the same code base)14:16
jeblairjhesketh: most of the novel code in v2 is built around jenkins-job-builder14:16
*** rbrndt has joined #openstack-infra14:17
phschwartzfungi: That is exactly what I think it is. I will work it through with him14:17
Shrewsjeblair: have you spent much time fleshing out how jobs in v3 will actually look?14:18
jheskethjeblair: I guess thinking about how the node-assigning is proposed to change I agree some more (had forgotten about that)14:18
jeblairShrews: in the spec14:18
jeblairjhesketh: yes that too14:18
jeblairShrews: as far as implementation -- that's about where i am now -- when i get back to v3, i'm going to be working further on collecting the job configurations and passing them to the launcher appropriately14:19
*** matrohon has quit IRC14:20
*** inc0 has quit IRC14:21
*** inc0 has joined #openstack-infra14:21
jheskethjeblair: hmm so I think I'll sleep on it some more but I'm seeing less merit in what I had imagined.. I would like to find a way to keep any similarities in v25 and v3 in sync to minimise reproducing it and making sure we get bugs etc fixed in both14:22
jeblairjhesketh: kk14:22
jheskethjeblair: How did the launcher node you were trying out yesterday go btw?14:23
*** amotoki has quit IRC14:23
jeblairjhesketh: failed for reasons unknown, next step is adding -vvv to ansible-playbook14:23
*** rfolco has quit IRC14:23
*** rossella_s has quit IRC14:23
*** jordanP has joined #openstack-infra14:24
*** rfolco has joined #openstack-infra14:24
jheskethjeblair: my testing found that if it fails during setup it isn't handled very well.. https://review.openstack.org/#/c/317481/1 may help with taht14:24
*** rossella_s has joined #openstack-infra14:24
openstackgerritMerged openstack-infra/project-config: [sahara] make grenade job temporary non-voting  https://review.openstack.org/31576714:25
jeblairjhesketh: oh thanks!14:25
*** zeih has joined #openstack-infra14:26
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Start fleshing out ansible-launcher tests  https://review.openstack.org/31748214:26
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Clean up results returned from ansible-launcher  https://review.openstack.org/31748114:26
* jhesketh notices a silly mistake14:26
jheskethjeblair: changing topics.. it's probably not worth discussing (particularly at the moment), but have you ever considered taskflow for zuul?14:26
jeblairjhesketh: i'm not sure that solves a problem we have right now14:28
jheskethjeblair: thinking abstractly for the future14:28
jeblairjhesketh: i like that zuul is not abstract14:29
mriedemnova would like to add a job to the experimental queue that uses the lvm image backend for libvirt, we need this before refactoring a bunch of that code. it has a +2, so i'm asking for another project-config core to take a look. https://review.openstack.org/#/c/316298/14:30
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Use the overcloud full images from the cache  https://review.openstack.org/31239714:30
*** vhosakot has joined #openstack-infra14:30
*** eeiden has joined #openstack-infra14:30
mriedembtw, i've already tested the job config with a devstack-gate change14:30
*** zeih has quit IRC14:30
*** mtanino has joined #openstack-infra14:30
openstackgerritMerged openstack-infra/project-config: Change groups for Fuel Plugin Cinder Netapp  https://review.openstack.org/31665914:30
openstackgerritPaul Belanger proposed openstack-infra/puppet-nodepool: Add nodepool_ssh_public_key  https://review.openstack.org/31618714:31
pabelangerjhesketh: hopefully that clears up the commit^14:31
*** _amrith_ is now known as amrith14:32
jheskethpabelanger: would it be neater to pass through the pubkey elsehow? eg configdrive?14:32
jheskethjust doesn't feel right to pick up a key if it exists etc14:32
*** bhunter71 has joined #openstack-infra14:33
pabelangerjhesketh: we want to use DIB elements to configure zuul-workers. Since we are moving away from puppet14:33
pabelangernot sure why it would be an issue14:34
jheskethpabelanger: not an issue, it just doesn't seem the best place.. but I'm not doing the work so.. *shrug*14:35
*** markvoelker has joined #openstack-infra14:35
*** markvoelker has quit IRC14:35
jheskethpabelanger: would it make more sense as a custom dib element in system-config that holds the key there?14:35
pabelangerjhesketh: not really. The key is in system-config today, we are simply passing it through to nodepool.o.o to keep a local copy of the public key.14:36
*** markvoelker has joined #openstack-infra14:36
jheskethokay, thanks for the explaination :-)14:37
*** nadya has quit IRC14:38
*** kushal has joined #openstack-infra14:38
*** kushal has quit IRC14:38
*** kushal has joined #openstack-infra14:38
*** markvoelker has quit IRC14:39
*** markvoelker has joined #openstack-infra14:39
*** ajmiller has joined #openstack-infra14:40
jheskethokay I'm off... night all14:40
*** nwkarsten has quit IRC14:40
*** nwkarsten has joined #openstack-infra14:41
*** anteaya has joined #openstack-infra14:42
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Bump tripleo containers job back to experimental  https://review.openstack.org/31754514:42
krotscheckAnyone hear any issues with the npm mirrors? I think those went into rotation this morning...14:45
*** Julien-zte has quit IRC14:45
* krotscheck is technically not at work today, but can be pinged in an emergency14:45
*** nwkarsten has quit IRC14:45
*** Julien-zte has joined #openstack-infra14:45
*** zeih has joined #openstack-infra14:45
*** yamahata has joined #openstack-infra14:46
*** sirushti has quit IRC14:46
openstackgerritMerged openstack-infra/tripleo-ci: Revert check for centos to free HDD space  https://review.openstack.org/31697314:48
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Bump tripleo containers job back to experimental  https://review.openstack.org/31754514:49
*** sirushti has joined #openstack-infra14:49
*** yamamoto has joined #openstack-infra14:50
*** zeih has quit IRC14:50
*** nwkarsten has joined #openstack-infra14:50
*** markvoelker has quit IRC14:51
*** bdemers has joined #openstack-infra14:52
dhellmanninfra core, when you have a few minutes please add https://review.openstack.org/#/c/315761 to your review queue. I'd like to close the testing gap we've been trying to address manually and all of the dependencies for that patch are merged. thanks!14:52
*** markvoelker has joined #openstack-infra14:53
mriedemjeblair: thanks re https://review.openstack.org/#/c/316298/14:54
jeblairmriedem: np14:54
*** dtardivel has joined #openstack-infra14:56
*** laurelm has quit IRC14:56
*** schang has quit IRC14:57
*** abramley has quit IRC14:57
*** markvoelker has quit IRC14:57
*** dougshelley66 has quit IRC14:57
*** gildub has joined #openstack-infra14:57
*** abramley has joined #openstack-infra14:57
*** laurelm has joined #openstack-infra14:58
*** markvoelker has joined #openstack-infra14:58
*** IlyaG_ has quit IRC14:58
*** schang has joined #openstack-infra14:59
*** dougshelley66 has joined #openstack-infra14:59
*** salv-orlando has quit IRC14:59
*** jogo has quit IRC14:59
*** rhallisey has quit IRC14:59
*** rhallisey has joined #openstack-infra14:59
*** salv-orlando has joined #openstack-infra15:00
*** salv-orlando has quit IRC15:00
*** cody-somerville has quit IRC15:00
*** yamahata has quit IRC15:00
*** salv-orlando has joined #openstack-infra15:00
*** jogo has joined #openstack-infra15:00
*** yamahata has joined #openstack-infra15:01
dhellmannthanks, anteaya15:01
anteayadhellmann: welcome15:01
anteayadhellmann: may it do as you intend15:01
*** salv-orlando has quit IRC15:01
*** Qiming has quit IRC15:02
* dhellmann crosses his fingers15:02
*** matrohon has joined #openstack-infra15:02
*** salv-orlando has joined #openstack-infra15:02
*** mixos has joined #openstack-infra15:03
*** claudiub has joined #openstack-infra15:03
* anteaya also crosses her fingers15:03
openstackgerritMerged openstack-infra/project-config: Use regionOne for tripleo-test-cloud-rh1  https://review.openstack.org/31744715:03
*** Julien-zte has quit IRC15:04
anteayamriedem: so I looked in all the obvious places I could think of and couldn't find a way of putting my hand in the air to indicate my intent to attend the nova mid-cycle15:04
anteayamriedem: have I missed something obvious?15:04
*** markvoelker has quit IRC15:04
*** gildub has quit IRC15:04
*** yamamoto has quit IRC15:05
*** tonytan4ever has quit IRC15:05
mriedemanteaya: there was an rsvp email in the -dev list15:05
mriedemfor nova15:05
*** sdake_ is now known as sdake15:06
mriedemanteaya: http://lists.openstack.org/pipermail/openstack-dev/2016-May/093815.html15:06
*** kushal has quit IRC15:06
anteayaI did miss it15:06
anteayathank you15:06
*** Jeffrey4l has joined #openstack-infra15:06
*** kushal has joined #openstack-infra15:07
anteayamriedem: I've missed the deadline in the email15:07
anteayamriedem: may I still sign up?15:07
*** nelsnelson has quit IRC15:07
anteayamriedem: I've signed up, sorry I am late to do so15:08
*** nelsnelson has joined #openstack-infra15:08
mriedemanteaya: i think we can make an exception15:09
mriedemjust this once15:09
anteayathank you15:09
openstackgerritMerged openstack-infra/system-config: Use regionOne for tripleo-test-cloud-rh1  https://review.openstack.org/31745015:09
*** amitgandhinz has quit IRC15:09
*** weshay is now known as weshay_mtg15:09
*** amotoki has joined #openstack-infra15:09
*** amitgandhinz has joined #openstack-infra15:10
openstackgerritMerged openstack-infra/project-config: ensure all projects syncing global requirements use check-requirements  https://review.openstack.org/31576115:11
pabelangerfungi: jeblair: any issues if I bring online the EPEL AFS mirror today? https://review.openstack.org/#/c/315620/15:14
*** nwkarsten has quit IRC15:14
pabelangerfungi: jeblair: I still need to create the AFS bits for mirror.epel15:14
*** nwkarsten has joined #openstack-infra15:15
*** rhochmuth has joined #openstack-infra15:15
*** esikachev has quit IRC15:16
*** kushal has quit IRC15:17
jeblairpabelanger: no objection here15:17
*** amitgandhinz has quit IRC15:17
openstackgerritchenying proposed openstack-infra/project-config: Add smaug fullstack job  https://review.openstack.org/31756615:17
*** amitgandhinz has joined #openstack-infra15:17
anteayais anyone interested/available to help stand up a storyboard-dev server today?15:18
anteayanibalizer tells me that parts are ready to go, he just has internal things he has to work on for the next few days15:19
*** kushal has joined #openstack-infra15:20
*** nwkarsten has quit IRC15:20
*** jlanoux has quit IRC15:21
fungipabelanger: go for it15:22
fungianteaya: i can try to boot one in a sec15:22
clarkbpabelanger: only thing I can think of is available disk but we should have plenty after we added the second volume15:23
anteayafungi: thank you15:23
openstackgerritGraham Hayes proposed openstack-infra/project-config: Add api-ref jobs to Designate  https://review.openstack.org/31700115:24
*** mikelk has quit IRC15:24
*** ayoung has joined #openstack-infra15:28
*** yamamoto has joined #openstack-infra15:30
notmorganN/query greghaynes15:30
*** Liuqing has joined #openstack-infra15:31
fungianteaya: nibalizer: Zara: SotK: i've got the storyboard-dev instance booting now. fingers crossed15:31
anteayayay! \o/15:31
notmorganfungi: woot15:31
* anteaya also crosses her fingers15:31
SotKthanks fungi15:32
fungi4gb flavor, ubuntu trusty (14.04 lts) image15:32
Zara\o/ yay! :D15:32
*** Swami has joined #openstack-infra15:32
anteayafungi: what flavour is the production instance on for comparison15:33
anteayafungi: like when they load test the dev instance15:33
*** dmk0202 has quit IRC15:33
fungianteaya: i used the same flavor as the memory utilization profile in production looked reasonable15:33
anteayafungi: awesome, thank you15:33
anteayaso right now only difference between prod and dev is prod is on precise and dev is on trusty, yeah?15:34
fungiso modulo memory performance differences between precise and trusty (which are hopefully negligible) this should be capable of operating similarly to the production server15:34
fungihopefully deployment will only take a few more minutes before we find out whether it worked/broke15:35
*** hamzy has quit IRC15:35
fungias long as it can bootstrap its puppet manifests to completion, the remaining issues can be solved through reviews and loading a sample dataset15:35
fungiyay! looks like it worked. adding to dns in a moment15:36
*** gyee has joined #openstack-infra15:36
anteayaI'm a yay bot15:36
pabelangerfungi: AFS bits for https://review.openstack.org/#/c/315620 complete, do you mind reviewing15:37
openstackjeblair: Error: "yay" is not a valid command.15:37
pabelangerfungi: when you have spare cycles15:37
anteayaha ha ha15:37
pabelangeror clarkb :)15:37
*** rkukura has joined #openstack-infra15:38
*** jistr has quit IRC15:38
clarkbya I can review in a few15:38
jeblairi'm trying a different approach with zuul-launcher today -- i've pulled an idle node off of jenkins, set it to 'hold' in nodepool, and am manually assigning it to zuul-launcher15:38
pabelangerclarkb: also, the region-name for tripleo is 'regionOne'15:38
jeblairi'll manually run a job on it in a bit too15:38
pabelangerclarkb: keystone did list it15:38
*** akshai has joined #openstack-infra15:38
pabelangerjeblair: acl15:39
*** nadya has joined #openstack-infra15:39
*** maishsk has quit IRC15:39
fungianteaya: nibalizer: Zara: SotK: forward/reverse dns is created now, so should be resolving for you now or within a few minutes15:39
fungii'm able to ssh into it by name already15:39
anteayafungi: ack15:39
fungilooks like we still need to add storyboard developer accounts for ssh access15:40
fungiwas there an open review for them?15:40
anteayafungi: yes15:40
anteayaso far no dns for me15:40
wznoinskany infra/zuul cores around?15:40
SotKfungi: https://review.openstack.org/#/c/316711/15:40
fungianteaya: my browser also says unable to connect, but i think it's that apache isn't running or something15:41
anteayafungi: so after SotK is in users we can add him to https://review.openstack.org/#/c/315706/, which still needs some fixing , I welcome suggestions15:41
pabelangerwznoinsk: likely better just to ask your question then ask if a person is around15:41
anteayafungi: ah okay15:41
fungiSotK: that makes your account details available for us to realize on a server, but i think we still need a realize stanza for you in the openstack_project::storyboard_dev class or something15:41
fungihrm, there is no storyboard_dev class just storyboard15:42
fungiactually, there's no storyboard-dev server in here either http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp15:43
*** nadya has quit IRC15:43
wznoinskpabelanger sure thing, forgive if it's a basic question but it looks like my zuul prefers to run certain younger jobs rather than starting the 2nd job from an older set of jobs (same pipeline, same diskimage used so not a nodepool issue), am I missing something here maybe?15:43
anteayafungi: yes, I hadn't added Adam yet to 31570615:43
*** cloudtrainme has joined #openstack-infra15:44
anteayafungi: I think this patch adds it: https://review.openstack.org/#/c/315706/15:44
fungiaha, so when nibalizer said things were "ready to go" he just meant there were reviews open to implement this server, not that they were merged and i just needed to launch it?15:45
anteayafungi: ah sorry15:45
anteayaI was just wanting a server15:45
pabelangerwznoinsk: same project?15:45
wznoinskan example: http://pasteboard.co/ZCjBRxU.png15:45
anteayaI have fallen down as messenger15:45
anteayayay server15:45
wznoinskpabelanger yes15:45
*** brucet has joined #openstack-infra15:45
anteayaboo not proper patches in place15:45
fungianteaya: yep, this is basically a blank slate server because we don't have any puppet additions for storyboard-dev merged yet15:46
* anteaya notes she got to use the word 'proper' in a non facetious way15:46
anteayafungi: ah15:46
anteayawell yay for server15:46
*** sridhar_ram has joined #openstack-infra15:46
e0nehi. could anybody help me with https://review.openstack.org/287938, please?15:46
pabelangerwznoinsk: we'd have to look at debug.log to see what is happening.  But I would expect zuul to run a patchset from oldest to newest.  But I am not 100% sure of that15:46
*** lakshmiS has joined #openstack-infra15:46
openstackgerritMerged openstack-infra/openstackid-resources: Improved External Order Error  https://review.openstack.org/30862915:47
e0neI don't understand why jenkins put -1 on it15:47
fungianteaya: trying to burn through the open reviews for it now15:47
anteayafungi: ack, standing by to fix or help as you instruct15:47
wznoinskpabelanger: Zuul version: 2.1.1.dev149, I thought I may be missing some tunning setting I missed, I can upload the log somewhere if you like15:47
*** dimtruck is now known as zz_dimtruck15:48
pabelangerwznoinsk: Ya, if you look at the debug.log you can see why zuul is doing the things it is15:48
clarkbwznoinsk: pabelanger if those jobs started and failed in a known way that should be retried that may explain it15:48
*** rhallisey has quit IRC15:48
clarkbbasically job starts in order, fails, gets rescheduled at end of queue15:48
*** andreas_s has quit IRC15:48
wznoinskclarkb, pabelanger I will look into log looking for this15:49
pabelangerRight, I forgot about that15:49
pabelangersure, feel free to PB the debug.log too for an extra set of eyes15:49
openstackgerritThierry Carrez proposed openstack-infra/irc-meetings: Bump yaml2ical to 0.6.1  https://review.openstack.org/31759015:49
openstackgerritMerged openstack-infra/system-config: Add Adam Coldrick to users  https://review.openstack.org/31671115:50
*** |-paul-| has quit IRC15:50
*** kushal has quit IRC15:51
*** hongbin has joined #openstack-infra15:52
*** asselin__ has joined #openstack-infra15:52
*** bpokorny has joined #openstack-infra15:52
*** jlanoux has joined #openstack-infra15:52
jlvillalDoes 'recheck' verify again for merge conflicts? If my patch says it is in merge conflict, but when I checkout master and cherry-pick my patch I get no issues.15:53
*** earlephilhower has joined #openstack-infra15:53
jlvillalSo wondering if I can do 'recheck' to force it to verify again if it is really in merge conflict.15:53
jeblairjlvillal: yes -- but also check whether it has any dependencies which might be in conflict.  it's usually not wrong.15:54
fungijlvillal: also be aware that zuul uses git merge --strategy=resolve in an attempt to approximate the limited merge conflict resolution capabilities of the java jgit library gerrit uses15:54
jlvillaljeblair: As in 'Depends-On' dependency?15:54
*** brucet has quit IRC15:54
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Upgrade to LV 5.2  https://review.openstack.org/30552115:55
fungijlvillal: yes, if zuul sees a merge conflict on anything in the depends-on chain it will complain about a merge conflict on the dependency too15:55
jeblairjlvillal: yes, though also git dependencies (which should generally not cause a conflict themselves, more or less by definition, though *they* may have Depends-On deps with conflicts)15:55
*** asettle has quit IRC15:55
jlvillaljeblair, fungi Thanks. I think I see the issue. I had a patch series (2 patches) and the first patch did change.15:56
*** asselin has quit IRC15:56
jlvillalNot sure if that would cause it, my guess is yes15:56
*** Liuqing has quit IRC15:57
*** armax has joined #openstack-infra15:57
anteayafungi: you okay with me adding SotK to 315706 as well as updating the commit message on that patch as yolanda points out?15:58
clarkbpabelanger: at least with the ceph mirror I think I reused the existing principal for the deb mirrors15:58
fungianteaya: i have that update staged and am about to push it15:58
fungicommit message clarification and all15:58
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to reset backport-potential tags once backports are merged  https://review.openstack.org/29599315:58
*** gokrokve has joined #openstack-infra15:59
anteayafungi: thank you, thought you might15:59
fungianteaya: also needed a rebase on the updated state of the parent change15:59
* anteaya stands down15:59
anteayaah yes, makes sense15:59
fungijust waiting to make sure that gets a good result on the recheck15:59
anteayayup, standing by15:59
*** yamamoto has quit IRC15:59
nibalizerfungi: ooops!15:59
fungii'm pretty sure it's fine, but don't want to rebase twice if not15:59
anteayafungi: agreed15:59
nibalizeri should have said that i think hiera is figured out15:59
clarkbI am not sure we need a new keytab/principal for every single volume16:00
nibalizerso if the patch landed launch node should work16:00
anteayanibalizer: well you kind of told me that but I didn't understand enough to relay that bit16:00
*** cloudtrainme has quit IRC16:00
anteayaso I stuck with 'can I get a storyboard-dev server' it has a nice ring to it16:00
*** yamamoto has joined #openstack-infra16:01
bdemersdougwig: you mentioned at the summit that in your a10-ci repo, you had a project config that was using an array as an arg to a job config ?16:01
bdemers(we were chatting about building multiple branches, if that helps you remember)16:01
*** zz_dimtruck is now known as dimtruck16:02
*** savihou has quit IRC16:02
bdemerswith the goal of basically running py27,34 against multiple configurations16:02
fungiwhen did we make gate-infra-puppet-apply-debian-jessie voting? it looks totally busted (didn't even upload a console log)16:02
anteayapabelanger: I think that was your patch yes?16:03
anteayawas that last week?16:03
clarkbfungi: when we added the image I think since all it tests is we can apply puppet for the slaves on those images iirc16:03
jeblairfungi: it's possible a job that failed yesterday without a console log was due to me16:03
*** brucet_ has joined #openstack-infra16:04
fungijeblair: this one insta-failed just now (my recheck is still running other jobs in the check pipeline)16:04
pabelangerfungi: Ya, it has been passing regularly16:04
jeblair(also, i am using it to test zuul-launcher now, but that should not be visible)16:04
jeblairfungi: oh, it's possible that was me16:04
*** kushal has joined #openstack-infra16:05
pabelangerclarkb: Ya, I created new credentials for epel.  I think I am going to revert that and maybe condense centos / epel into an rpm-mirror user16:05
jeblairsorry about that -- i chose a job that i figured was unlikely to have too many contenders16:05
*** cody-somerville has joined #openstack-infra16:05
*** cody-somerville has quit IRC16:05
*** cody-somerville has joined #openstack-infra16:05
jeblairfungi: ah, i have an idea of how to avoid that -- i'll change the label i'm using, so hopefully it won't run any real jobs16:06
fungijeblair: ahh, no worries. just trying to get these storyboard-dev patches in shape so i can have teh server reachable for the mancheser sprinters16:06
fungijeblair: for reference, it was 24/312124/4/check/gate-infra-puppet-apply-debian-jessie/678fb7c16:06
jeblairfungi: yep, that ran on my node.  sorry :(16:06
*** nmagnezi has quit IRC16:07
*** sdague has quit IRC16:07
fungijust glad we don't have some other unknown new failure mode16:07
* anteaya hands out manchester sprinter jerseys to those round the table16:07
*** cloudtrainme has joined #openstack-infra16:08
anteayathough I have to admit the better characterization is manchester sweet consumers while sitting in a loft room with a nice breeze16:09
anteayabut that is hard to fit on a jersey16:09
anteayaneed a very small font16:09
fungii think we're still having serious fip leaks16:10
pabelangerfungi: yes, I delete some FIPs this morning16:11
fungiseems like as soon as we remove floating ips in osic, it fills right back up again16:11
fungiit was working great for a while... did we change something in shade again?16:12
clarkbfungi: I want to say they upgraded the osic cloud, possibly new interaction between shade and new neutron16:13
kashyapclarkb: When you get a sec, seems like 'gate64' CPU model (remember adding that to devstack-gate? :-)) seems to be not detected in this job -- http://logs.openstack.org/03/315703/1/experimental/gate-tempest-dsvm-multinode-live-migration/1288a98/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-13_21_32_3016:13
kashyap(That's the change where it ('gate64') was defined) -- https://review.openstack.org/#/c/168407/)16:13
fungiclarkb: oh, yep the timing does correspond loosely to their kilo->liberty upgrade16:13
* kashyap is about to step out, will get in a few hours and then check scroll. Or in the morning.16:14
clarkbkashyap: yes we know someone was looking into it16:14
clarkbkashyap: earlier assumptions were that maybe libvirt changed where it looked for the models or the model as defined is invalid16:14
clarkbkashyap: someone will need to take xenail libvirt and sort it out16:14
kashyapOkay.  Is there a ticket / bug that is tracking this?16:14
clarkbno, not yet we don't have more information that "it fails"16:15
kashyapOkay, this came up in today's live-migration meeting, and I volunteered for an action item to follow up with you.  Good that you already have noticed it. :-)16:15
*** nwkarsten has joined #openstack-infra16:16
*** nwkarsten has quit IRC16:16
*** degorenko is now known as _degorenko|afk16:16
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs with specific tag  https://review.openstack.org/29645416:16
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Add tool that annotates bug numbers with details from LP  https://review.openstack.org/29647016:16
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to filter out bugs of specified importance  https://review.openstack.org/29642116:16
openstackgerritIhar Hrachyshka proposed openstack-infra/release-tools: Added tool to scan project repo for new bug fixes in master  https://review.openstack.org/29642016:16
*** nwkarsten has joined #openstack-infra16:16
*** thiagop is now known as thiagop-afk16:17
*** maishsk has joined #openstack-infra16:17
*** rhallisey has joined #openstack-infra16:17
*** rhochmuth has quit IRC16:17
*** matrohon has quit IRC16:18
openstackgerritMerged openstack-infra/system-config: Add EPEL mirror  https://review.openstack.org/31562016:19
clarkbkashyap: the only reason that exists is to make live migration work, it would be most excellent if the individuals caring for live migration could figure it out16:21
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for Jenkins OpenStack Cloud Plugin  https://review.openstack.org/30694816:21
*** jtomasek has quit IRC16:22
anteayafungi: as a time frame, I'm about 30 minutes away from needing to leave for my agreed upon dinner appointment16:22
anteayafungi: if I disappear from channel, know that spritually and emotionally I'm still here supporting the work16:23
anteayago you!16:23
fungianteaya: yep, floating ip leak situation is slowing down check results and i'm waiting on this failure to age out before i can recheck it16:23
*** yamamoto has quit IRC16:23
fungisorry about that16:23
anteayafungi: understood16:23
anteayafungi: don't apologize for normal working conditions16:23
anteayayay that you are able to shepard the server16:23
*** yamamoto has joined #openstack-infra16:24
wznoinskpabelanger, clarkb: ok it looks like the jobs are failing hence they're rescheduled at the end of the queue, thanks for the hint....16:24
anteayathank you16:24
fungiit's still waiting for node assignment on over half its jobs after 22 minutes16:24
anteayafungi: yup16:24
*** jordanP has quit IRC16:24
anteayanot rushing, just wanted to explain my impending absence in advance16:24
fungilooks like it finally got assignments for everything besides trusty16:24
clarkbfungi: is bluebox also leaking more quickly? bluebox has not been updated to newer openstack yet aiui16:26
fungii'm not holding my breath though. there are changes in check waiting 2.5 hours for a trusty node assignment so far16:26
clarkbcomparing the two might help narrow down where the issue is16:26
anteayafungi: yup16:27
kashyapclarkb: Okay, will spend sometime tomorrow in the morning.  I only learned today the word 'Xenial' means Ubuntu 16.04 LTS :-) Me and a couple of others were thinking it's a project named as play on the 'Xen' hypervisor.16:27
fungiclarkb: when i cleaned up leaked fips in bluebox yesterday they didn't seem to _immediately_ return, but i'll try again16:27
*** Apoorva has joined #openstack-infra16:27
*** doug-fis_ has quit IRC16:28
BobBallkashyap: Have you seen http://www.channelpro.co.uk/advice/9797/amd-zen-a-serious-challenge-to-intel-4?  Next year you'll be able to run OpenStack on Xenial on Xen on Zen...16:28
*** e0ne has quit IRC16:28
anteayaBobBall: oooooooh16:28
* kashyap really wraps up for the day, on that note :-016:28
kashyapBobBall: Cool16:28
BobBallIt hurts my head.16:29
anteayaha ha ha16:29
kashyapGuess I tripped off your IRC alert system by mentioning Xen16:29
anteayait is supposed to bring peace to your heart16:29
BobBallyes.  I need to set up regular expression highlighting in irssi :P  Too many people talking about xenial these days ;)16:29
*** yamamoto has quit IRC16:29
*** ajmiller has quit IRC16:29
anteayaBobBall: and it will only get worse16:30
*** yamamoto has joined #openstack-infra16:30
kashyapBobBall: Hehe, that worsened it16:30
*** ajmiller has joined #openstack-infra16:30
* BobBall forces himself to listen to "Always look on the bright side of life" until he calms down16:30
*** hongbin_ has joined #openstack-infra16:30
anteayaBobBall: does that help?16:31
*** kushal has quit IRC16:31
clarkbkashyap: it would be neat if nova could just boot instances with an lcd of all its known hypervisors cpu flags16:31
clarkbbut that might be too restrictive in some envs16:31
BobBallSometimes, when I'm lucky.16:32
*** trown is now known as trown|lunch16:32
*** yamamoto has quit IRC16:32
*** hongbin has quit IRC16:32
*** cloudtrainme has quit IRC16:32
clarkbhowever if cloud knows they want live migration to function it is almost required to do that16:32
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Set IPXE disabled for grenade-ironic  https://review.openstack.org/31761516:32
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Disable IPXE for grenade-ironic  https://review.openstack.org/31761516:32
kashyapclarkb: Yeah, I hear you -- to use the lowest common denominator topic came up in the meeting too.  As oppoesd to the current hard-coding in the gateGate.16:32
*** kushal has joined #openstack-infra16:33
*** brucet_ has quit IRC16:33
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Remove quotes from zuul.conf launcher sites  https://review.openstack.org/31761716:33
*** brucet_ has joined #openstack-infra16:34
anteayamorning zaro16:34
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished  https://review.openstack.org/29732816:37
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Revert "Test with latest master heat"  https://review.openstack.org/30905116:37
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job  https://review.openstack.org/29676516:37
*** akaszuba has quit IRC16:37
*** rhochmuth has joined #openstack-infra16:37
openstackgerritKhai Do proposed openstack-infra/project-config: add a build job for Gerrit importer plugin  https://review.openstack.org/31722316:37
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Enable IPXE for grenade-ironic  https://review.openstack.org/31761516:37
*** yaume has quit IRC16:40
*** _ari_ has quit IRC16:41
*** doug-fish has joined #openstack-infra16:41
*** brucet_ has quit IRC16:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Fix and extend the object model documentation  https://review.openstack.org/31762016:41
*** cloudtrainme has joined #openstack-infra16:42
zarofungi: Gerrit 2.11.9 was just released.  Would you be able to create an 'openstack/2.11.9' branch in openstack-infra/gerrit repo?  should be based off of http://git.openstack.org/cgit/openstack-infra/gerrit/tag/?h=v2.11.916:43
*** kushal has quit IRC16:43
fungizaro: done16:44
clarkbzaro: any chance that includes the event emission fix you wrote?16:45
*** derekh has quit IRC16:46
zaroclarkb: unfortunately not.  that was a semi-big change and can't cherry pick either.  will need to wait for 2.1216:46
zarofungi: thanks!16:46
*** yamamoto has joined #openstack-infra16:46
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Set Designate+Tempest on Mitaka jobs as voting  https://review.openstack.org/31762316:47
fungiclarkb: pabelanger: 15 stale disassociated fips in bluebox, 54 in osic16:48
fungii'm deleting them now16:48
anteayathank you16:48
*** tphummel has joined #openstack-infra16:49
*** heyongli has quit IRC16:50
*** heyongli has joined #openstack-infra16:52
*** armax_ has joined #openstack-infra16:52
mfischfungi: yo16:54
mfischfungi: I think it's this guy causing the bounces16:54
mfischX-MS-Exchange-Inbox-Rules-Loop: tgreenyc@outlook.com16:54
*** armax has quit IRC16:54
*** armax_ is now known as armax16:54
sc68calmfisch: stay classy outlook16:55
mfischevery bounce header I have has his email in it16:55
mfischI pinged him last week, no reply16:55
mfischlets have an admin find him on every openstack ML and remove him16:56
mfischworks at pivotal16:56
zxiiroelectrofelix: So what's the plan for a 1.6.0 release? can we get one this week?16:57
raildofungi: hey, if you have some time, can you take a look on this patch, just to see if you agree with this method to change this job https://review.openstack.org/#/c/313588/16:57
*** abregman has joined #openstack-infra16:57
raildofungi: I want to make similar work for the other jobs16:57
*** abe_music has joined #openstack-infra16:57
anteayame attending the infra meeting today is 50/50, I don't think I have any agenda items up16:58
*** jogo has quit IRC16:58
clarkbraildo: you will also want to exclude liberty and mitaka.16:58
*** jogo has joined #openstack-infra16:59
*** psilvad has joined #openstack-infra16:59
raildoclarkb: ++16:59
fungimfisch: your eyes are keener than mine... so it's a subscriber at outlook.com forwarding to a filtered group at googlegroups.com which is then bouncing back to the original sender17:00
mfischI reached out to him on twitter I dont see him here17:00
*** piet has joined #openstack-infra17:00
*** gyee has quit IRC17:00
mfischi thought it was just me last week17:00
*** hongbin_ has quit IRC17:01
*** hongbin has joined #openstack-infra17:01
*** claudiub has quit IRC17:01
*** javeriak has joined #openstack-infra17:01
*** lucasagomes is now known as lucas-dinner17:02
*** dtantsur is now known as dtantsur|afk17:03
fungimfisch: honestly, when i post to a lot of our lists i get random backscatter from semi-broken subscribers (out of office/vacation notices, authorization failures, et cetera). mailman does a pretty good job of auto-unsubscribing any accounts who bounce back to the listserv itself, but if their mailservers try to be "smart" and figure out a pre-list originator to whom to reply it misses that17:03
anteayafungi: I have to leave now17:03
*** doug-fish has quit IRC17:03
anteayathank you for all the help17:03
fungihave a good dinner, anteaya!17:03
anteayaI'll try to check back in later tonight17:03
anteayafungi: thanks17:03
clarkbfungi: the subject on http://lists.openstack.org/pipermail/openstack/2016-May/016191.html is one of my favorites all time17:04
*** javeriak_ has joined #openstack-infra17:04
*** doug-fish has joined #openstack-infra17:04
*** anteaya has quit IRC17:04
clarkb[SUSPECTED SPAM] OMG it's true17:04
fungiso accurate17:04
jeblairpabelanger, Shrews: https://github.com/ansible/ansible/issues/13323#issuecomment-214553006  i think i'm seeing this17:04
*** javeria__ has joined #openstack-infra17:05
jeblairpabelanger, Shrews: http://paste.openstack.org/show/497389/17:05
*** javeriak has quit IRC17:05
jeblairnote "...FOR USER: None"17:05
jeblairpabelanger, Shrews: that's setting "delegate_to" and "remote_user" on the task17:06
jeblairpabelanger, Shrews: http://paste.openstack.org/show/497390/17:06
*** nwkarsten has quit IRC17:07
*** doug-fis_ has joined #openstack-infra17:07
jeblairpabelanger, Shrews: i'm going to try adding static.openstack.org to the inventory to fix it.  i think that should be an okay fix because we write the playbook as 'hosts: node'.  though still, it scares me slightly to do so.17:08
*** nwkarsten has joined #openstack-infra17:08
*** sshnaidm has quit IRC17:08
*** doug-fish has quit IRC17:08
*** gyee has joined #openstack-infra17:08
clarkbif it is in the inventory and the ssh key is valid what protects it from random job things?17:08
*** javeriak_ has quit IRC17:09
jeblairclarkb: that bit about us writing the inventory as specifying 'hosts: node'17:09
clarkbyou mean playbook?17:09
*** cody-somerville has quit IRC17:09
jeblairclarkb: yeah sorry17:09
*** cody-somerville has joined #openstack-infra17:10
*** cody-somerville has joined #openstack-infra17:10
jeblairthis is something that i think should be okay in v2.5, but could be more problematic in v317:10
wznoinskwhat does infra use to monitor their infrastructure hosts (cpu, memory, space, network) ? I'm thinking of dstat but not sure how to graph it on the web for easier eyeballing17:10
*** eeiden has quit IRC17:10
clarkbwznoinsk: currently cacti for those things17:11
clarkbwznoinsk: http://cacti.openstack.org17:11
jeblairpabelanger, Shrews, clarkb: and yes, having it just be in the inventory is enough to get it to pay attention to the parameter in the playbook task (i didn't even need to set the user in the inventory)17:11
*** earlephilhower has quit IRC17:11
clarkbwznoinsk: in the jobs themselves you can setup dstat and devstack does this for us on many jobs17:11
*** doug-fis_ has quit IRC17:12
*** nwkarsten has quit IRC17:12
*** rhochmuth1 has joined #openstack-infra17:12
*** rhochmuth has quit IRC17:12
*** jerryz has joined #openstack-infra17:12
wznoinskclarkb: I've got nagios deployed so I'll be trying a plugin for dstat for this monitoring framework, does cacti only parse the logfiles of always running dstat on the hosts?17:13
*** dmsimard has quit IRC17:13
*** doug-fish has joined #openstack-infra17:13
fungiwznoinsk: cacti only makes snmp requests and records the data those return17:13
*** Goneri has quit IRC17:13
*** rhochmuth1 has quit IRC17:14
clarkbours is setup to poll snmp every 5 minutes17:14
*** rhochmuth has joined #openstack-infra17:14
fungiwznoinsk: so if you have a custom snmp mib for dstat, then we might be able to set it up to poll that data17:14
*** kushal has joined #openstack-infra17:14
fungiwznoinsk: but cacti doesn't parse any logs17:14
*** jlanoux has quit IRC17:16
*** shashank_hegde has joined #openstack-infra17:16
*** dmsimard has joined #openstack-infra17:16
clarkbfungi: we appear to be leaking in bluebox fairly quickly but not at the same rate as osic17:17
clarkbpossibly due to difference in total quotas (we can spin up about 1/3 the number of machines in bb as osic at any time)17:18
fungiclarkb: yeah, no new stale floating ips in bluebox yet, but 21 new stale floating ips in osic just minutes after i cleaned them up17:18
*** jlanoux has joined #openstack-infra17:18
*** Goneri has joined #openstack-infra17:18
clarkbmaybe we should just ask for ipv6 everywhere and stop ipv4ing completely :/17:18
clarkbunfortunatley then we los eall the clouds17:19
*** inc0 has quit IRC17:20
*** zz_ja is now known as zz_zz_ja17:20
fungilots of boot errors in obh-bhs1 too... "OpenStackCloudException: Error in creating the server."17:21
*** maishsk has quit IRC17:22
clarkbfungi: no valid hosts found?17:22
fungithe traceback doesn't say17:22
fungimight be masking the exception17:22
clarkbwe were hitting those last week, I htink that just means they don't have the capacity for us there17:22
*** earlephilhower has joined #openstack-infra17:22
fungiand we're over quota in rax-ord somehow... "OpenStackCloudException: Error in creating instance (Inner Exception: Quota exceeded for instances, ram: Requested 1, 8192, but already used 200, 1638400 of 200, 1638400 instances, ram (HTTP 403) (Request-ID: req-5b75e88d-407d-41ed-9b07-8474c6d73a23))17:22
clarkbwas hoping it was transient but if it persists we should probably contact them andfigure out what has happend since iirc we have dedicated hosts with our own flavor. Maybe a hypervisor or two are down17:23
clarkbfungi: for that I think it may be related to multinode if we don't account for the usbnodes properly17:23
clarkbbut I hvaen't managed to investigate it at all and that is just a hunch17:23
*** jlanoux has quit IRC17:23
jeblairclarkb: we at least try to17:24
jeblairclarkb: perhaps there is a bug, but it is at least not the case that we didn't just forget to account for them in the allocator :)17:24
jeblairit might be related to all the stop/starts i did yesterday17:24
clarkbjeblair: rax-ord has been doing that for a couple weeks17:24
clarkbpabelanger noticed a little while ago17:24
*** tosky has quit IRC17:25
fungiif that's the case, it's a lot of multinode hosts. we have 162 nodes reported by nodepool list there at the moment and 18 with 2-node in the name17:25
*** johnny__ has joined #openstack-infra17:25
fungiand we have enough ram quota for 20017:25
*** rhochmuth has quit IRC17:25
*** Jeffrey4l has quit IRC17:26
*** sambetts is now known as sambetts|afk17:26
KiallHey folks, a tiny review to set some new jobs as voting? https://review.openstack.org/#/c/317623/ - we've got some changes up at the moment which these are catching and failing (failing correctly) with :)17:26
*** electrofelix has quit IRC17:27
clarkbfungi: jeblair another possibility is bug in usage accounting nova side17:27
clarkbI know quotas are traditionally problematic in nova17:27
fungichecking the theory17:28
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only  https://review.openstack.org/31358817:29
*** flepied has joined #openstack-infra17:29
*** _ari_ has joined #openstack-infra17:30
*** weshay_mtg is now known as weshay17:30
*** cloudtra_ has joined #openstack-infra17:31
openstackgerritKhai Do proposed openstack-infra/system-config: Schedule Gerrit garbage collection  https://review.openstack.org/31763717:31
*** nadya has joined #openstack-infra17:31
fungiweird. if i export OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml OS_CLOUD=openstackjenkins-rax OS_REGION=ORD then openstack server list returns instances from dfw not ord17:31
clarkbfungi: OS_REGION_NAME is the flag iirc17:32
clarkbor --os-region-name as an arg17:32
fungigah, then our documentation is wrong https://git.openstack.org/cgit/openstack-infra/system-config/tree/launch/README#n1817:32
clarkbfungi: no the docs there are correct17:33
*** cloudtrainme has quit IRC17:33
fungiwhat's the difference between OS_REGION and OS_REGION_NAME?17:33
clarkbfungi: in those docs we don't use openstackclient it is substituting that var on the command line for the launch command17:33
clarkbopenstackclient has its own set of args and env vars that it respects17:33
fungigah, you're right17:34
*** IlyaG has joined #openstack-infra17:34
clarkbunfortuantely most of the other clients used OS_REGION and openstackclient made it OS_REGION_NAME17:34
clarkbdouble unfortunately is that we kept the OS_* prefix for everything when openstackclient in theory implies openstack :)17:34
jeblairi can see the arg for env vars but not command line17:34
*** Sukhdev has joined #openstack-infra17:35
*** nadya has quit IRC17:35
jeblairopenstack --openstack=openstack17:35
*** tonytan4ever has joined #openstack-infra17:35
*** psachin has quit IRC17:36
clarkbfungi: if we want one set of vars that we need to set we could normalize those docs though. Just change what vars we substitute in the later commands17:36
fungiaccounting from the nova api seems to match nodepool's view of the situation17:36
jeblairclarkb: ++17:36
*** pilgrimstack has quit IRC17:37
*** salv-orl_ has joined #openstack-infra17:37
*** Sukhdev has quit IRC17:37
*** javeriak has joined #openstack-infra17:38
clarkbya OS_ as an env var prefix makes esnse if you worry about collisions with other vars, they only make sense on the cli flags when you consider the mapping between those and the env vars17:38
clarkbI am guessing the oslo.config makes that automagic so not easy to do it the way eg pip does it17:38
zaroclarkb: would you be able to help here? http://logs.openstack.org/75/316175/3/check/gate-puppet-gerrit-puppet-beaker-rspec-ubuntu-trusty/58cf329/console.html17:38
*** javeria__ has quit IRC17:39
clarkbzaro: let me see17:39
*** amotoki has quit IRC17:39
zaroclarkb: why is it trying to reference, 2.11.4-13-gcb9800e ?17:39
zaroi don't think that's available.17:39
*** salv-orlando has quit IRC17:40
clarkbzaro: http://logs.openstack.org/75/316175/3/check/gate-puppet-gerrit-puppet-beaker-rspec-ubuntu-trusty/58cf329/console.html#_2016-05-17_06_13_55_227 appears to be where it failed17:40
*** kushal has quit IRC17:40
clarkbzaro: it will use the war as specified in the puppet manifests17:40
clarkbzaro: the tests have their own manifest let me find it for you17:41
openstackgerritMorgan Fainberg proposed openstack-infra/project-config: Add python3-testing to ldappool  https://review.openstack.org/31764617:41
clarkbzaro: https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/spec/acceptance/fixtures/default.pp that is what is used during testing17:41
Shrewsjeblair: sorry, was out to lunch. looking now17:41
*** banix has joined #openstack-infra17:42
ihrachysfungi: you were somewhat involved in constraints right?17:42
*** nadya has joined #openstack-infra17:42
clarkbihrachys: it was largely lifeless and Nakato, do you have a question?17:43
*** hamzy has joined #openstack-infra17:43
ihrachysclarkb: ah right, lifeless. yeah, I am trying to make python-neutronclient use constraints, and it fails in mystic way for me.17:43
ihrachysclarkb: I merely added -c... to install_command17:43
clarkbah yes magnum ran into that too, I do not know what the resolution of their discussion with lifeless was17:44
ihrachysclarkb: but when initialize tox env, I get: Could not satisfy constraints for 'python-neutronclient': installation from path or url cannot be constrained to a version17:44
clarkbihrachys: ya it complains about installing via a file or url specificatio nrather than just a package name iirc17:44
ihrachysclarkb: which does not happen when I e.g. remove usedevelop = True17:44
clarkbI do not know why17:44
ihrachysbut... neutron has usedevelop = True, and it works17:44
clarkbihrachys: let me see if  Ican dig up the magnumclient change to see if there was explanation made there17:45
clarkbihrachys: https://review.openstack.org/#/c/301541/ lifeless left a comment there17:45
clarkbapparently the issue is that magnumclient and neutronclient are both in the constraints file17:46
clarkbwhereas neutron or magnum themselves rae not17:46
ihrachysclarkb: oh I now get how it fails :017:46
fungiyeah, you're installing from source and at the same time telling it to constrain to a specific version17:47
ihrachysclarkb: so we should grab the file, remove a line with sed, and then use the modified version?17:47
clarkbihrachys: ya either remove it or modify it to be a file path uri for that specific dep17:47
*** e0ne has joined #openstack-infra17:47
ihrachyswhich would require some hack like in neutron-*aas with tox_install.sh :-|17:47
* clarkb doesn't want to know what tox_install.sh is17:48
ihrachysclarkb: :D17:48
*** earlephilhower has quit IRC17:48
ihrachysfor the sake of scaring everyone http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh17:48
clarkbfungi: so nova and nodepool bot hagree that we have less than 200 instances?17:49
clarkbfungi: in that case I do think it is likely a problem with our quota acocunting17:49
*** hparekh has quit IRC17:50
*** mtanino has quit IRC17:51
clarkbihrachys: I am inclined tosay we should modify the line to be a file path since that is what devstack is doing and it works well17:52
clarkbrather than just remove it17:52
*** hparekh has joined #openstack-infra17:52
*** sarob has joined #openstack-infra17:52
*** piet has quit IRC17:52
*** trown|lunch is now known as trown17:54
*** piet has joined #openstack-infra17:55
*** cody-somerville has quit IRC17:56
ihrachysclarkb: ok, let me look at what I can do17:56
fungiclarkb: yes, so i agree with your earlier comment about nova's quota calculations lagging reality17:57
fungiif sdague were around i'd ask him for details there, but instead i'll pester mriedem17:58
mriedemissues with quotas in nova?! unpossible.17:59
openstackgerritAndreas Florath proposed openstack/diskimage-builder: partitioning-parted: extraced and adapted element  https://review.openstack.org/31393818:01
clarkbfungi: another possile cause is we have leaked some other resource that isn't immediately clear but being factored in, however that razor from occam would imply buggy quota tracking is at fault18:01
*** pvaneck has joined #openstack-infra18:01
*** akshai has quit IRC18:01
fungiclarkb: it's complaining specifically about ram quota18:02
*** claudiub has joined #openstack-infra18:02
*** sshnaidm has joined #openstack-infra18:02
*** nwkarsten has joined #openstack-infra18:02
openstackgerrityolanda.robla proposed openstack-infra/shade: Add magnum baymodel calls to shade.  https://review.openstack.org/30800918:02
*** akshai has joined #openstack-infra18:03
*** vkramskikh_ has quit IRC18:03
*** sarob has quit IRC18:04
*** sarob has joined #openstack-infra18:04
*** pilgrimstack has joined #openstack-infra18:05
notmorganugh. i know we're fighting spam in the wiki but it's blocking me from posting google urls for things like "RSVP forms"18:05
*** ihrachys has quit IRC18:05
clarkbnotmorgan: does it prevent it even when it isn't a proper link?18:06
notmorganclarkb: i keep having to un-shorten it because i only had the twitter-fied link18:06
notmorganclarkb: it blocks it in all cases with the google-shortened urls.18:06
notmorganclarkb: it's minor, just realized what the problem was.18:06
clarkbclearly if you use a url shortener to link off site you are a spammer. Stop spamming notmorgan18:07
* jeblair isn't sure that's a bug :)18:07
notmorganclarkb: lol :)18:07
notmorganclarkb: clearly i'm a spammer then.18:07
*** thiagop-afk is now known as thiagop18:08
*** abe_music has quit IRC18:08
openstackgerritAndreas Florath proposed openstack/diskimage-builder: [WIP] Add: lvm element  https://review.openstack.org/31652918:08
openstackgerritAndreas Florath proposed openstack/diskimage-builder: partitioning-parted: extraced and adapted element  https://review.openstack.org/31393818:08
*** asselin__ has quit IRC18:08
fungiyeah, i'd rather see un-minified urls in published form anyway18:08
fungiseems less bug more feature to me18:08
*** sarob has quit IRC18:09
*** _ari__ has joined #openstack-infra18:09
*** _ari_ has quit IRC18:10
*** inc0 has joined #openstack-infra18:12
openstackgerritIldiko Vancsa proposed openstack-infra/irc-meetings: Add Cinder-Nova-API-interaction meeting series  https://review.openstack.org/31703818:12
*** kzaitsev_mb has quit IRC18:13
*** doug-fis_ has joined #openstack-infra18:15
*** sarob has joined #openstack-infra18:15
clarkbfungi: we will probably need to submit a ticket with rax for the quota thing, maybe double check the flavors are all 8GB first, would be odd to use bigger flavors on some instances but not impossible18:16
*** inc0 has quit IRC18:16
* clarkb tries to figure out the flavor checking18:16
*** akshai has quit IRC18:16
*** _sarob has joined #openstack-infra18:16
*** mtanino has joined #openstack-infra18:18
notmynameit doesn't look like today's -infra meeting has any golang-related stuff (https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting). is that correct? if so, I'll be happy to join. but if not, I'll get lunch before the TC meeting18:18
notmynamefungi: ^18:18
*** doug-fish has quit IRC18:18
clarkbfungi: I count 184 instances of performance1-8 flavor18:18
*** akshai has joined #openstack-infra18:18
clarkbwhich lines up with the all instance count18:19
*** inc0 has joined #openstack-infra18:19
clarkbso ya probably ticket time18:20
funginotmyname: that matches my understanding of the agenda. it wasn't a standing weekly topic so i removed it after last week's discussion18:20
notmynamefungi: ok, great. thanks18:20
*** sarob has quit IRC18:20
funginotmyname: there was an ad-hoc meeting in #openstack-release about go-related stuff around 13:00 utc if you're interested, but that was mostly just bringing the release management team up to speed on the situation so far18:20
e0nefungi: hi. could you please help with failed jobs on cinder? we don't have any idea why it fails:(18:20
zaroclarkb: i didn't notice, that change is going thru recheck. that job passed previously so i'm betting it will pass next time.18:21
notmynamefungi: yeah, I got that in buffer scrollback (and a brief summary from dhellmann). thanks18:22
*** rossella_s has quit IRC18:23
*** rossella_s has joined #openstack-infra18:24
fungie0ne: that's not much to go on. can you elaborate?18:24
*** sridhar_ram has quit IRC18:25
e0nefungi: https://review.openstack.org/#/c/287938/ - jenkins put -1.18:25
clarkbfungi: openstack server list --flavor performance1-8 fwiw18:25
fungie0ne: also i'm preparing my notes for chairing the infra team meeting in about 30 minutes, and want to be able to pay attention to the tc meeting after that, so i'm headed into a not-much-free-time zone18:25
clarkbfungi: pipe that through wc -l and compare against the output from server list without --flavor18:25
fungie0ne: oh, that'll be an easy one to track down... just a sec18:25
e0nefungi: i understand.. maybe somebody else will be able to help?18:26
fungie0ne: i definitely don't see a job named gate-brick-cinderclient-ext-dsvm-functional registered in zuul's gearman18:26
fungie0ne: looking now to see if it should be there according to the config18:27
e0nefungi: here is my patch with that job https://review.openstack.org/#/c/265925/18:27
fungioh, the admin status command isn't returning quickly for me. that's why18:27
*** cody-somerville has joined #openstack-infra18:28
*** vkramskikh_ has joined #openstack-infra18:28
e0nefungi: I'm sorry, I don't understand what do you mean18:28
fungie0ne: i've fallen back to troubleshooting the troubleshooting18:29
*** KaiyanSheng has joined #openstack-infra18:30
*** baoli_ has quit IRC18:30
*** rcernin has quit IRC18:31
KaiyanShengHi All, can I get some reviews for https://review.openstack.org/#/c/316293/ please? This review is about adding monasca tempest tests to all the monasca projects as non-voting for now.18:33
fungie0ne: so it looks like https://review.openstack.org/265925 added that and merged last friday18:33
fungie0ne: when did it start failing like that?18:33
*** tphummel_ has joined #openstack-infra18:34
e0nefungi: we don't have a lot of patches to that project, looks like my is a first and only one is created and apprroved after the job is merged18:35
*** tphummel has quit IRC18:35
*** tphummel_ is now known as tphummel18:35
*** rhallisey has quit IRC18:35
*** rhallisey has joined #openstack-infra18:36
e0nefungi: I've created one more patch https://review.openstack.org/#/c/317668/18:37
fungilooking on jenkins01 the job is definitely there and configured in /var/lib/jenkins/jobs/gate-brick-cinderclient-ext-dsvm-functional (updated May 13 09:00 utc)18:37
*** salv-orl_ has quit IRC18:37
funginot sure yet why it's not registered in gearman18:37
e0neit's still: gate-brick-cinderclient-ext-dsvm-functional: NOT_REGISTERED18:38
*** rockyg has joined #openstack-infra18:38
*** nadya has quit IRC18:39
e0nedid I miss anything in https://review.openstack.org/265925?18:39
funginot that i can tell18:39
fungii'm still digging18:39
*** tphummel has quit IRC18:39
e0nethank you for the help. I appreciate it18:40
fungii'm starting to think we've got something wrong with jenkins or zuul18:40
fungiwonder if any other jobs added since friday are experiencing similar lack of registration18:41
*** whoops has quit IRC18:42
*** akshai has quit IRC18:43
*** tphummel has joined #openstack-infra18:43
openstackgerritHongbin Lu proposed openstack-infra/project-config: Disable python3 for old branches of magnumclient  https://review.openstack.org/31688018:44
openstackgerritThiago da Silva proposed openstack-infra/project-config: add python jobs to pyeclib project  https://review.openstack.org/31767218:44
*** whoops has joined #openstack-infra18:45
*** kevinbenton has quit IRC18:49
*** kevinbenton has joined #openstack-infra18:50
fungihrm, i don't seem to be able to easily toggle the gearman plugin via lynx18:50
fungisomething about it not passing along an expected crumb18:51
clarkbthhats the csrf protection18:51
*** sdague has joined #openstack-infra18:51
fungiyeah, seems like their csrf protection is applied to configuration changes even when not enabled for the rest of the interface18:51
*** tlian has joined #openstack-infra18:52
jeblairi think we have it enabled?18:52
fungiand i guess lynx doesn't respect it18:52
fungioh, did we enable csrf protection as well as firewalling everything?18:52
clarkbyes should be enabled18:52
fungii thought csrf protection was breaking nodepool18:52
clarkbfungi: no we just needed to restart nodepool18:53
clarkbbecayse python jenkins only got the key at startup18:53
fungii wasn't around at that point i guess18:53
clarkber first request18:53
openstackgerritPhilip Marc Schwartz proposed openstack-infra/ansible-puppet: Fix incorrect param type and use of host vars directly.  https://review.openstack.org/31767818:53
tdasilvahello, I'd like to add a new gate job for project https://github.com/openstack/liberasurecode. It is a C library, so I'm wondering if I just need to add a new job-template under project-config/jenkins/jobs18:53
fungiclarkb: makes sense18:53
fungiclarkb: jeblair: yep, configure global security confirms csrf protection is turned on18:54
fungiup to 17 leaked fips on bluebox and 71 leaked fips in osic18:56
clarkbya we turned it on because it would still be exploitable if you logged in via proxy18:56
fungideleting them again18:56
nibalizerfungi: maybe we should be tracking that in graphite18:57
clarkbso we set up layers of protection18:57
funginibalizer: not sure tracking it in graphite right now would help much. it shoots right back up after we clear it out at the moment18:57
clarkbtdasilva: ya you will likely want a template for something like make18:57
clarkbtdasilva: then you can have jobs that make build make test etc18:58
fungigah, it's meeting time and i got sidetracked by bugs again18:58
*** annegentle has joined #openstack-infra18:59
tdasilvaclarkb: got it! thanks. are you aware of any other projects that I could use as a template?18:59
clarkbtdasilva: I dont know of anyone using make curre tly18:59
*** akshai has joined #openstack-infra18:59
*** sridhar_ram has joined #openstack-infra18:59
fungiif someone else gets time later, we need to figure out why e0ne's gate-brick-cinderclient-ext-dsvm-functional job is configured on our jenkins masters since early friday but still not registered in zuul's gearman18:59
clarkbbut you could probably take the tox template and replace tox with make as appropriate18:59
fungiit's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour19:00
*** vkramskikh_ has quit IRC19:00
tdasilvaclarkb: cool, I will take a look, thanks! :-)19:01
*** akshai_ has joined #openstack-infra19:02
pleia2oh hey, meeting time19:02
*** akshai has quit IRC19:04
*** doug-fis_ has quit IRC19:08
*** ihrachys has joined #openstack-infra19:12
*** vkramskikh_ has joined #openstack-infra19:13
*** sdake has quit IRC19:15
*** doug-fish has joined #openstack-infra19:16
ianwpabelanger: hmmm ... in an f23 job -> "msg": "ERROR! mkdir: cannot create directory '/home/jenkins/.ansible/tmp/ansible-tmp-1463511209.13-154152794496369': No space left on device\n",19:19
*** woodster_ has joined #openstack-infra19:20
*** sdake has joined #openstack-infra19:20
*** piet has quit IRC19:22
*** sdake has quit IRC19:22
*** sdake has joined #openstack-infra19:22
*** sdake has quit IRC19:23
*** akshai_ has quit IRC19:25
*** sdake has joined #openstack-infra19:26
*** jyuso1 has quit IRC19:26
*** jyuso1 has joined #openstack-infra19:26
*** arxcruz has quit IRC19:27
jokke_ianw: smalls like running out of inodes19:27
*** dtardivel has quit IRC19:30
*** nadya has joined #openstack-infra19:31
*** comstud has joined #openstack-infra19:31
*** sdake_ has joined #openstack-infra19:32
*** sdake has quit IRC19:32
*** nadya has quit IRC19:33
*** gyee has quit IRC19:33
*** jcoufal has quit IRC19:34
*** banix has quit IRC19:34
*** mriedem has quit IRC19:35
*** kzaitsev_mb has joined #openstack-infra19:35
*** salv-orlando has joined #openstack-infra19:35
*** gokrokve has quit IRC19:36
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds doc publishing jobs for Sphinx extension, os-api-ref  https://review.openstack.org/31769019:36
*** rvasilets__ has quit IRC19:36
*** rhochmuth has joined #openstack-infra19:38
*** ihrachys has quit IRC19:39
*** ihrachys has joined #openstack-infra19:41
*** mriedem has joined #openstack-infra19:41
*** dprince has quit IRC19:41
*** vkramskikh_ has quit IRC19:41
*** pilgrimstack has quit IRC19:42
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW  https://review.openstack.org/31549919:42
*** rvasilets__ has joined #openstack-infra19:43
hasharclarkb: zaro: for the git repack thing.  I cant find analysis traces by myself :-(   Might want to state the problem again on the openstack-infra list and I will catch up dig more19:45
hasharclarkb: zaro: I have no immediate answer / archive :(19:46
hasharas for the git mirrors, I am pretty sure you will have to repack them individually19:48
*** e0ne has quit IRC19:48
*** matrohon has joined #openstack-infra19:49
openstackgerritThiago da Silva proposed openstack-infra/project-config: add python jobs to pyeclib project  https://review.openstack.org/31767219:49
*** e0ne has joined #openstack-infra19:51
*** ihrachys has quit IRC19:52
*** ihrachys has joined #openstack-infra19:52
*** sdake_ has quit IRC19:53
*** e0ne has quit IRC19:54
*** inc0 has quit IRC19:56
openstackgerritMadhuri Kumari proposed openstack-infra/project-config: Add new project Higginsclient to openstack repo  https://review.openstack.org/31769919:56
*** ihrachys has quit IRC19:57
*** sdake has joined #openstack-infra19:58
*** abregman has quit IRC19:58
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class  https://review.openstack.org/29966119:58
*** rcernin has joined #openstack-infra19:58
zarohashar: this was the topic, https://www.mail-archive.com/openstack-infra@lists.openstack.org/msg04264.html19:59
*** piet has joined #openstack-infra20:00
*** doug-fis_ has joined #openstack-infra20:00
zarohashar: in additional to that we wanted to know how our mirrors perfomed before vs after gerrit gc20:00
pabelangersorry for missing our weekly meeting, back now20:01
pabelangerreading up on eavsedrop now20:01
prometheanfirepabelanger: convienent20:01
*** e0ne has joined #openstack-infra20:01
jeblairprometheanfire: using a commont topic in gerrit can help make tracking reviews easier20:02
*** _sarob has quit IRC20:02
zarohashar: was wondering if your doing something similar and wanted to get your experience20:02
*** matrohon has quit IRC20:02
*** bpokorny has quit IRC20:03
*** doug-fi__ has joined #openstack-infra20:03
openstackgerritMerged openstack-infra/tripleo-ci: Use the overcloud full images from the cache  https://review.openstack.org/31239720:03
*** doug-fish has quit IRC20:03
hasharzaro: I have seen that thread though apparently I haven't replied to it. So maybe it was on IRC but the eavedrops logs yields nothing :20:04
openstackgerritJay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs  https://review.openstack.org/31380020:04
*** doug-fis_ has quit IRC20:04
*** e0ne has quit IRC20:04
*** Goneri has quit IRC20:04
hasharzaro: at Wikimedia people clone directly from Gerrit and if that turns out to be a problem they use a shallow clone /  restrict to a single branch  or use the github mirrors20:05
hasharzaro: we mirror everything to github.  Have some replication to Phabricator as well20:05
*** banix has joined #openstack-infra20:06
hasharzaro: and for the Zuul mergers I have some cron jobs that  repack and pack-refs  + a custom script to garbage collect old refs under refs/zuul/*20:06
*** baoli has joined #openstack-infra20:06
*** sarob has joined #openstack-infra20:07
*** inc0 has joined #openstack-infra20:07
*** sarob has quit IRC20:07
*** _sarob has joined #openstack-infra20:07
*** ffrank has joined #openstack-infra20:07
zarojeblair: regarding project renames, would you be able to enlighten me on the impact of change ids changing?  is that something with how zuul works?20:08
hashar+ we run gerrit gc on the repos from time to time20:08
jeblairzaro: people link to "https://review.openstack.org/1234".  gerrit called those "permalinks" for a while.  it would be nice if they kept working.20:09
pabelangerjeblair: I am just looking at your playbook now for delegate_to.  Why are we running the rsync command from static.o.o over just using rsyc from the node?20:09
fungizaro: well, a big thing is that change urls are based on the change number, so we'd end up with all new urls for migrated changes (presumably both merged and open)20:09
pabelangerjeblair: that way, I think we don't need to delegate_to20:09
fungiso any old links to changes for projects from before the rename would cease to be valid20:09
pabelangerianw: where did you see that?20:09
jeblairpabelanger: we want to copy from the node to static.  normally synchronize copies between the node and the controller.20:10
ianwpabelanger: random change -> http://logs.openstack.org/42/306842/5/check/gate-tempest-dsvm-platform-fedora23-nv/bcfe6cc/console.html20:10
jeblairpabelanger: however, even that's not enough, because if we delegate to static, it won't have access to the ssh key there.  so it turns out we need to copy to the controller and then copy to static :(20:10
zarofungi, jeblair : ok.  i thought it was more technical issue with how the tools integrated20:11
*** yamahata has quit IRC20:11
jeblairpabelanger: even then, we still need to delegate to static though, because that's how we specify 'copy to static' in a play where the remote host is otherwise the worker node.20:11
jeblairzaro: nope20:11
pabelangerjeblair: Ya, that's what I was thinking. Unless we are pushing over the SSH key to the node, we'll need to work how jenkins did. Back to master then over to static20:11
jeblairzaro: it's a people issue.20:11
openstackgerritMerged openstack-infra/tripleo-ci: Explode log tars so they can be indexed  https://review.openstack.org/31427220:11
jeblairpabelanger: so yeah, i have a local change that implements that; sorry i haven't pushed it up yet, i'm trying to find all the things we're hitting at once, then will push up fixes.20:12
zaroi guess people are harder to fix..20:12
*** _ari__ has quit IRC20:13
pabelangerjeblair: Can we do dest=rsync://static.o.o/... and delegate_host: local from zuul-launcher20:13
jeblairpabelanger: i don't know -- you want to try that? :)20:13
pabelangerjeblair: which is a push, I think the other way (current) we need to setup mode=pull20:13
jeblairpabelanger: (i've moved on to a jjb issue)20:13
pabelangerjeblair: sure, I can see if I can reproduce locally20:14
jeblairthanks :)20:14
fungizaro: though having the change numbers shift out from under a running zuul might be disruptive in that it will have cached other dependency relationships. i don't know exactly how disruptive in practice20:14
zarohashar: thanks for the info.  i guess your use case doesn't apply to what we want to verify.20:14
jeblairpabelanger: (btw, if we decided to put logs in afs, this would be super easy)20:14
fungizaro: but for example, when you fix a bug and links get added to the bug with the url to the change that fixed it... or appear in a published security advisory...20:15
fungihaving those disappear could be human-painful20:15
*** javeriak has quit IRC20:15
pabelangerjeblair: ++20:15
fungiand i have to wonder whether the git notes get properly updated when making a copy too20:15
*** doug-fish has joined #openstack-infra20:16
hasharzaro: then if you do the repack on gerrit, the mirror should be unaffected imho.  I am pretty sure you would need to repack/gc on the mirrors as well20:16
pabelangerianw: not sure what happen, looks like space issue on HDD20:17
*** fesp has joined #openstack-infra20:17
*** vkramskikh_ has joined #openstack-infra20:17
*** doug-fi__ has quit IRC20:18
zarohashar: are you saying that the replication won't change what's on the mirror?20:18
ianwpabelanger: yeah ... but during setup?  anyway, will keep an eye out, maybe it was just a fluke20:18
*** ccamacho has quit IRC20:18
hasharzaro: I would not sell my soul on that assumption, but intuitively yeah I would expect the mirror / receiving hand to write new packs and eventually end up with loose objects as refs are deleted20:19
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class  https://review.openstack.org/29966120:19
hasharzaro: you would need to ask a git guru really :-(20:19
*** anteaya has joined #openstack-infra20:19
pabelangerianw: possible fluke. I'd check logstash.o.o to see if other jobs had it20:20
*** kzaitsev_mb has quit IRC20:20
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci  https://review.openstack.org/30330620:20
*** kgiusti has left #openstack-infra20:21
zarohashar: ok. i assumed that replication would update the mirror with GCed repo.20:24
hasharzaro: you can look at .git/objects of the nova replicated repo on one of the mirror20:24
hasharzaro: that might confirms20:24
jeblairif so, this may be a good reason to avoid using gerrit gc and just going with the same git gc command everywhere20:25
*** akshai has joined #openstack-infra20:25
*** pwgravel has quit IRC20:26
*** akshai_ has joined #openstack-infra20:26
hasharthen I am not sure how Gerrit behave when you use the git CLI to garabge collect stuff20:26
hasharit might bork20:26
hasharexplode or whatever20:27
zarohashar: you'de probably want to use the same jgit CLI that gets packaged with gerrit20:28
zaroalthought that might be difficult to maintain20:28
jeblairlet's avoid speculation20:28
*** chem`` has joined #openstack-infra20:29
jeblairso far, gerrit devs have said and practice has confirmed that gerrit is a well-behaved git client that respects the file locking necessary for git to work20:29
jeblairthat allows us to run 'git repack' behind gerrit's back20:29
*** dmk0202 has joined #openstack-infra20:29
*** jpr has quit IRC20:29
jeblairperhaps something about 'git gc' would make it impossible to run like that, but we have no evidence of that at the moment.  a gerrit dev saying that they are certain it would not work, or some testing showing that would be useful.20:30
jeblairbut without that, i see no reason to think it wouldn't work.20:30
*** akshai has quit IRC20:30
fungibtw, the faux-cron git repack completed a little while ago, no appreciable change in the usage graph http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all20:30
jeblair(especially since we've been repacking repos behind gerrit's back for *years*)20:30
*** chem` has quit IRC20:31
fungiso my guess is that it is actually running. it indicates in its output that it's repacking things20:31
jeblairfungi: ok.  i suppose it's running as expected, thanks for double checking :)20:31
fungiand it doesn't take an inordinate amount of time to complete20:31
*** bhunter71 has quit IRC20:31
fungiit's just not solving this particular growth issue20:31
fungiso likely we're piling up unreferenced objects20:31
*** sridhar_ram has quit IRC20:32
*** sridhar_ram1 has joined #openstack-infra20:32
zarofungi: yeah, i believe i've confirmed that in my gerrit testing emails.  you need to do a gc to clean up those refs.20:32
hasharzaro: gerrit gc  would use the jgit  so that it is straightforward.  Then has jeblair said,  git gc might just work20:33
anteayafungi: back now, sorry I missed the infra meeting20:33
zarofungi: it's actually the '--prune' that cleans up the loos objects.  gc runs with prune20:35
hasharfungi: git count-objects -v20:36
*** kzaitsev_mb has joined #openstack-infra20:36
hasharfungi: would give you the number of loose objects and disk consumption + a bunch of other metrics20:37
hasharzaro: on a different topic. Jenkins latest LTS strips parameters passed to Gearman function unless they are explicitly defined in the job :D20:39
fungii've got that running for the raw file backend of nova now20:39
mfischinfra team - who is the contact for a stackalytics question?20:39
fungiwill paste stats as soon as it finishes20:39
mfischabout updating it20:39
fungimfisch: the stackalytics core reviewers probably20:39
hasharzaro: there is a workaround, and I have filled a task in Jira for you to get the Gearman plugin to autowhitelist parameters (no idea how to do so though) https://issues.jenkins-ci.org/browse/JENKINS-3488520:39
mfischyeah as soon as I asked I smacked my head20:39
fungimfisch: https://review.openstack.org/#/admin/groups/183,members20:39
*** gyee has joined #openstack-infra20:40
mfischour merger is making things complicated20:40
fungimfisch: not an infra project (or even an official project), just something written by some devs at mirantis20:40
*** rcernin has quit IRC20:41
*** flepied1 has joined #openstack-infra20:41
*** baoli has quit IRC20:43
*** baoli has joined #openstack-infra20:43
*** gildub has joined #openstack-infra20:43
zarohashar: i'm not sure why that's a gearman plugin issue?  i mean shouldn't that be optioned in jenkins core?20:44
*** yamahata has joined #openstack-infra20:44
*** flepied has quit IRC20:44
hasharzaro: Jenkins has a transitional flag to allow untrusted parameters.  From the infra list, the idea would be to have the plugin to autowhitelist the parameters20:46
*** tonytan_brb has joined #openstack-infra20:46
*** rfolco has quit IRC20:47
hasharzaro: so one can still prevent untrusted parameters from other sources; And maybe later Jenkins will enforce it / remove the option20:47
fungiso this is the count-objects data for the nova repo on review.o.o: http://paste.openstack.org/show/497418/20:47
*** rfolco has joined #openstack-infra20:47
fungiinteresting that it flags no garbage there20:47
*** raildo is now known as raildo-afk20:47
fungii guess the "size" there is in kilobytes20:48
hasharthen you have 4 million loose objects :D20:48
mugsiewhen someone has a chance, could I get some reviews on https://review.openstack.org/#/c/317001/ please?20:48
*** gildub has quit IRC20:49
*** tonytan4ever has quit IRC20:49
sdaguemugsie: +220:49
hasharzaro: dont obsess about it though. I have mentioned it for discussion and as a warning in case you upgrade to latest Jenkins version20:49
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished  https://review.openstack.org/29732820:49
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job  https://review.openstack.org/29676520:49
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't pipe yes to a function  https://review.openstack.org/31773020:49
mugsiesdague: ty20:49
hasharfungi: looked at the man page for git-count-objects  that is in KiB  and there is a -H option for human friendly display !20:50
*** rhallisey has quit IRC20:50
*** kzaitsev_mb has quit IRC20:50
hasharfungi: then I am not sure why Gerrit would have so many loose objects.  Maybe all the merges attempts it did, bunch of caches etc.  Then 1.3 millions in pack vs  4 millions loose objects .. that is concerning20:51
rm_workI assume the people who maintain DIB hang out here? :P20:51
greghaynesThey all left20:51
rm_work^^ per that I can only use git / tar? there's no way to copy from the local filesystem?20:52
rm_workor like20:52
rm_work... I need the files from the current directory on the machine that's building...20:52
clarkbjust cp them20:53
greghaynesrm_work: there is not, but you can copy in files in during the extra-data.d phase20:53
rm_workin that case would I just completely ignore that and copy them?20:53
rm_workok yeah20:53
*** shardy has quit IRC20:53
*** kzaitsev_mb has joined #openstack-infra20:53
rm_worknot sure why that didn't occur to me until just now when I was trying to ask, lol20:54
hasharfungi: for comparison  mediawiki/core on Wikmedia Gerrit http://paste.openstack.org/show/497419/   1 Million in pack but a few hundred loose objects20:54
rm_workgreghaynes: is there a doc for extra-data.d ?20:54
fungihashar: thanks, that makes some sense20:55
greghaynesrm_work: kind of - http://docs.openstack.org/developer/diskimage-builder/developer/developing_elements.html#phase-subdirectories20:55
fungihashar: we've not been garbage-collecting or pruning at all20:55
fungihashar: only repacking20:55
rm_workhmm k, thanks20:55
greghaynesrm_work: your best bet is to probably cargo the devuser element20:55
fungihashar: specifically, git repack -afd20:55
Kiallfungi: how exactly are you participating in the TC meet, and having a technical chat here... That's just not something I can manage ;)20:56
greghaynesrm_work: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/devuser/extra-data.d/50-devuser20:56
rm_workok, thanks. will give it a shot20:56
fungiKiall: "participating" is a loose concept around here20:56
greghaynesrm_work: actually, http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/apt-sources/extra-data.d/99-override-default-apt-sources is probably easier to cargo (the sources.list part)20:56
*** psilvad has quit IRC20:56
fungiKiall: since i'm not on the tc, i can get away with mostly lurking that meeting20:56
KiallI saw messages from you in TC, switched over here for something else.. and more messages from you popped up ;)20:57
pabelangerclarkb: I haven't heard back from OVH about switching to qcow2. Did you want to try and reach out to them?20:57
pabelangerclarkb: re: https://review.openstack.org/#/c/315680/20:57
clarkbpabelanger: maybe it is a french holiday? I think we can go ahead and switch if we want and beg forgiveness if issues ensue20:57
pabelangerclarkb: WFM20:58
*** rprakash has joined #openstack-infra20:58
fungipabelanger: clarkb: oh, also i totally didn't talk to mordred because he's on vacation and nowhere near a computer, but apparently the change to add fip cleanup in nodepool is missing a followup change to configure it to happen. which explains the current osic/bluebox situation i guess?20:58
clarkbI actually am in theory getting keys to a house shortly and left my laptop at current place derp20:58
*** sdake_ has joined #openstack-infra20:58
*** thorst has quit IRC20:59
clarkbfuno we shouldnt leak this quickly even with that cleanup20:59
*** aysyd has quit IRC20:59
clarkbfungi ^20:59
clarkbcleanup was there to handlr a few leaks per day not hundred an hour20:59
* fungi will answer to most misspellings of his nick20:59
rprakash#info rprakash20:59
pabelangerfungi: Ya, we landed the cleanup FIP stuff in nodepool, but it is not enabled.20:59
*** salv-orlando has quit IRC20:59
pabelangerbut agree with clarkb, we seem to leaking a lot now21:00
rprakash#help for creating new project, can I ask some questions?21:00
*** salv-orlando has joined #openstack-infra21:00
*** sridhar_ram1 has quit IRC21:00
openstackgerritMerged openstack-infra/system-config: Puppetboard renamed the module  https://review.openstack.org/31387621:00
*** sdake has quit IRC21:00
jeblairclarkb, pabelanger, fungi: agree, but i think we should enable the fip cleanup and then continue to pursure the cause of the leak utilizing the logs to tell us how bad it is (rather than how annoyed fungi is)21:01
*** rfolco has quit IRC21:01
*** julim has quit IRC21:01
*** trown is now known as trown|outtypewww21:02
*** kzaitsev_mb has quit IRC21:02
clarkbI think it might even be worth reverting... at least to see if the issue is relatrd to shade or not21:02
pabelangerI believe we have a good way to reproduce the issue too21:02
clarkbwe never leaked this badly in osic on the old code, wpuld be good info to know if old code leaks like this now21:02
hasharfungi: you could ask ostriches in #wikimedia-releng on my behalf.  He is the maintainer for Gerrit and must have some maintenance trick.  Your nova repo seems to have the same size as our mediawiki/core one21:02
hasharfungi: so any tip ostriches have would probably apply to your case21:02
*** bdemers has quit IRC21:03
fungihashar: well, it sounds like you're actually pruning/garbage-collecting yours? in which case that would totally explain the difference21:03
*** thorst_ has joined #openstack-infra21:03
jeblairclarkb: reverting sounds nasty; i think we're past that now.21:03
fungias i said, we haven't been yet, and we're still evaluating whether we should (and whether we should use git or jgit via gerrit to do so)21:04
clarkbjeblair: how so?21:04
jeblairclarkb: we've landed some stuff post-shade in nodepool21:04
*** asettle has joined #openstack-infra21:04
clarkbya there may be conflicts I havent tried21:04
fungijeblair: clarkb: we're quite a few changes down the road from the shade switch, and have zuul v2.5 support implemented21:04
fungiworried that will be unpleasant to untangle21:04
clarkbits just been impossible to isolate the problems and we dont know if it is shade or the cloud21:04
jeblairclarkb: are you working on that?21:05
clarkbon what?21:05
jeblairclarkb: are you working on isolating the problem?21:05
clarkbno, mordred cloudnull and armax were21:05
jeblairi'm trying to figure out if we've hit some kind of roadblock in your work21:05
clarkbbut aiui they didnt get anywhere21:05
jeblairyeah, so mordred is on vacation21:05
fungiit did get significantly worse a few days ago, so if we didn't change nodepool then it seems likely related to a behavior change from their kilo->liberty upgrade21:05
jeblairand is not expected to make progress for a while21:05
*** inc0 has quit IRC21:05
jeblairwhich is why i think we should enable the fip cleanup so it isn't bothering us as much21:06
*** jpr has joined #openstack-infra21:06
jeblairand then, when a sufficient quorum of mordred/cloudnull/armax/Shrews are around to continue, they can without us being harried21:06
fungiwhich still doesn't exonerate shade/nodepool, but at least helps us narrow the field a bit21:06
hasharfungi: I am pretty sure we run  "gerrit gc" on all repos on a something-ly basis.  ostriches would confirm he would be the one running them21:06
anteayafungi: what if anything can I do to help the storyboard-dev effort move along?21:07
clarkbfwiw the pre shade leak rate was a handful of IPs per week21:07
fungii'm definitely not keeping up with fip cleanup with enough regularity to prevent it from causing significant impact on our node starvation21:07
clarkbbut it may or may not be worse now due to cloud or shade21:07
*** thorst_ has quit IRC21:08
hasharclarkb: if it can help the Wikimedia Nodepool is  pre shade21:08
jeblairis there a patch to turn on the cleanup?21:08
hasharclarkb: we are running 0.1.1 tag21:09
pabelangerjeblair: I have not seen one21:09
jeblairpabelanger: do you have a moment to write one?21:10
fungianteaya: not sure. it looks like the failed 312124 check results finally reported halfway into the infra meeting21:10
pabelangerjeblair: sure21:10
anteayafungi: ah okay21:10
jeblairpabelanger: thanks!21:10
fungianteaya: at least the other failure was transient, so i'm confident enough i can push the update to the child patch as well21:10
*** dmk0202 has quit IRC21:10
*** ewilson has quit IRC21:11
anteayafungi: awesome, thank you21:11
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Realize SotK and Zara on the storyboard-dev server  https://review.openstack.org/31570621:11
*** amotoki has joined #openstack-infra21:11
fungiand there it is21:11
jeblairi was trying to figure out what was taking so long on this zuul-launcher job...21:12
jeblairand the answer was -- it's working! :)21:12
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Use clean-floating-ips for yaml setting  https://review.openstack.org/31773421:13
pabelangerjeblair: trivial fix, for style ^21:13
jeblairpabelanger: thanks :)21:14
*** sdague has quit IRC21:14
*** ldnunes has quit IRC21:15
openstackgerritKaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add check-requirements check job for monasca projects  https://review.openstack.org/31773521:15
fungiclarkb: so what was that you were saying about filtering openstack server list by flavor in rax ord?21:15
fungii'm finally able to circle around and look into that some more21:16
clarkbfungi: just to confirm we dont have anything booted witg bigger flavors using more ram21:16
fungioh, right21:16
clarkbwe dont but you can pass --flavor performance1-8 to server list to confirm21:16
fungiyep, identical counts21:17
*** bpokorny has joined #openstack-infra21:17
fungialso well below the 200 we should be able to launch21:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Enable clean-floating-ips for OVH and BlueBox  https://review.openstack.org/31773721:17
armaxjeblair: you are talking about bug 1581220?21:17
openstackbug 1581220 in neutron "race leads same floating ip to be associated to separate ports" [High,Incomplete] https://launchpad.net/bugs/158122021:17
pabelangerfungi: clarkb: jeblair: ^21:18
fungiclarkb: i'm seeing similar errors in osic too21:18
fungiOpenStackCloudException: Error in creating instance (Inner Exception: Quota exceeded for instances, ram: Requested 1, 8192, but already used 100, 819200 of 100, 819200 instances, ram (HTTP 403) (Request-ID: req-ca6abcbb-96e1-44ff-9679-ae095037a897))21:18
*** baoli has quit IRC21:18
mescanefhello. please review some puppet code in https://review.openstack.org/#/c/303306/ and https://review.openstack.org/#/c/299661/21:18
*** baoli has joined #openstack-infra21:19
*** rhochmuth has quit IRC21:19
*** daemontool has joined #openstack-infra21:20
*** thorst_ has joined #openstack-infra21:21
jeblairarmax: i'm not certain, sorry.21:21
fungiclarkb: there we're hovering at/near the quota too. i wonder if we're leaking instances and hitting over quota condition intermittently until it cleans up21:21
armaxjeblair: ok21:21
clarkbor possibly something got lost in the upgrade21:22
*** kzaitsev_mb has joined #openstack-infra21:22
*** doug-fis_ has joined #openstack-infra21:22
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class  https://review.openstack.org/29966121:22
rprakash#info fungi prakash here you gave me http://docs.openstack.org/project-team-guide/ for project ceration but still not able to create a project, is there atemplate or gi clone I can use to create a new project?21:22
jeblairpabelanger: telnet 808821:23
*** doug-fish has quit IRC21:23
fungirprakash: can you provide any details on your inability to create a project? the documented steps are http://docs.openstack.org/infra/manual/creators.html21:24
fungirprakash: how far did you get with that?21:24
rprakashi was not able to follow as what template in zull do I use and if so how I gi clone and submit specs along with the project creation21:25
*** hamzy has quit IRC21:26
*** tonytan_brb has quit IRC21:26
rm_workgreghaynes: do I need to do somethign special to get the extra-data.d stuff to run? it seems like it's just ... not doing it. don't see any mention of it in the logs, directory looks right and the script is executable :/21:26
clarkbrm_work: is your file executable?21:26
clarkboh derp21:26
clarkbfinish reading21:26
fungirprakash: i don't understand what you're asking with regard to "submit specs"21:26
rprakash#info like I have created a project in https://launchpad.net/meghdwar with all project details and how do I bring it under openstack/meghdwar21:27
clarkbrm_work: maybe the dir isnt traversable?21:27
clarkbrm_work: shoukd be 55521:27
rm_work-rwxrwxr-x 1 stack stack 629 May 17 21:08 elements/amphora-agent-ubuntu/extra-data.d/50-copy-local-source-files21:27
rm_workdrwxrwxr-x 2 stack stack 4096 May 17 21:08 elements/amphora-agent-ubuntu/extra-data.d/21:27
*** annegentle has quit IRC21:27
*** gildub has joined #openstack-infra21:27
fungirprakash: it's covered in the "adding the project to the ci system" section at http://docs.openstack.org/infra/manual/creators.html#adding-the-project-to-the-ci-system21:27
*** matt-borland has quit IRC21:28
pabelangerjeblair: Niiice21:28
KaiyanShengHi I have a question about this review generated by OpenStack Proposal Bot https://review.openstack.org/#/c/315792/. How can I trace back to which version change in global-requirments lead to this review? Thank you!21:28
*** weshay has quit IRC21:28
*** kzaitsev_mb has quit IRC21:28
pabelangerjeblair: very cool21:28
fungirprakash: is it possible your confusion is that you've never previously pushed proposed changes to any openstack project before, and so you need to start from the new contributor documentation?21:28
rm_workbasically, there isn't anywhere i need to add the script to a list of stuff or anything, it SHOULD be autodetected21:28
rm_workjust making sure i'm not missing something dumb21:28
clarkbya it should be21:28
clarkbdib just collects all those things based on your elements then run parts them21:29
clarkbis any other part of that element runnig?21:29
fungirprakash: our project creators guide assumes a basic level of experience submitting changes to other existing openstack projects in gerrit21:29
pabelangerjeblair: is the change live to display the telnet links on status.o.o/zuul ?21:29
rprakash#info fungi the guide stated name requiements specs and other details to be ready with and I have it on the link https://launchpad .net/meghdwar now how do i get a name psace in https://launchpad.net/openstack/meghdwar and I could not folow what zull was suggesting like yaml file and experimental project etc.21:29
clarkbKaiyanSheng: it isnt necessarily a single change. you can look at the git log in openstack/requirements to see the various changes21:30
jeblairpabelanger: no, that's disabled in the javascript right now; it will be easy to switch them back on once we have this running21:30
pabelangerjeblair: ack21:30
rprakashBut this is not an existing project for me to follow gerrit bug submission, if that is the case which template shall I use?21:30
*** jamielennox is now known as jamielennox|away21:30
pabelangerjeblair: so, that IP is the zuul-worker right?21:30
fungirprakash: the project creators guide walks you through that, but our infrastructure is maintained through the same workflows as any openstack project, so you'll need to first get familiar with using git and git-review to create and push a change for review to the openstack-infra/project-config project21:31
jeblairpabelanger: yep21:31
jeblairpabelanger: 3 things on my punch list right now: a) log copying still broken; b) i have a bug in my tail routine for console logs; c) we need to open port 8088 on the workers21:32
fungirprakash: the configuration change you propose to the openstack-infra/project-config git repository will be reviewed and, once it's correct, will be consumed by our project creation automation to implement the configuration you've written21:32
rm_workclarkb: yeah the rest is running21:32
pabelangerjeblair: ack21:33
jeblairpabelanger: if you want to take (c) that'd be cool21:33
pabelangerjeblair: great work BTW!21:33
pabelangerjeblair: sure21:33
rm_workand failing because the extra-data.d script isn't running T_T21:33
*** ayoung has quit IRC21:33
clarkbrm_work: ok I would need to see logs to further debug21:33
*** kzaitsev_mb has joined #openstack-infra21:33
pabelangerwe also have an EPEL mirror now: http://mirror.dfw.rax.openstack.org/epel/21:33
pabelangerstill need to test it, but looks good so far21:34
jeblairpabelanger: (i think we may have existing iptables config for workers; maybe in puppet somewhere; might be as simple as adding to that list)21:34
jeblairpabelanger: at least, i noticed that ports 8000, 8003, and 8004 are open21:34
rprakash#info fungi you mean I git clone https://git.openstack.org/cgit/openstack-infra/project-config/ and start from their by reviewing, adding specs rst file, readme ect and submit back?21:34
pabelangerjeblair: sure, looking now21:34
*** lakshmiS has quit IRC21:35
fungirprakash: clone that repo, but it's not a template for your new repo. it's configuration for our continuous integration and code hosting platform21:35
*** cody-somerville has quit IRC21:36
*** ccamacho has joined #openstack-infra21:36
fungirprakash: the http://docs.openstack.org/infra/manual/creators.html#adding-the-project-to-the-ci-system section tells you what sections of which files to change or add entries in21:36
KaiyanSheng@clarkb Thank you. So does the openstack proposal bot job get triggered every several days or someone manually triggered it?21:36
fungirprakash: then you push that change to gerrit for review21:36
rm_workclarkb: yeah, i'll try to get some... i'm looking through currently too21:37
clarkbKaiyanSheng: it gets triggered when code merges to openstack/requirements but it includes all outstanding updates when it runa21:37
rprakash#info OK I will clone and see if I need to add proejct name etc. there for infra team to take this eperimental project to try set repo...any other hints I wil follow the guide..Thanks Prakash21:37
fungirprakash: if you don't have an existing project git repo you want imported during the new project creation process, it will start you with a simple, basically empty one and then you can begin building your project by sending changes for review to that project once it exists (once our project creation automation runs on your project-config change)21:37
KaiyanSheng@clarkb Thank you! For https://review.openstack.org/#/c/315792/ seems like it's keep failing, does monasca need to do anything to try to fix it?21:38
*** thorst_ has quit IRC21:38
fungirprakash: separately, there is a tool called cookiecutter we have some templates for, documented at http://docs.openstack.org/infra/manual/creators.html#preparing-a-new-git-repository-using-cookiecutter21:38
rprakash#fungi apprecaite all your inputs and will try follow and see it gets in to experimental mode to start with -Thanks Prakash21:38
fungirprakash: you can worry about that part once your basically empty project exists in gerrit21:38
*** Swami_ has joined #openstack-infra21:38
*** thorst_ has joined #openstack-infra21:39
*** jamielennox|away is now known as jamielennox21:39
clarkbKaiyanSheng: yes it is failing due to a known issue with constraints and projects in the constraints list, let me get you a link with details on how to address it21:39
KaiyanShengAwesome :-) Thank you!21:39
*** doug-fish has joined #openstack-infra21:40
clarkbhttps://review.openstack.org/#/c/301541/ see lifeless' comment21:40
*** kzaitsev_mb has quit IRC21:41
*** ccamacho has quit IRC21:41
KaiyanSheng@clarkb got it. Thanks!21:41
*** annegentle has joined #openstack-infra21:42
fungii'm going to step out and get something to eat, then back to hopefully keep plugging at the storyboard-dev situation21:42
Zaraappreciate it21:42
*** Swami has quit IRC21:42
openstackgerritPaul Belanger proposed openstack-infra/system-config: Enable tcp/8088 on jenkins slaves  https://review.openstack.org/31774321:42
jeblairpabelanger: did you manage to get that rsync command working?21:42
*** doug-fis_ has quit IRC21:42
*** tphummel has quit IRC21:42
pabelangerjeblair: ^ I think that is what we need, I'll fire up a test locally in a few hours21:43
*** kzaitsev_mb has joined #openstack-infra21:43
pabelangerjeblair: not yet21:43
*** thorst_ has quit IRC21:43
jeblairpabelanger: i think we may either need to do that approach, install the jenkins/zuul ssh key on static.o.o, or use local_action21:43
*** hashar has quit IRC21:44
openstackgerritPaul Belanger proposed openstack-infra/system-config: Enable tcp/8088 on jenkins slaves  https://review.openstack.org/31774321:44
pabelangerjeblair: agreed.21:45
*** edmondsw has quit IRC21:45
*** ccamacho has joined #openstack-infra21:45
rm_workclarkb: nm i think i found it :P21:45
clarkbrm_work: what is the issue?21:46
wznoinskhi infra, I'm thinking of touching /etc/fixed_disk_layout to avoid devstack-gate creating swap, I have 16GB for host VMs and always see more than 30% free even during tempest full, I do understand disk space is cheaper when you buy a cloud VM but would there be any reasons why not to disable the swap? (I'm ignoring the fact some bad code can go highwire as there isn't amount of ram/swap it would be enough in most of these situatio21:46
rm_workit's running, but there's an error somewhere in my paths21:46
*** kzaitsev_mb has quit IRC21:48
*** annegentle has quit IRC21:49
*** ccamacho has quit IRC21:50
*** tqtran has joined #openstack-infra21:52
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add epel support for configure_mirror.sh  https://review.openstack.org/31774721:52
pabelangerclarkb: fungi: jeblair: tested / working locally ^21:52
*** annegentle has joined #openstack-infra21:52
*** asettle has quit IRC21:53
jeblairpabelanger, Shrews, clarkb: do you have a minute for an ansible brainstorming session?21:54
clarkbI derped and left laptop at home but can try to phone it21:54
*** sdake_ is now known as sdake21:54
*** maishsk has joined #openstack-infra21:54
*** mriedem has quit IRC21:55
jeblairpabelanger, Shrews, clarkb: https://etherpad.openstack.org/p/jZjPISYSOu21:56
jeblairi'm trying to figure out how to scp the logs from the worker to static.o.o21:56
jeblairi'm still typing in the etherpad, gimme a sec21:56
jeblairclarkb: does etherpad on phone work for you?21:57
clarkbjeblair: ya21:57
SpamapSSo, I have a general purpose Zuul question....22:01
SpamapSI have a job which may run on changes to let's say, 10 repos22:01
SpamapSone of those repos builds a package if there has been a change22:01
*** netsin has quit IRC22:02
SpamapS(it looks in the repo, compares to package repo, and if it's different, builds and uploads the new version)22:02
*** netsin has joined #openstack-infra22:02
SpamapSthat version I just built.. I want to get that from the place where the job is running, back into... something.. as a build artifact22:02
*** mixos has quit IRC22:03
jeblairpabelanger, Shrews, clarkb: okay, i've filled out the etherpad with my ideas of how we could copy the logs off;  1 and 2 don't work (tried); 3 and 4 are ways that might work; can you think of any others?22:03
*** daemontool has quit IRC22:03
jeblairSpamapS: i think we need precise terminology here -- you said a repo builds a change; that doesn't compute22:03
SpamapSI suppose if the gate passes and my job is merged, I can discern the new version number and use the superrepo.22:03
jeblairSpamapS: (sorry, i'm not trying to be pedantic; it may have a substantial effect on how to think about this)22:04
SpamapSjeblair: the repo defines a build.22:04
SpamapSdude you don't have to explain taxonomy explosion to me. ;)22:04
*** weshay has joined #openstack-infra22:04
jeblairSpamapS: you just smurf the smurf22:04
SpamapSI understand that 98% of all discussions in openstack are about which meanings we're using for all the words right now.22:04
SpamapSjeblair: don't smurf the smurf without a smurf though22:04
SpamapSso, I have a repo which defines a package build22:05
SpamapSchanges to it are fed into a zuul pipeline22:05
SpamapSa job which is running in that pipeline will compare the package repository to the version defined in the build, and if that version does not exist yet, it will build the package.22:05
SpamapSbacking up22:06
*** doug-fish has quit IRC22:06
SpamapSI have a git repository22:06
SpamapSwhich defines a package build :)22:06
*** psilvad has joined #openstack-infra22:06
jeblairSpamapS: we have a number of jobs which put both intermediate or final artifacts on to tarballs.openstack.org22:08
*** sigmavirus24 is now known as sigmavirus24_awa22:08
jeblairSpamapS: (jobs that build, say, diskimages do this sort of thing)22:08
jeblairSpamapS: does that sound like what you need?22:08
SpamapSjeblair: yes I think that's the pattern I need. I want to build a {bag of bytes} and be able to determine which {bag of bytes} the pipeline produced.22:09
SpamapSso that I can use the same {bag of bytes} with the same {bag of commits} elsewhere.22:10
jrollSpamapS: sounds like you want a publisher stanza similar to https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ironic-python-agent-jobs.yaml#L4922:10
jrollSpamapS: that one in particular is run in the post queue, but I assume you could do it elsewhere if you desired (we use post to only do it once merged)22:11
jeblairSpamapS: is it a temporary or semi-permanent blob?  like, does this make sense only in the context of check/gate builds for speculative changes?22:11
pabelangerjeblair: I have to run, but will review etherpad once I get back22:11
jeblairSpamapS: or is it as jroll suggests, something that might run in the post queue to update something public semi/permanent?22:11
*** anteaya has quit IRC22:11
jrollyeah, that's a good question22:11
*** bhunter71 has joined #openstack-infra22:12
SpamapSjeblair: semi-permanent. It is very important that the same build artifact used in passing the gate be combined with the other artifacts.22:12
SpamapSA post job would not make sense no.22:13
SpamapSwell a post job might make sense for _combining_ the two things22:13
jeblairSpamapS: okay, so we're talking about the pre-merge context22:13
SpamapS(the two things being the package and the commits)22:13
SpamapSyes pre-merge22:13
*** markvoelker has joined #openstack-infra22:14
SpamapSOk hm, actually I think I may have my answer now that we've gone through externalizing it from my head.22:14
*** maishsk_ has joined #openstack-infra22:15
jeblairSpamapS: i've been thinking about this a lot for zuulv3, and it's something i'd like to solve better there.... but basically the way i think about it is that each zuul item has a proposed future state of the world22:15
*** baoli has quit IRC22:15
SpamapSjeblair: go on22:16
*** baoli has joined #openstack-infra22:16
jeblairSpamapS: for git, zuul handles that internally -- it creates the proposed future state of all the git repos, and it gives that state a name, and puts that name in the git repo as the zuul ref for the item22:16
*** maishsk has quit IRC22:16
*** maishsk_ is now known as maishsk22:16
jeblairSpamapS: but we might want to construct other aspects of a speculative state -- like a container, image, package repo, etc.22:17
*** tpsilva has quit IRC22:17
KaiyanSheng@clarkb Thank you for your link. I still don't understand why https://review.openstack.org/#/c/315792/4 failed to install python-monascaclient but not earlier? We changed our tox file almost a month ago to follow upper-ocnstraint and it was fine at the beginning.22:18
SpamapSfor my problem space, what I might do is just require that a user change the package version in their patch...22:18
SpamapSin this way, the zuul ref does contain the reference to the package version22:18
*** jpr has quit IRC22:19
*** xyang1 has quit IRC22:19
*** rockyg has quit IRC22:19
SpamapSso when it lands, I can use the https://review.openstack.org/Documentation/user-submodules.html feature to link the package repository change to commits on all of the other repositories that were tested with it.22:20
clarkbKaiyanSheng: because monascaclient wasnt in the constraints list at the time I bet22:20
SpamapSessentially, when I go to consume this, I just say "what's the version in the package's git repository?" and that version is the one I use.22:20
clarkbjeblair: re file copying I have nothing22:21
jeblairSpamapS: we can kinda/sorta approximate that for other jobs within a buildset -- a job could put something on tarballs keyed off of the zuul ref, and other jobs in the same buildset can find it based on the zuul ref.  but a second item stacked on the first wouldn't see it.  so that's part of what we need to add to zuulv3, is a way to pass down that state from one change to its children22:21
*** shashank_hegde has quit IRC22:21
*** markvoelker has quit IRC22:21
KaiyanSheng@clarkb Yes you are right!!!22:21
*** markvoelker has joined #openstack-infra22:22
lifelessclarkb: Nakato and I are discussing how to tackle it22:22
SpamapSjeblair: Could it be as simple as having a monotonically advancing counter for any pipeline?22:22
SpamapSI guess that only works for post22:22
*** annegentle has quit IRC22:23
SpamapSjeblair: I do very much appreciate you helping me think this through.22:23
*** dansmith has quit IRC22:23
*** dansmith has joined #openstack-infra22:23
jeblairSpamapS: you're welcome.  i may not fully understand what you're saying yet though.  :)22:24
*** rossella_s has quit IRC22:24
SpamapSjeblair: I'm not really sure I understand the problem with child commits, but I don't want to make you explain it to me. :)22:24
*** dougshelley66 has quit IRC22:24
SpamapSjeblair: we probably need a white board and hands to wave.22:24
*** rossella_s has joined #openstack-infra22:24
*** sridhar_ram has joined #openstack-infra22:24
*** Swami has joined #openstack-infra22:24
*** dougshelley66 has joined #openstack-infra22:24
jeblairSpamapS: it would probably be best if both of us understood each other, but yeah, we may not be able to accomplish that in a reasonable amount of time right here :)22:24
*** Swami has quit IRC22:25
*** Swami has joined #openstack-infra22:26
*** rhochmuth has joined #openstack-infra22:26
*** baoli has quit IRC22:26
*** baoli has joined #openstack-infra22:26
*** dimtruck is now known as zz_dimtruck22:27
*** Swami_ has quit IRC22:28
*** kzaitsev_mb has joined #openstack-infra22:29
*** rhochmuth has quit IRC22:30
*** nwkarsten has quit IRC22:30
*** aorourke__ is now known as aorourke22:30
openstackgerritDavid Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job  https://review.openstack.org/31479522:32
*** doug-fish has joined #openstack-infra22:32
*** Jeffrey4l has joined #openstack-infra22:35
*** doug-fish has quit IRC22:36
*** ccamacho has joined #openstack-infra22:37
*** doug-fish has joined #openstack-infra22:39
*** _sarob has quit IRC22:40
*** ayoung has joined #openstack-infra22:42
Shrewsjeblair: i'm not aware of other options, but i'm personally a fan of letting static.o.o do the work.22:43
Shrewscompared to the other options you listed, that is22:43
*** doug-fish has quit IRC22:44
*** rprakash_ has joined #openstack-infra22:45
rprakash_#info fungi there?22:45
*** sdake_ has joined #openstack-infra22:45
*** zz_dimtruck is now known as dimtruck22:45
*** _prad_ has quit IRC22:47
zarofungi: i was wrong about import rename keeping subscriptions.  it does not map subscriptions to the new project nor does persist starred setting on those changes.22:48
*** sdake has quit IRC22:48
*** kzaitsev_mb has quit IRC22:49
*** psilvad has quit IRC22:49
*** baoli has quit IRC22:51
*** baoli has joined #openstack-infra22:51
*** cody-somerville has joined #openstack-infra22:52
KaiyanSheng@clarkb I understand why the check job failed but I still follow lifeless's comment about "convert to a file reference" or "remove from constraint file". What constraints file we are talking about? The tox.ini? I added a "install_command = pip install -U --force-reinstall {opts} {packages}" under testenv:venv in review https://review.openstack.org/#/c/317758/22:52
KaiyanSheng@clarkb not sure that's the solution or not...22:52
openstackgerritKhai Do proposed openstack-infra/nodepool: Remove duplicate python-jenkins code from nodepool  https://review.openstack.org/25915722:54
*** sdake_ has quit IRC22:54
*** nwkarsten has joined #openstack-infra22:55
*** shashank_hegde has joined #openstack-infra22:56
*** camunoz has quit IRC22:56
*** camunoz has joined #openstack-infra22:57
*** hongbin has quit IRC22:58
*** doug-fish has joined #openstack-infra22:59
*** nwkarsten has quit IRC22:59
*** rbrndt has quit IRC23:00
*** r-daneel has joined #openstack-infra23:01
*** doug-fish has quit IRC23:03
openstackgerritMerged openstack-infra/system-config: Add a storyboard-dev server  https://review.openstack.org/31212423:05
*** baoli has quit IRC23:06
*** Goneri has joined #openstack-infra23:07
*** baoli has joined #openstack-infra23:07
*** lakshmiS has joined #openstack-infra23:08
*** woodster_ has quit IRC23:08
pabelangerlooks like we're trying to recover from some issue in ovh-bhs1 in nodepool23:09
*** kzaitsev_mb has joined #openstack-infra23:09
openstackgerritMerged openstack-infra/system-config: Realize SotK and Zara on the storyboard-dev server  https://review.openstack.org/31570623:10
*** |-paul-| has joined #openstack-infra23:10
*** vhosakot has quit IRC23:10
*** gordc has quit IRC23:11
*** markvoelker has quit IRC23:12
*** baoli has quit IRC23:16
*** baoli has joined #openstack-infra23:17
openstackgerritMerged openstack-infra/project-config: Switch to qcow2 for OVH clouds  https://review.openstack.org/31568023:17
*** kzaitsev_mb has quit IRC23:17
Jeffrey4lhey pabelanger , could check this PS? https://review.openstack.org/315860  the hostname on centos is not changed, which case the rabbitmq crash.23:17
Jeffrey4lhere is the log http://logs.openstack.org/60/315860/25/check/gate-kolla-dsvm-deploy-centos-binary/0cec277/console.html#_2016-05-17_17_26_31_50523:17
ianwSpamapS / greghaynes : mind just sanity checking https://review.openstack.org/#/c/317216/2 ?  trying to make the dib logs a bit more usable, since i seem to be pulling them apart a lot23:18
Jeffrey4lcould u have any idea about this?23:18
*** cschwede has quit IRC23:18
*** kzaitsev_mb has joined #openstack-infra23:19
*** pahuang has quit IRC23:19
ianwSpamapS / greghaynes : also, do you think dib-utils is probably past it's time of being useful?23:19
pabelangerJeffrey4l: I am not that familiar with kolla23:20
pabelangerJeffrey4l: but keep in mind, the hostname is unique for each job we run23:21
*** salv-orlando has quit IRC23:21
*** cschwede has joined #openstack-infra23:22
Jeffrey4lpabelanger, i think this is introduced by the new dib image.23:22
jeblairpabelanger: Shrews is voting for 'install key on static.o.o and pull from there'. you indicated that's not your favorite, but is acceptable.  i'm starting to lean that way.23:22
Jeffrey4lok. i will simplify the script and explain what's the issue is.23:23
pabelangerJeffrey4l: I confirmed in the experimental pipeline kolla passed your votes before switching to DIB.  When I was atleast looking at the jobs, I didn't see any errors.  Not to say it isn't an issue now23:23
Shrewsjeblair: i'm not understanding the downside of that, tbh. is it a security concern?23:23
Jeffrey4lok. thanks.23:24
*** sdake has joined #openstack-infra23:24
*** baoli has quit IRC23:24
pabelangerjeblair: Shrews: Ya, my only real concern is, we now need to bring static.o.o into the fold with zuul-launcher.  EG: installing zuul bits onto static.o.o, I am fine with doing it now, but wouldn't like to see us do that long term23:24
*** whoops has quit IRC23:24
jeblairpabelanger: well, it's just the ssh key, not zuul itself, but point taken23:25
jeblairit is an extra requirement23:25
Shrewson the plus side, the controller(s) do not need massive disk storage23:25
*** pahuang has joined #openstack-infra23:26
jeblairlike, the only requirement to have jenkins copy something to a remote host is that it have the public key.  this means that to do the same thing with zuul-launcher/ansible, it *additionally* needs a private key.23:26
jeblairthat rubs me the wrong way, even though it's sensible :)23:26
jeblairShrews: yeah -- if we use the controller as intermediate, we wouldn't need the storage for long, but if a whole bunch of jobs ended at once and each had many GB of logs, it could (briefly) be a problem23:27
pabelangerI should note, doing number 2, is how we do it today with puppet logs on our servers. So, we should have a playbook around with this concept working23:28
jeblairpabelanger: well, #2 doesn't work, because of the reasons i mentioned there -- if you know how to fix it, i'd love to see it23:29
*** kzaitsev_mb has quit IRC23:29
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic-Grenade: Remove DEVSTACK_GATE_TEMPEST_REGEX='ironic'  https://review.openstack.org/31776623:30
pabelangerjeblair: sure, I don't mind taking a stab at it, but can't do it this evening.  I say we proceed with the plan you have laid out currently23:30
jeblairokay, i'll write a patch to put the key on static for now23:30
*** rprakash_ has quit IRC23:31
jeblair(honestly, *either* logs-in-swift or logs-in-afs make this easier, but we have to move off of jenkins first)23:31
*** rhochmuth has joined #openstack-infra23:31
*** rprakash has quit IRC23:31
*** dansmith has quit IRC23:31
*** dansmith has joined #openstack-infra23:32
Shrewsjeblair: technically, i'd think that #2 would be possible. your playbook seems a bit wonky to me, though.23:34
*** dingyichen has joined #openstack-infra23:34
pabelangerjeblair: is there a log for the failure on #2 some place?23:34
jeblairShrews: well, the playbook doesn't work.  but yeah, the trick is that i don't know that we can say in the second task "copy from controller to static" when we're running a play in the context of "node".23:35
*** Swami_ has joined #openstack-infra23:35
*** KaiyanSheng has quit IRC23:36
*** mixos has joined #openstack-infra23:36
jeblairpabelanger: pasted output into etherpad23:36
*** kzaitsev_mb has joined #openstack-infra23:36
pabelangerjeblair: thanks23:37
Shrewsjeblair: the trick would be use of local_action23:37
*** armax_ has joined #openstack-infra23:37
jeblairShrews: yeah, i think the way to fix #2 is #423:37
*** Goneri has quit IRC23:39
*** Qiming has joined #openstack-infra23:39
*** Swami has quit IRC23:39
*** armax has quit IRC23:39
*** armax_ is now known as armax23:39
Shrewsjeblair: well, that or just separate into two tasks23:39
jeblairShrews: what two?23:40
pabelangerYa, I think I see the issue with #2. it will basically look like: https://github.com/openstack-infra/ansible-puppet/blob/master/tasks/main.yml#L8123:40
pabelangerreplacing puppet_post_puppetdb with rsync23:40
Shrewsjeblair: one to syncronize to controller, another (using local_action) to send to static.o.o23:40
openstackgerritMerged openstack-infra/nodepool: Use clean-floating-ips for yaml setting  https://review.openstack.org/31773423:41
pabelangerShrews: agreed23:41
jeblairShrews: there are two tasks there23:41
Shrewsjeblair: i could try to cook up an example if you want, but i still like option #3 better23:41
*** Swami_ has quit IRC23:41
*** rhochmuth has quit IRC23:42
*** cody-somerville has quit IRC23:43
jeblairwith synchronize, it boils down to: in push mode, the controller or delegate is the source.  that *only* leaves the node as the possible destination.  so there's no way to specify static.o.o as the target23:43
Shrewsjeblair: yes, there are, but the 2nd one isn't executing locally23:44
jeblairbut i'd love to see why that's wrong23:44
Shrewsjeblair: the 2nd would have to be something like:  local_action: command /bin/scp <src> static.o.o:<dest>23:44
jeblairShrews: right that's what i wrote in #423:45
jeblairso i think we're in agreement23:45
Shrewsjeblair: oh, so you did.23:45
Shrewsjeblair: consensus!23:46
jeblairthe thing i kinda like about that is that it's the fastest way to get this going, and i don't have to explain to anyone why our static webserver needs a private key on it23:46
*** kzaitsev_mb has quit IRC23:46
*** IlyaG has quit IRC23:50
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add PS4 to show file/function/line in debug output  https://review.openstack.org/31726023:50
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add jenkins/zuul private key to static.o.o  https://review.openstack.org/31777323:51
jeblairpabelanger, Shrews: let's see what other infra-root folks think about that^.  in the mean time, let's give the local_action a try23:51
*** kzaitsev_mb has joined #openstack-infra23:54
jeblair [WARNING]: Consider using synchronize module rather than running rsync23:54
jeblairhehe :)23:54
Shrewsjeblair: left my thoughts on that review for others to pick over23:55
*** chem`` has quit IRC23:55
jeblairthat's an improvement -- the directory exists23:55
Shrewsi think i should probably have supper now23:55
jeblairit is empty23:55
jeblairi will go try to find out where on our 14TB log server those files ended up23:56
jeblairoh, heh, they are there, the directory is just mode 70023:57
*** pradk has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!