Friday, 2016-07-01

*** thorst_ has quit IRC00:00
*** nwkarsten has joined #openstack-infra00:01
*** sihan has joined #openstack-infra00:01
mriedemetherpad.o.o down?00:03
sihanHi guys, what is gerrit format comment for UI? Our ci result does not show color, and it did not show on the top, only show in the commit message section.00:03
*** kzaitsev_mb has joined #openstack-infra00:04
*** nwkarsten has quit IRC00:05
fungimriedem: midnight utc database backups. still down for you?00:07
*** bhavik has joined #openstack-infra00:08
*** sdake has joined #openstack-infra00:09
openstackgerritMerged openstack/diskimage-builder: Fix the bug that "mktemp: failed to create directory"
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci
clarkbzl01, 03 and 06 done manually. the oyhers are running against playbook without async now00:10
clarkbsihan: you can find it in zuul or system-config/modules/openstack-project/manifests/review.pp iirc00:11
*** ffrank has quit IRC00:13
mriedemfungi: now it's back00:14
*** darvon has quit IRC00:15
*** darvon has joined #openstack-infra00:15
*** PollyZ has joined #openstack-infra00:15
*** sdake_ has joined #openstack-infra00:16
*** thumpba has joined #openstack-infra00:17
thumpbai need to update my ICLA with my mailing address so i can commit, how can i do this?00:17
*** IlyaG has quit IRC00:19
*** kaisers1 has joined #openstack-infra00:19
*** sdake has quit IRC00:19
*** PollyZ has quit IRC00:21
thumpbanevermind seems to do the trick00:21
*** piet has joined #openstack-infra00:21
*** zeih has joined #openstack-infra00:21
*** kaisers has quit IRC00:21
*** bhunter71 has joined #openstack-infra00:22
*** markvoelker has joined #openstack-infra00:22
*** SumitNaiksatam has quit IRC00:23
*** ddieterly has joined #openstack-infra00:24
*** zeih has quit IRC00:26
*** pt_15 has quit IRC00:26
*** xarses has joined #openstack-infra00:26
*** markvoelker has quit IRC00:27
*** PollyZ has joined #openstack-infra00:29
*** Apoorva has joined #openstack-infra00:30
*** itisha has quit IRC00:30
*** thumpba has quit IRC00:31
*** nwkarsten has joined #openstack-infra00:31
*** lvdongbing has joined #openstack-infra00:33
*** PollyZ has quit IRC00:34
*** piet has quit IRC00:36
*** karthik__ has quit IRC00:38
*** adrian_otto has quit IRC00:39
*** shashank_hegde has quit IRC00:39
*** ddieterly has quit IRC00:41
*** Jeffrey4l has joined #openstack-infra00:42
*** PollyZ has joined #openstack-infra00:43
*** Sukhdev has quit IRC00:47
*** PollyZ has quit IRC00:47
*** yamahata has joined #openstack-infra00:48
*** woodster_ has quit IRC00:49
*** amotoki has joined #openstack-infra00:54
*** rcernin has quit IRC00:55
*** PollyZ has joined #openstack-infra00:56
*** thorst_ has joined #openstack-infra00:59
*** amitgandhinz has quit IRC01:01
*** PollyZ has quit IRC01:01
*** amitgandhinz has joined #openstack-infra01:01
*** chlong has quit IRC01:06
*** sihan has quit IRC01:06
*** thorst_ has quit IRC01:07
*** Apoorva has quit IRC01:07
*** tqtran_ has quit IRC01:07
*** pvaneck has quit IRC01:08
*** xyang1 has quit IRC01:10
*** PollyZ has joined #openstack-infra01:10
*** Apoorva has joined #openstack-infra01:11
*** zhurong has joined #openstack-infra01:12
*** sdake_ has quit IRC01:13
*** PollyZ has quit IRC01:15
*** Apoorva has quit IRC01:15
*** zshuo has joined #openstack-infra01:16
*** karthik__ has joined #openstack-infra01:18
*** zeih has joined #openstack-infra01:22
*** thorst_ has joined #openstack-infra01:23
*** PollyZ has joined #openstack-infra01:24
*** thorst_ has quit IRC01:24
*** zeih has quit IRC01:26
*** julim has quit IRC01:26
*** PollyZ has quit IRC01:28
*** kzaitsev_mb has quit IRC01:30
*** yanyanhu has joined #openstack-infra01:30
*** gouthamr has quit IRC01:34
*** claudiub has quit IRC01:34
*** rwsu has joined #openstack-infra01:36
*** iurygregory has joined #openstack-infra01:36
*** bradm_ has joined #openstack-infra01:36
*** dulek has joined #openstack-infra01:36
*** mkoderer__ has joined #openstack-infra01:36
*** ianw has joined #openstack-infra01:36
*** strigazi has joined #openstack-infra01:36
*** trown|outtypewww has joined #openstack-infra01:36
*** hamzy has joined #openstack-infra01:36
*** mkoderer has joined #openstack-infra01:36
*** gothicmindfood has joined #openstack-infra01:36
*** PollyZ has joined #openstack-infra01:37
*** karthik__ has quit IRC01:38
*** ddieterly has joined #openstack-infra01:38
openstackgerritBin Hu proposed openstack-infra/irc-meetings: Add Meeting Information for New Project Gluon
*** zhurong has quit IRC01:40
*** sdake has joined #openstack-infra01:40
*** zhurong has joined #openstack-infra01:41
*** PollyZ has quit IRC01:43
*** ddieterly has quit IRC01:43
*** ams__ has quit IRC01:44
*** ianychoi has joined #openstack-infra01:49
*** PollyZ has joined #openstack-infra01:51
*** PollyZ has quit IRC01:56
*** ddieterly has joined #openstack-infra01:57
*** PollyZ has joined #openstack-infra01:59
*** savihou has quit IRC02:04
*** savihou has joined #openstack-infra02:05
*** Apoorva has joined #openstack-infra02:10
*** signed8bit_Zzz is now known as signed8bit02:10
*** sdake has quit IRC02:12
*** nwkarsten has quit IRC02:13
*** Apoorva has quit IRC02:16
*** thorst_ has joined #openstack-infra02:19
*** thorst_ has quit IRC02:20
*** Daisy has joined #openstack-infra02:20
*** thorst_ has joined #openstack-infra02:20
*** zeih has joined #openstack-infra02:23
*** markvoelker has joined #openstack-infra02:24
*** zeih has quit IRC02:27
*** markvoelker has quit IRC02:28
*** thorst_ has quit IRC02:29
*** ddieterly has quit IRC02:29
*** chlong has joined #openstack-infra02:34
*** shashank_hegde has joined #openstack-infra02:34
*** PollyZ has quit IRC02:35
*** mriedem has quit IRC02:38
*** rossella_s has quit IRC02:39
*** rossella_s has joined #openstack-infra02:39
*** karthik__ has joined #openstack-infra02:41
*** shashank_hegde has quit IRC02:42
*** kushal has joined #openstack-infra02:47
*** piet has joined #openstack-infra02:51
*** PollyZ has joined #openstack-infra02:52
*** ramishra has joined #openstack-infra02:52
*** yuanying_ has quit IRC02:52
*** karthik__ has quit IRC02:54
*** Daisy has quit IRC02:56
*** Daisy has joined #openstack-infra02:56
*** PollyZ has quit IRC02:56
openstackgerritSteve Martinelli proposed openstack-infra/project-config: set check-osc-job to voting
*** rook- has quit IRC02:59
*** coolsvap has joined #openstack-infra02:59
*** gyee has quit IRC03:00
*** rook- has joined #openstack-infra03:00
*** karthik__ has joined #openstack-infra03:00
*** kushal has quit IRC03:01
*** jamielennox is now known as jamielennox|away03:03
*** asettle has joined #openstack-infra03:08
*** kushal has joined #openstack-infra03:09
*** Daisy has quit IRC03:11
*** Daisy has joined #openstack-infra03:12
openstackgerritIan Wienand proposed openstack/diskimage-builder: Clear up "already provided" message
openstackgerritIan Wienand proposed openstack/diskimage-builder: Make element-info output env variables
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML
openstackgerritIan Wienand proposed openstack/diskimage-builder: Making element overriding explicit
openstackgerritIan Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables
*** nwkarsten has joined #openstack-infra03:14
*** asettle has quit IRC03:15
*** Daisy has quit IRC03:16
*** nwkarsten has quit IRC03:18
masayukigjhesketh or someone in infra, could you get logs of ? Now, is returning a 500.03:18
masayukigI'd like to see what's happening.03:19
*** PollyZ has joined #openstack-infra03:19
*** harlowja_at_home has joined #openstack-infra03:20
*** zeih has joined #openstack-infra03:23
*** gsilvis has quit IRC03:24
*** markvoelker has joined #openstack-infra03:25
*** gsilvis has joined #openstack-infra03:25
*** links has joined #openstack-infra03:26
*** zeih has quit IRC03:28
*** Qiming has quit IRC03:28
*** yamamoto_ has joined #openstack-infra03:28
*** zshuo has quit IRC03:28
*** markvoelker has quit IRC03:29
*** PollyZ has quit IRC03:31
*** zshuo has joined #openstack-infra03:31
*** jamielennox|away is now known as jamielennox03:31
*** Qiming has joined #openstack-infra03:32
*** karthik__ has quit IRC03:33
*** binbincong has quit IRC03:33
*** karthik__ has joined #openstack-infra03:33
*** chlong has quit IRC03:34
*** harlowja_at_home has quit IRC03:35
*** thorst_ has joined #openstack-infra03:35
*** Daisy has joined #openstack-infra03:38
*** Liuqing has joined #openstack-infra03:39
*** Daisy has quit IRC03:40
*** Daisy has joined #openstack-infra03:41
*** thorst_ has quit IRC03:44
*** Daisy has quit IRC03:45
*** piet has quit IRC03:46
*** binbincong has joined #openstack-infra03:49
*** karthi___ has joined #openstack-infra03:50
*** yuanying has joined #openstack-infra03:50
*** karthik__ has quit IRC03:52
openstackgerritAdam Spiers proposed openstack-infra/project-config: Fix gerrit ACLs for openstack-resource-agents-specs
*** sputnik13 has joined #openstack-infra03:52
*** PollyZ has joined #openstack-infra03:54
*** PollyZ has quit IRC03:59
*** Daisy has joined #openstack-infra03:59
*** Daisy has quit IRC03:59
*** Daisy has joined #openstack-infra04:00
*** sputnik13_ has joined #openstack-infra04:02
*** zhurong has quit IRC04:03
*** Daisy has quit IRC04:04
*** Daisy has joined #openstack-infra04:04
jheskethmasayukig: Here's what happens when I hit that page:
*** tqtran has joined #openstack-infra04:05
*** sputnik13_ has quit IRC04:06
masayukigjhesketh : thanks!04:06
*** pgadiya has joined #openstack-infra04:07
masayukigI have no idea to fix this, though... :(04:07
*** PollyZ has joined #openstack-infra04:08
*** Daisy has quit IRC04:09
*** Daisy has joined #openstack-infra04:10
*** tqtran has quit IRC04:10
*** Daisy has quit IRC04:12
jheskethmasayukig: yeah sorry I'm not around to help today either :-(04:12
*** Daisy has joined #openstack-infra04:12
masayukigno worries, thanks :)04:13
*** sputnik13_ has joined #openstack-infra04:13
*** PollyZ has quit IRC04:13
*** _ody has joined #openstack-infra04:13
*** Apoorva has joined #openstack-infra04:13
*** zhurong has joined #openstack-infra04:13
*** Daisy has quit IRC04:14
*** Daisy has joined #openstack-infra04:15
*** sputnik13_ has quit IRC04:15
openstackgerritIan Wienand proposed openstack/diskimage-builder: Make element-info output env variables
openstackgerritIan Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables
*** nwkarsten has joined #openstack-infra04:16
*** signed8bit is now known as signed8bit_Zzz04:18
*** Daisy has quit IRC04:19
*** yamamoto_ has quit IRC04:19
*** Apoorva has quit IRC04:20
*** nwkarsten has quit IRC04:21
*** Daisy has joined #openstack-infra04:21
*** yamamoto_ has joined #openstack-infra04:21
*** IlyaG has joined #openstack-infra04:22
masayukigmtreinish : hrm, it seems like memcached doesn't accept a key contains spaces.. : MemcacheIllegalInputError: Key contains spaces: '_lockopenstack_health.api:_query_test_runs_by_build_name|gate-monasca-api-python27 2016-05-31 03:00:00+00:00 2016-07-01 03:00:00+00:00'04:23
*** zeih has joined #openstack-infra04:24
*** Daisy has quit IRC04:25
*** Daisy has joined #openstack-infra04:25
*** markvoelker has joined #openstack-infra04:25
*** Daisy has quit IRC04:27
*** Daisy has joined #openstack-infra04:28
*** coolsvap has quit IRC04:28
*** zeih has quit IRC04:29
*** pgadiya has quit IRC04:29
*** markvoelker has quit IRC04:30
*** sputnik13 has quit IRC04:30
*** rlandy has quit IRC04:31
*** shashank_hegde has joined #openstack-infra04:32
*** Daisy has quit IRC04:33
*** asselin__ has joined #openstack-infra04:33
*** karthi___ has quit IRC04:34
*** karthik__ has joined #openstack-infra04:34
*** PollyZ has joined #openstack-infra04:35
*** coolsvap has joined #openstack-infra04:35
*** IlyaG has quit IRC04:36
*** sree has joined #openstack-infra04:36
*** asselin__ has quit IRC04:38
*** yamamoto_ has quit IRC04:38
*** IlyaG has joined #openstack-infra04:39
*** psachin has joined #openstack-infra04:39
*** PollyZ has quit IRC04:40
*** bhavik has quit IRC04:40
*** sputnik13 has joined #openstack-infra04:40
*** nwkarsten has joined #openstack-infra04:40
*** xarses has quit IRC04:41
*** xarses has joined #openstack-infra04:41
*** sree_ has joined #openstack-infra04:42
*** sree_ is now known as Guest4639904:43
*** thorst_ has joined #openstack-infra04:43
*** nwkarsten has quit IRC04:45
*** sree has quit IRC04:45
*** PollyZ has joined #openstack-infra04:48
*** thorst_ has quit IRC04:49
*** karthi___ has joined #openstack-infra04:52
*** IlyaG has quit IRC04:52
*** karthik__ has quit IRC04:53
*** PollyZ has quit IRC04:53
*** signed8bit_Zzz is now known as signed8bit04:53
*** signed8bit has quit IRC04:54
*** zhurong has quit IRC04:58
*** piet has joined #openstack-infra04:58
*** e0ne has joined #openstack-infra04:59
*** zhurong has joined #openstack-infra05:00
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Build wholedisk images capable of UEFI secure boot
*** Sukhdev has joined #openstack-infra05:03
*** SumitNaiksatam has joined #openstack-infra05:07
*** e0ne has quit IRC05:07
*** ayoung has quit IRC05:09
*** chlong has joined #openstack-infra05:15
*** PollyZ has joined #openstack-infra05:16
*** pgadiya has joined #openstack-infra05:16
*** Guest46399 has quit IRC05:17
*** ianw is now known as ianw_pto05:19
*** sree has joined #openstack-infra05:20
*** PollyZ has quit IRC05:20
*** yamamoto_ has joined #openstack-infra05:24
*** zeih has joined #openstack-infra05:25
*** markvoelker has joined #openstack-infra05:26
*** yamahata has quit IRC05:27
*** PollyZ has joined #openstack-infra05:29
*** liusheng has quit IRC05:29
*** zeih has quit IRC05:30
*** liusheng has joined #openstack-infra05:30
*** markvoelker has quit IRC05:31
openstackgerritBabu Shanmugam proposed openstack-infra/project-config: Adding a new JOB for networking-ovn project
*** dingyichen has joined #openstack-infra05:32
*** PollyZ has quit IRC05:35
*** yamahata has joined #openstack-infra05:37
*** Liuqing has quit IRC05:38
*** ramishra has quit IRC05:39
*** Liuqing has joined #openstack-infra05:39
*** ramishra has joined #openstack-infra05:40
*** Daisy has joined #openstack-infra05:41
openstackgerritIan Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point
*** PollyZ has joined #openstack-infra05:43
*** zhurong has quit IRC05:43
*** thorst_ has joined #openstack-infra05:46
*** PollyZ has quit IRC05:47
*** zhurong has joined #openstack-infra05:48
*** YorikSar has quit IRC05:51
*** javeriak has joined #openstack-infra05:52
*** YorikSar has joined #openstack-infra05:53
*** thorst_ has quit IRC05:53
*** javeriak has quit IRC05:53
*** amitgandhinz has quit IRC05:56
*** amitgandhinz has joined #openstack-infra05:56
*** PollyZ has joined #openstack-infra05:56
*** tqtran has joined #openstack-infra05:58
*** Ravikiran_K has joined #openstack-infra05:59
*** yuanying has quit IRC06:01
*** yuanying has joined #openstack-infra06:02
*** PollyZ has quit IRC06:02
*** sputnik13 has quit IRC06:02
*** Daisy has quit IRC06:02
*** sputnik13 has joined #openstack-infra06:02
*** Daisy has joined #openstack-infra06:02
*** tqtran has quit IRC06:03
*** fawadkhaliq has joined #openstack-infra06:03
*** claudiub has joined #openstack-infra06:03
*** aeng has quit IRC06:04
*** isq has joined #openstack-infra06:05
*** yuanying has quit IRC06:05
*** yuanying has joined #openstack-infra06:08
*** fawadkhaliq has quit IRC06:09
*** fawadkhaliq has joined #openstack-infra06:10
*** PollyZ has joined #openstack-infra06:10
*** ccamacho has quit IRC06:13
openstackgerritMathieu Mitchell proposed openstack-infra/project-config: Bump VM memory for latest stable CoreOS IPA
*** PollyZ has quit IRC06:15
*** sputnik13 has quit IRC06:17
*** gouthamr has joined #openstack-infra06:17
*** tlian has quit IRC06:18
*** pgadiya has quit IRC06:20
*** chem|off has quit IRC06:25
*** chem|off has joined #openstack-infra06:25
*** zeih has joined #openstack-infra06:25
*** chem|off is now known as chem06:26
*** pgadiya has joined #openstack-infra06:26
*** armax has quit IRC06:26
*** markvoelker has joined #openstack-infra06:27
*** andreas_s has joined #openstack-infra06:28
*** Sukhdev has quit IRC06:29
*** rcernin has joined #openstack-infra06:29
*** zeih has quit IRC06:30
openstackgerritMathieu Mitchell proposed openstack-infra/project-config: Introduce experimental job to test TinyIPA with less RAM
yamamoto_"Unable to load this page! An API error occurred." on openstack health is known?06:31
*** markvoelker has quit IRC06:32
openstackgerritMathieu Mitchell proposed openstack-infra/project-config: Introduce experimental job to test TinyIPA with less RAM
masayukigyamamoto_ : yeah, I already pushed a patch for it :
yamamoto_masayukig: thank you06:34
openstackgerritMerged openstack-infra/project-config: Add a heat job to use devstack plugin
*** zeih has joined #openstack-infra06:36
*** PollyZ has joined #openstack-infra06:37
*** mat128|afk is now known as mat128|holiday06:38
*** ccamacho has joined #openstack-infra06:41
*** zhurong has quit IRC06:41
*** nwkarsten has joined #openstack-infra06:42
*** PollyZ has quit IRC06:42
*** zhurong has joined #openstack-infra06:44
*** isq has quit IRC06:44
*** Nikolay_St has joined #openstack-infra06:46
*** nwkarsten has quit IRC06:46
*** piet has quit IRC06:46
*** ociuhandu has joined #openstack-infra06:47
*** isq has joined #openstack-infra06:47
*** camunoz has quit IRC06:49
*** PollyZ has joined #openstack-infra06:51
*** thorst_ has joined #openstack-infra06:52
*** jed56 has joined #openstack-infra06:54
*** PollyZ has quit IRC06:56
*** pcaruana has joined #openstack-infra06:57
yolandagood morning06:58
*** thorst_ has quit IRC06:59
*** ociuhandu_ has joined #openstack-infra07:00
*** ociuhandu has quit IRC07:00
*** ociuhandu_ is now known as ociuhandu07:00
openstackgerrityolanda.robla proposed openstack-infra/system-config: As initial step to deploy with HA, start using pacemaker module
*** javeriak has joined #openstack-infra07:02
*** tesseract- has joined #openstack-infra07:02
*** karthi___ has quit IRC07:03
*** PollyZ has joined #openstack-infra07:04
*** yuanying has quit IRC07:05
*** yuanying has joined #openstack-infra07:08
*** PollyZ has quit IRC07:09
*** Daisy_ has joined #openstack-infra07:10
*** asettle has joined #openstack-infra07:13
*** Daisy has quit IRC07:14
openstackgerritMerged openstack-infra/project-config: Add new project Zunclient to openstack repo
*** javeriak has quit IRC07:15
*** esikachev has joined #openstack-infra07:15
*** gouthamr has quit IRC07:16
*** asettle has quit IRC07:17
AJaegermorning, yolanda !07:17
*** yamahata has quit IRC07:18
openstackgerritMerged openstack-infra/project-config: Revert "Add experimental wheel-build-ubuntu-xenial-amd64 job"
openstackgerritMerged openstack-infra/project-config: Set wheel-build-ubuntu-xenial-amd64 voting
openstackgerritMerged openstack-infra/project-config: Fix gerrit ACLs for openstack-resource-agents-specs
openstackgerritMerged openstack-infra/project-config: Read only mode for fuel-upgrade project
*** chlong has quit IRC07:23
*** dtardivel has joined #openstack-infra07:24
openstackgerritMerged openstack-infra/project-config: update osc-check-plugin script
*** jlanoux has joined #openstack-infra07:24
*** gnuoy` is now known as gnuoy07:25
*** javeriak has joined #openstack-infra07:25
*** ihrachys has joined #openstack-infra07:26
openstackgerritMerged openstack-infra/project-config: Puppet-refstack doesn't have any beaker jobs
*** arxcruz has joined #openstack-infra07:27
openstackgerritMerged openstack-infra/project-config: Add Networking OVN Data to Grafana
*** markvoelker has joined #openstack-infra07:28
*** kushal has quit IRC07:28
*** kushal has joined #openstack-infra07:29
*** ihrachys has quit IRC07:31
*** markvoelker has quit IRC07:32
*** openstackgerrit has quit IRC07:33
*** ihrachys has joined #openstack-infra07:33
*** openstackgerrit has joined #openstack-infra07:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rework publishing jobs
AJaegerclarkb, this reorg changes the release jobs - it removes 800 jobs ^07:34
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rework publishing jobs
*** Daisy_ has quit IRC07:35
*** Daisy has joined #openstack-infra07:36
openstackgerritMerged openstack-infra/project-config: Add rally job for Watcher service and support for rally-gate
AJaegerclarkb: my change will merge-conflict heavily in projects.yaml. If my change merges first, I'll rebase yours.07:37
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI
AJaegeryolanda, do you have time to review 336405, please? It's largish...07:39
yolandaAJaeger, sure, i can take a look in a while07:40
*** Daisy has quit IRC07:41
*** rvba has joined #openstack-infra07:41
*** rvba has quit IRC07:41
*** rvba has joined #openstack-infra07:41
AJaegerthanks, yolanda - and thanks for your review batch this morning07:41
yolandano problem, just tell me if you need something more, i'll review as time permits07:42
*** ajmiller has joined #openstack-infra07:43
openstackgerritMerged openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only
*** fawadkhaliq has quit IRC07:44
AJaegeryolanda: I'm fine now, thanks07:44
*** ajmiller_ has quit IRC07:44
*** jpich has joined #openstack-infra07:46
openstackgerritMerged openstack-infra/project-config: kuryr: add libnetwork and k8s to gerritbot
*** amoralej|off is now known as amoralej07:48
*** Kiall_ has quit IRC07:50
*** ihrachys_ has joined #openstack-infra07:51
*** ccamacho is now known as ccamacho|afk07:51
*** ihrachys_ has quit IRC07:52
*** ihrachys has quit IRC07:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Further kilo cleanup
*** rcernin has quit IRC07:57
*** thorst_ has joined #openstack-infra07:57
*** ifarkas has joined #openstack-infra07:58
*** PollyZ has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** shardy has joined #openstack-infra08:01
*** PollyZ has quit IRC08:04
*** thorst_ has quit IRC08:04
*** Na3iL has joined #openstack-infra08:05
AJaegertonyb, do you want to review the change above? ^08:06
*** yamahata has joined #openstack-infra08:06
*** dtantsur|afk is now known as dtantsur08:07
tonybAJaeger: Thanks I'll have a look after dinner08:07
*** asettle has joined #openstack-infra08:09
*** hashar has joined #openstack-infra08:10
*** Daisy has joined #openstack-infra08:10
*** rcernin has joined #openstack-infra08:11
AJaegertonyb: enjoy dinner!08:12
*** eezhova has joined #openstack-infra08:13
*** PollyZ has joined #openstack-infra08:13
*** Daisy has quit IRC08:15
*** Daisy has joined #openstack-infra08:16
*** acoles_ is now known as acoles08:16
*** mrmartin has joined #openstack-infra08:17
fricklerclarkb: merged the pypy changes and fixed the remaining issues hopefully, please crosscheck (, not sure why gerritbot didn't log my update)08:17
*** PollyZ has quit IRC08:17
*** openstackgerrit has quit IRC08:18
*** openstackgerrit has joined #openstack-infra08:18
*** derekh has joined #openstack-infra08:19
*** Daisy_ has joined #openstack-infra08:20
AJaegerthanks, frickler !08:20
*** sorantis has joined #openstack-infra08:20
*** jianghuaw has joined #openstack-infra08:22
*** Daisy has quit IRC08:23
*** kushal has quit IRC08:24
jianghuawIs there anyway to make nodepool keep the node if the test failed.08:24
*** Daisy_ has quit IRC08:24
*** Daisy has joined #openstack-infra08:31
*** ams__ has joined #openstack-infra08:31
*** ihrachys has joined #openstack-infra08:31
*** Jeffrey4l has quit IRC08:33
*** yamamoto_ has quit IRC08:33
*** yamamoto has joined #openstack-infra08:35
*** zshuo has quit IRC08:36
*** ifarkas_ has joined #openstack-infra08:36
*** yanyanhu has quit IRC08:37
*** _degorenko|afk is now known as degorenko08:37
*** ifarkas has quit IRC08:37
hasharjianghuaw: in short now  unless you manually create the job from the command line and pass --hold-on-failure08:38
hasharjianghuaw: relevant code would be in nodepool/ in NodeCompleteThread.handleEvent()08:39
*** nwkarsten has joined #openstack-infra08:44
fricklerclarkb: nova will need before the python35 jobs can run properly. I also found a python3.5 incompatibility issue in nova, will push a patch for that later08:46
*** abregman has joined #openstack-infra08:47
*** nwkarsten has quit IRC08:48
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** YorikSar has quit IRC08:51
*** YorikSar has joined #openstack-infra08:52
*** shashank_hegde has quit IRC08:52
*** Jeffrey4l has joined #openstack-infra08:55
*** ccamacho|afk has quit IRC08:55
*** javeriak has quit IRC08:57
*** javeriak has joined #openstack-infra08:58
*** ccamacho has joined #openstack-infra08:59
*** zshuo has joined #openstack-infra09:00
*** electrofelix has joined #openstack-infra09:00
*** thorst_ has joined #openstack-infra09:02
*** Daisy has quit IRC09:02
jianghuaw@hashar: thanks.09:03
hasharjianghuaw: a hack I am doing from time to time is to have the job triggered as usual, then rush to the nodepool CLI command and manually hold the node09:03
hasharjianghuaw: with something like nodepool hold 12345609:03
hasharbut maybe nodepool job-create --hold-on-failure     is the way to go. I am not running that version of nodepool yet though :(09:04
jianghuawhashar: Yes, nodepool hold # can hold a target node.09:06
jianghuawIt will be helpful if there is any parameter in nodepool to keep node for failures. Specially useful for random failure.09:06
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** PollyZ has joined #openstack-infra09:07
*** thorst_ has quit IRC09:09
*** salv-orl_ has quit IRC09:09
*** salv-orlando has joined #openstack-infra09:10
*** PollyZ has quit IRC09:12
*** andymaier has joined #openstack-infra09:15
*** sree has quit IRC09:16
*** piet has joined #openstack-infra09:17
*** eezhova has quit IRC09:19
*** ajmiller has quit IRC09:20
*** ajmiller has joined #openstack-infra09:20
AJaegerfrickler: python35 jobs are non-voting - so we can figure that one out one by one - which is good.09:22
AJaegerfrickler: thanks for helping to move this forward!09:23
fricklerAJaeger: I'll push a new patch addressing your comments once my local tox run finishes successfully09:24
AJaegerhashar, jianghuaw have a look at - that's an auto-hold feature for failing jobs09:24
hasharAJaeger: you and jeblair are magic!09:25
*** eezhova has joined #openstack-infra09:27
openstackgerritJens Rosenboom proposed openstack-infra/project-config: Add python35 jobs
AJaegerhashar: jeblair is for sure ;)09:29
*** john-davidge has joined #openstack-infra09:29
*** markvoelker has joined #openstack-infra09:29
*** fawadkhaliq has joined #openstack-infra09:30
*** fawadkhaliq has quit IRC09:30
*** mrmartin has quit IRC09:30
*** john-dav_ has joined #openstack-infra09:32
*** dtantsur is now known as dtantsur|bbl09:33
*** markvoelker has quit IRC09:34
*** john-davidge has quit IRC09:35
*** ajmiller has quit IRC09:36
*** ajmiller has joined #openstack-infra09:36
*** Daisy has joined #openstack-infra09:37
*** ajmiller has quit IRC09:38
*** yolanda has quit IRC09:38
*** ajmiller has joined #openstack-infra09:38
*** yolanda has joined #openstack-infra09:38
*** john-dav_ is now known as john-davidge09:41
*** Daisy has quit IRC09:41
jianghuawAjaeger: thanks for the sharing. That's useful.09:45
*** david-lyle has quit IRC09:45
*** fawadkhaliq has joined #openstack-infra09:46
*** yolanda has quit IRC09:47
*** zshuo has quit IRC09:48
*** yolanda has joined #openstack-infra09:49
*** abregman is now known as abregman|afk09:50
*** Daisy has joined #openstack-infra09:53
*** tosky has joined #openstack-infra09:55
*** ociuhandu has quit IRC09:55
*** maishsk has quit IRC09:56
*** ifarkas_ has quit IRC09:58
*** maishsk has joined #openstack-infra09:58
*** david-lyle has joined #openstack-infra09:59
*** tqtran has joined #openstack-infra10:01
*** jcoufal has quit IRC10:04
*** yolanda has quit IRC10:05
*** lvdongbing has quit IRC10:05
*** tqtran has quit IRC10:05
*** piet has quit IRC10:06
*** yolanda has joined #openstack-infra10:06
*** gmann has quit IRC10:06
*** piet has joined #openstack-infra10:08
openstackgerritMerged openstack-infra/irc-meetings: Add Meeting Information for New Project Gluon
*** thorst_ has joined #openstack-infra10:09
*** rcernin has quit IRC10:10
*** _nadya_ has joined #openstack-infra10:11
*** ggnel_t has quit IRC10:13
*** zqfan has quit IRC10:13
*** thorst_ has quit IRC10:14
*** kushal has joined #openstack-infra10:16
*** zhurong has quit IRC10:17
openstackgerritBernard Cafarelli proposed openstack/diskimage-builder: Export YUM variable in centos bases
*** linuxmaniac_ has joined #openstack-infra10:19
*** linuxmaniac has quit IRC10:19
*** fawadkhaliq has quit IRC10:22
*** fawadkhaliq has joined #openstack-infra10:23
*** linuxmaniac_ has quit IRC10:23
*** rcernin has joined #openstack-infra10:23
*** linuxmaniac has joined #openstack-infra10:23
AJaegerinfra-root, I checked bindep 2.0 changes and it looks fine with current image  - but rax-dfw still has old images. Will those be updated automatically? Or do we need manual upload again?10:23
*** dtantsur|bbl is now known as dtantsur10:24
*** sorantis has quit IRC10:27
openstackgerritJames Page proposed openstack-infra/project-config: Add repositories for Charm layers
*** fawadkhaliq has quit IRC10:28
*** amotoki has quit IRC10:28
*** PollyZ has joined #openstack-infra10:29
*** fawadkhaliq has joined #openstack-infra10:29
*** markvoelker has joined #openstack-infra10:30
openstackgerritGerman proposed openstack-infra/git-review: Story 2000662
*** Ravikiran_K has quit IRC10:34
*** PollyZ has quit IRC10:34
*** markvoelker has quit IRC10:35
*** piet has quit IRC10:35
*** jed56 has quit IRC10:35
*** fawadkhaliq has quit IRC10:35
*** ifarkas has joined #openstack-infra10:36
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Move ironic-inspector grenade job to the check pipeline
*** maishsk_ has joined #openstack-infra10:37
*** fawadkhaliq has joined #openstack-infra10:37
*** maishsk has quit IRC10:37
*** fawadkhaliq has quit IRC10:40
*** fawadkhaliq has joined #openstack-infra10:40
*** maishsk_ has quit IRC10:41
*** PollyZ has joined #openstack-infra10:42
*** nwkarsten has joined #openstack-infra10:46
*** fawadk has joined #openstack-infra10:46
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Add jobs based on ipmitool drivers to ironic-python-agent
*** fawadkhaliq has quit IRC10:47
*** fawadk has quit IRC10:47
*** PollyZ has quit IRC10:47
*** fawadkhaliq has joined #openstack-infra10:47
dtantsurlucasagomes, ^^^10:47
lucasagomesdtantsur, cool, will take alook10:48
*** vgridnev has quit IRC10:50
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** nwkarsten has quit IRC10:51
*** sorantis has joined #openstack-infra10:51
*** eezhova has quit IRC10:51
*** sdake has joined #openstack-infra10:51
*** maishsk has joined #openstack-infra10:52
*** fawadk has joined #openstack-infra10:52
*** fawadkhaliq has quit IRC10:52
openstackgerritLau Mercadal proposed openstack-infra/git-review: Story 2000662
*** ociuhandu has joined #openstack-infra10:53
*** sdake_ has joined #openstack-infra10:54
*** vgridnev has joined #openstack-infra10:55
*** sdake has quit IRC10:56
*** PollyZ has joined #openstack-infra10:56
*** yolanda has quit IRC10:57
*** sdake_ has quit IRC10:57
*** fawadkhaliq has joined #openstack-infra10:58
*** amotoki has joined #openstack-infra10:59
*** maishsk has quit IRC10:59
*** fawadk has quit IRC10:59
*** nwkarsten has joined #openstack-infra11:00
*** jcoufal has joined #openstack-infra11:00
*** asettle has quit IRC11:01
*** PollyZ has quit IRC11:02
*** maishsk has joined #openstack-infra11:03
*** mgould has joined #openstack-infra11:03
mgouldmorning everyone!11:04
*** amotoki has quit IRC11:04
mgouldcan people please review a simple change to tripleo gate?
*** sdake has joined #openstack-infra11:04
*** sdake has quit IRC11:04
mgouldremoves non-docs checks for docs-only patches, because I recently spent a fortnight trying to get a docs-only patch through tripleo CI and don't want anyone to have to go through that pain again11:05
*** pgadiya has quit IRC11:05
*** fawadkhaliq has quit IRC11:05
*** fawadkhaliq has joined #openstack-infra11:05
AJaegermgould: those skip-ifs are really evil, they can have nasty side effects. Please have some tripleo cores review them as well, I'd like to see consensus by tripleo team that they want this...11:06
*** sdake has joined #openstack-infra11:07
mgouldAJaeger: I think the +1s are from tripleo cores, but I'll check11:07
*** fawadk has joined #openstack-infra11:07
*** fawadkhaliq has quit IRC11:08
AJaegermgould: there's a single +111:08
mgouldAJaeger: oh, yeah, all the others were for previous patchsets. Sorry.11:10
*** kzaitsev_mb has joined #openstack-infra11:10
*** fawadkhaliq has joined #openstack-infra11:11
AJaegermgould: your change does not work11:11
AJaegerdid you check the log file?11:11
*** fawadk has quit IRC11:12
*** fawadkhaliq has quit IRC11:12
*** fawadkhaliq has joined #openstack-infra11:12
mgouldAJaeger: thanks, looking now11:13
mgouldis there a way I can test this locally?11:13
AJaegermgould: run "tox -e zuul" and check the output, best pipe it to a file like "tox -e zuul|tee out"11:13
AJaegermgould: problem is that there's a skip-if in line 1351 for triploe-ci-centos-7 - and that one matches your jobs as well and thus wins.11:14
*** fawadkhaliq has quit IRC11:14
mgouldAJaeger: thanks!11:14
*** yolanda has joined #openstack-infra11:14
*** fawadkhaliq has joined #openstack-infra11:15
AJaegeryou need to split up your change - add to that skip if to exclcude the triploe-ci, and handle the rest elsewhere11:15
AJaegerAnd then double check your chnage ,please - these skip-ifs are indeed nasty11:15
mgouldright, will do11:16
* AJaeger has seen more than once questions here: Why does that job not run - and skip-if was often the answer ;(11:16
mgouldis there a less nasty way of achieving what I want?11:16
AJaegerThat's why I'd like broad consensus, so that you know what you do...11:16
apuimedoHi guys! Time for the daily kuryr related questions :$11:16
apuimedoas part of the recommended split of our packages into three repos11:16
AJaegermgould: There's the files conditions which means: Run this job only if certain files change. But that's difficult for some kind of jobs to do, have a look how it's used in layout.yaml11:17
apuimedoinfra created the new kuryr-libnetwork and kuryr-kubernetes11:17
apuimedoand asked us if they should be cloned from somewhere else11:17
apuimedoor if they should be created anew11:17
*** yolanda has quit IRC11:17
*** electrofelix has quit IRC11:17
*** fawadkhaliq has quit IRC11:18
*** fawadkhaliq has joined #openstack-infra11:18
*** claudiub|2 has joined #openstack-infra11:18
AJaegerapuimedo: yeah - that was your one time chnage for importing it. Now everything needs to go through gerrit.11:19
apuimedowe said "anew". This was the right decision for kuryr-kubernetes. But it turns out it is not the best course of action for kuryr-libnetwork, which should have been cloned from 'kuryr' (and then start dropping parts in both 'kuryr' and kuryr-libnetwork)11:19
apuimedoAJaeger: good. I wanted to know if it was possibe to have  a do-over for that one to preserve history11:19
apuimedobut if it is not, well, what can we do, we fscked up :P11:20
*** rodrigods has quit IRC11:20
apuimedowe'll go through gerrit11:20
*** rodrigods has joined #openstack-infra11:20
AJaegerapuimedo: at project creation, it's an automatic import.11:20
*** lucasagomes is now known as lucas-hungry11:20
AJaegerAnything to do now, would be a manual setup - and for that you need to discuss with one of the admins - I'm not one - what is possible and whether somebody has time to do this.11:21
apuimedoAJaeger: understood11:21
AJaegerapuimedo: That's why we ask explicitely at project creation time - to save all of us time on it.11:21
*** claudiub has quit IRC11:21
apuimedoI also considered deleting the project and creating it again :P11:21
apuimedo(the subproject)11:21
apuimedobut that feels a bit weird too11:22
AJaegerapuimedo: manual task as well ;)11:22
*** electrofelix has joined #openstack-infra11:22
apuimedothanks a lot for the info AJaeger11:22
AJaegerapuimedo: come back later (US time) and see whether an admin is able to help.11:22
apuimedoAJaeger: how do I find out who the admins are?11:22
*** fawadk has joined #openstack-infra11:23
*** fawadkhaliq has quit IRC11:23
AJaegerapuimedo: just ask here and one will answer if they're awake ;)11:23
*** abregman|afk is now known as abregman11:24
apuimedovery well!11:24
apuimedothanks again11:24
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for GitLab notifier and connection property
*** sdake has quit IRC11:25
openstackgerritMerged openstack-infra/jenkins-job-builder: Use 'false' and 'true' in default docstrings
*** amotoki has joined #openstack-infra11:25
*** abregman is now known as abregman|afk11:25
*** Liuqing has quit IRC11:26
*** fawadk has quit IRC11:26
*** fawadkhaliq has joined #openstack-infra11:26
*** alaski is now known as lascii11:27
*** salv-orl_ has joined #openstack-infra11:30
*** bradm_ is now known as bradm11:30
*** fawadkhaliq has quit IRC11:31
*** ldnunes has joined #openstack-infra11:31
*** fawadkhaliq has joined #openstack-infra11:31
*** markvoelker has joined #openstack-infra11:31
*** amotoki has quit IRC11:33
*** salv-orlando has quit IRC11:33
*** amoralej is now known as amoralej|lunch11:33
*** eezhova has joined #openstack-infra11:34
*** yolanda has joined #openstack-infra11:35
*** sdague has joined #openstack-infra11:35
*** markvoelker has quit IRC11:36
openstackgerritMerged openstack/diskimage-builder: Export YUM variable in centos bases
*** chem has quit IRC11:36
*** fawadkhaliq has quit IRC11:37
*** PollyZ has joined #openstack-infra11:37
*** fawadkhaliq has joined #openstack-infra11:37
*** fawadkhaliq has quit IRC11:39
*** fawadkhaliq has joined #openstack-infra11:39
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci
*** PollyZ has quit IRC11:42
*** fawadkhaliq has quit IRC11:42
*** fawadkhaliq has joined #openstack-infra11:42
*** eezhova has quit IRC11:42
*** Na3iL has quit IRC11:42
*** fawadk has joined #openstack-infra11:46
*** fawadkhaliq has quit IRC11:46
*** javeriak has quit IRC11:47
*** fawadk has quit IRC11:48
stevemarAJaeger: silly question about 33634411:48
*** fawadkhaliq has joined #openstack-infra11:48
stevemarisn't it already in check and gate?11:48
*** ddieterly has joined #openstack-infra11:48
stevemaroh no, it's not :(11:48
openstackgerritMerged openstack-infra/storyboard: Add a way to retrieve the history of edited comments
*** fawadkhaliq has quit IRC11:50
*** ccamacho is now known as ccamacho|lunch11:50
*** ldnunes has quit IRC11:50
*** eezhova has joined #openstack-infra11:50
*** fawadkhaliq has joined #openstack-infra11:50
*** ldnunes has joined #openstack-infra11:50
*** PollyZ has joined #openstack-infra11:50
*** eezhova has quit IRC11:51
*** yamamoto has quit IRC11:52
*** mrmartin has joined #openstack-infra11:54
AJaegerstevemar: I didn't check - but since non-voting jobs are only in check, I would be surprised if it's already in both ;)11:54
*** edmondsw has joined #openstack-infra11:54
stevemarAJaeger: no you are right11:54
stevemarAJaeger: i think should work?11:54
stevemarAJaeger: i will send out a note to the ML and ask project PTLs to look at it11:55
*** itisha has joined #openstack-infra11:55
*** pt_15 has joined #openstack-infra11:55
*** PollyZ has quit IRC11:55
*** Daisy has quit IRC11:55
Zarawe've noticed a test reporting success that actually errors out (and then fails in the gate)-- the linting test as seen on this patch:
Zarawondering if anyone knows what's up?11:56
*** Daisy has joined #openstack-infra11:56
Zarait's possible I've missed a memo, been ill this week11:56
*** javeriak has joined #openstack-infra11:56
openstackgerritSteve Martinelli proposed openstack-infra/project-config: set check-osc-job to voting
*** fawadkhaliq has quit IRC11:58
*** fawadk has joined #openstack-infra11:58
AJaegerZara: looking at, it looks like broken npm packages ;(12:00
AJaegerstevemar: you cannot combine this together - let me check your patch in detail...12:00
AJaegerstevemar: we have separate template sections - and conditions.12:01
AJaegerYour first need to create a template to use it - and the conditions apply to jobs.12:01
*** raildo-afk is now known as raildo12:01
*** asettle has joined #openstack-infra12:02
AJaegerZara, try a recheck12:03
ZaraAJaeger: hm, I think it should fail; I'm wondering why it reports as success when the error is present... the error itself isn't weird; there's an extraneous variable.
*** fawadkhaliq has joined #openstack-infra12:03
AJaegerZara, ahh, that one - let me show you a fix...12:04
*** fawadk has quit IRC12:04
*** PollyZ has joined #openstack-infra12:04
AJaegerZara, fixed with
*** markusry has joined #openstack-infra12:04
AJaegerAnd your job was run before that - so, the recheck I just triggered will make it fail12:04
*** maishsk has quit IRC12:05
*** mrmartin has quit IRC12:05
ZaraAJaeger: that explains it, then; the first check was run before that. thanks for the link! :)12:06
*** aysyd has joined #openstack-infra12:06
*** ddieterly has quit IRC12:07
*** asettle has quit IRC12:07
*** fawadkhaliq has quit IRC12:07
*** fawadkhaliq has joined #openstack-infra12:08
*** jed56 has joined #openstack-infra12:09
mordredapuimedo: I can do an import for you12:09
*** PollyZ has quit IRC12:09
openstackgerritMiles Gould proposed openstack-infra/project-config: Skip non-doc checks for docs-only TripleO patches
*** derekjhyang has quit IRC12:11
*** Daisy has quit IRC12:12
*** Daisy has joined #openstack-infra12:13
*** fawadk has joined #openstack-infra12:13
*** asselin__ has joined #openstack-infra12:14
*** fawadkhaliq has quit IRC12:14
openstackgerritMiles Gould proposed openstack-infra/project-config: Skip non-doc checks for docs-only TripleO patches
*** Na3iL has joined #openstack-infra12:14
openstackgerritSteve Martinelli proposed openstack-infra/project-config: set check-osc-job to voting
*** kdas_ has joined #openstack-infra12:16
*** erlon has joined #openstack-infra12:16
*** markvoelker has joined #openstack-infra12:16
openstackgerritMerged openstack-infra/system-config: Do not use async with zl restarter playbook
*** Daisy has quit IRC12:17
*** bauzas is now known as bauwser12:17
*** fawadkhaliq has joined #openstack-infra12:18
*** PollyZ has joined #openstack-infra12:18
*** amotoki has joined #openstack-infra12:18
*** fawadkhaliq has quit IRC12:18
*** kushal is now known as Guest2903912:18
*** psilvad has joined #openstack-infra12:19
*** pradk has joined #openstack-infra12:19
*** fawadk has quit IRC12:20
*** Guest29039 has quit IRC12:20
*** afazekas_ is now known as afazekas|real12:20
*** andymaier has quit IRC12:20
*** zumbi has quit IRC12:21
*** rlandy has joined #openstack-infra12:21
*** zumbi has joined #openstack-infra12:21
*** kdas_ is now known as kushal12:22
*** kushal has joined #openstack-infra12:22
*** PollyZ has quit IRC12:22
*** lucas-hungry is now known as lucasagomes12:23
*** ddieterly has joined #openstack-infra12:24
*** inc0 has joined #openstack-infra12:24
*** nwkarsten has quit IRC12:25
*** fawadkhaliq has joined #openstack-infra12:25
*** nwkarsten has joined #openstack-infra12:25
*** baoli has joined #openstack-infra12:27
*** junoyoon has joined #openstack-infra12:27
*** amotoki has quit IRC12:27
*** eezhova has joined #openstack-infra12:27
*** baoli_ has joined #openstack-infra12:28
*** ddieterly has quit IRC12:29
junoyoonHi. while saving the contact info in, it shows the "Code Review - Error Server Error Cannot store contact information". Is there someone to look into this?12:29
*** _nadya_ has quit IRC12:30
*** fawadkhaliq has quit IRC12:30
*** nwkarsten has quit IRC12:30
*** _nadya_ has joined #openstack-infra12:30
*** zumbi has quit IRC12:31
*** PollyZ has joined #openstack-infra12:31
*** baoli has quit IRC12:32
fricklerjunoyoon: please look at and see if that helps12:32
*** asselin__ has quit IRC12:34
*** _nadya_ has quit IRC12:35
junoyoonfrickler: Thanks!! It now works/12:35
*** yolanda has quit IRC12:35
*** trown|outtypewww is now known as trown12:36
*** PollyZ has quit IRC12:36
*** yolanda has joined #openstack-infra12:36
*** javeriak has quit IRC12:37
*** javeriak has joined #openstack-infra12:37
*** zumbi has joined #openstack-infra12:38
*** claudiub|2 has quit IRC12:38
*** asettle has joined #openstack-infra12:41
*** PollyZ has joined #openstack-infra12:45
*** yamamoto has joined #openstack-infra12:45
*** ddieterly has joined #openstack-infra12:46
*** asettle has quit IRC12:47
*** rfolco has joined #openstack-infra12:47
*** esikachev has quit IRC12:48
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for Clang Scan-Build plugin version 1.7
*** matt-borland has joined #openstack-infra12:50
*** PollyZ has quit IRC12:50
*** jsavak has joined #openstack-infra12:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Update OSA job skip for all jobs except docs/releasenotes
odyssey4meAJaeger ^ I think that one is right.12:51
*** pfallenop has quit IRC12:52
*** pfallenop has joined #openstack-infra12:52
*** pfallenop has joined #openstack-infra12:52
*** salv-orl_ has quit IRC12:53
*** d0ugal has quit IRC12:54
*** d0ugal has joined #openstack-infra12:54
*** d0ugal has quit IRC12:54
*** d0ugal has joined #openstack-infra12:54
*** jsavak has quit IRC12:54
*** d0ugal has quit IRC12:55
*** d0ugal has joined #openstack-infra12:55
*** d0ugal has quit IRC12:55
*** d0ugal has joined #openstack-infra12:55
*** jsavak has joined #openstack-infra12:55
AJaegerodyssey4me: skip-ifs are dangerous, I need to be fully awake and concentrated to review them ;). Might not come to that today - and need the log files to double check as well.12:55
*** ddieterly has quit IRC12:56
*** javeriak has quit IRC12:57
*** PollyZ has joined #openstack-infra12:57
*** esikachev has joined #openstack-infra12:57
*** tlian has joined #openstack-infra12:58
odyssey4meAJaeger yep, no problem :)12:58
apuimedomordred: that would be awesome12:59
mordredapuimedo: you want the current contents of the kuryr repo pushed over top of the current contents of kuryr-libnetwork?13:00
*** d0ugal has quit IRC13:00
apuimedomordred: yes, exactly13:00
*** d0ugal has joined #openstack-infra13:00
*** PollyZ has quit IRC13:00
apuimedosince it is a split13:00
apuimedoit makes sense that the new repo takes the content of the old and trims away13:00
apuimedothan to put everything anew13:01
*** dizquierdo has joined #openstack-infra13:02
*** binbincong has quit IRC13:02
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI
*** bin_ has joined #openstack-infra13:02
*** d0ugal has quit IRC13:02
Zaraduring the sprint, people asked about making task notes read only by default. there's a patch in review here (I can merge, just want to check people want it so we don't end up undoing it again later): now's the time to indicate if you feel strongly. :)13:02
apuimedothanks ;-)13:02
*** d0ugal has joined #openstack-infra13:03
*** d0ugal has quit IRC13:03
*** d0ugal has joined #openstack-infra13:03
*** d0ugal has quit IRC13:03
*** markusry has quit IRC13:03
mordredapuimedo: done13:03
mordredapuimedo: I also put up a gitreview patch to change it13:03
openstackgerritJames Page proposed openstack-infra/project-config: Add repositories for Charm interfaces
*** Guest12790 has joined #openstack-infra13:03
apuimedomordred: which patch?13:04
*** Guest12790 has quit IRC13:04
apuimedooh, of course13:04
*** junoyoon has quit IRC13:04
*** d0ugal has joined #openstack-infra13:04
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI
apuimedothanks for that13:04
apuimedovery thoughtful13:04
*** Ravikiran_K has joined #openstack-infra13:04
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Update eavesdrop site name
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Rename the eavesdrop site to be clearer
*** kushal has quit IRC13:06
*** markusry has joined #openstack-infra13:08
*** cshastri has joined #openstack-infra13:10
*** amoralej|lunch is now known as amoralej13:11
fungiAJaeger: rax-dfw images should eventually update. probably ran into an upload issue yesterday or something13:14
*** psilvad has quit IRC13:15
*** psilvad has joined #openstack-infra13:15
*** psachin has quit IRC13:16
*** fawadkhaliq has joined #openstack-infra13:17
*** fawadkhaliq has quit IRC13:21
*** azvyagintsev has joined #openstack-infra13:21
*** amotoki has joined #openstack-infra13:21
*** psilvad has quit IRC13:21
*** psilvad has joined #openstack-infra13:22
*** abregman|afk is now known as abregman13:24
AJaegerok, so I'll wait another day and check again.13:24
azvyagintsevHi folks and gurus ! i'm thinking about implementation CI for openstack/murano-apps . current idea is - combine hardware lab , and devstack-murano slave(via nodepool)   (i guess, currently it works in that way in openstack/murano-deployment ci)13:25
yolandahi mordred, around? can you review some of my changes with topic puppet-infracloud?13:25
azvyagintsevi can flame with someone from infra about this ? exactly - about best practice for devstack + *some_app* + hardware lab13:25
AJaegerfungi, do you want to merge clarkb's python35 patch today - or wait until Tuesday? Depending on that I'd like to push - or not - my rework of the publishing jobs Those two conflict heavily and if mine merges first, I'll rebase clarkb's.13:25
fungiAJaeger: clarkb indicated he expected the py35 change to need some rebase treadmill, so merging your publishing job rework seems fine13:27
*** psilvad has quit IRC13:27
*** mriedem has joined #openstack-infra13:27
*** signed8bit has joined #openstack-infra13:28
*** psilvad has joined #openstack-infra13:28
fungiwe were just hesitant to push the py35 change heading into a holiday weekend for a significant chunk of our community (canada is off today, usa monday)13:28
AJaegerfungi, frickler was so nice to rebase already, so clarkb's one is good to merge right now...13:28
*** sree has joined #openstack-infra13:28
*** akshai has joined #openstack-infra13:28
*** ayoung has joined #openstack-infra13:29
AJaegerfungi, understood - even if I see that myself not as problematic since the jobs are non-voting...13:29
fungiAJaeger: if clarkb pops in here in a bit, we can revisit13:30
*** ddieterly has joined #openstack-infra13:30
*** ddieterly is now known as ddieterly[away]13:30
AJaegeryeah, if we do the python35 today, we should do it early today - in case of problems...13:31
*** kushal has joined #openstack-infra13:33
*** eezhova has quit IRC13:34
*** dansmith is now known as superdan13:34
*** tpsilva has joined #openstack-infra13:34
*** Liuqing has joined #openstack-infra13:36
*** bryan_att has joined #openstack-infra13:37
*** Na3iL has quit IRC13:38
fungiazvyagintsev: chances are you'll find people in the third-party ci meetings with more experience in running a ci on bare metal. everything we do upstream is currently in virtual server instances13:38
*** ddieterly[away] is now known as ddieterly13:39
*** Jeffrey4l has quit IRC13:40
azvyagintsevfungi: thx! hm, did you ran vm inside vm ?(not cirros, ubuntu\debian at least) (i mean, at least for something, is nested performance enough for tests ?)13:40
*** ajmiller has quit IRC13:41
*** ajmiller has joined #openstack-infra13:41
*** abregman is now known as abregman|afk13:42
fungiazvyagintsev: there is a little of that, though performance is pretty terrible for that sort of thing so mocked interactions and avoiding nested virtualization where possible usually provide sufficient alternatives13:42
openstackgerritMerged openstack-infra/puppet-jenkins: Remove unused GnuPG keyring
openstackgerritMerged openstack-infra/system-config: Project renames may need governance updates
azvyagintsevok, thanks13:42
fungiazvyagintsev: for example, beaker normally expects to run things in a virtual machine/container, but we configure it to treat the local system as if it were a remote vm13:43
openstackgerritMerged openstack-infra/system-config: Filter nonexistent groups in
*** daemontool has joined #openstack-infra13:43
fungiazvyagintsev: and when complete contextual separation is required, we run jobs that span multiple virtual machines rather than trying to nest multiple virtual machines inside a virtual machine13:44
openstackgerritMerged openstack-infra/system-config: Publish eavesdrop to yaml2ical dir
fungiazvyagintsev: for example, jobs that utilize a multi-node devstack deployment13:45
*** eezhova has joined #openstack-infra13:45
openstackgerritMerged openstack-infra/jenkins-job-builder: Adding property-file into the pipeline publisher
*** ajmiller has quit IRC13:46
*** ajmiller has joined #openstack-infra13:47
azvyagintsevhmm, lost point - 'we run jobs that span multiple virtual machines' - it's not for case, when node should spawn VM inside openstack , right ?13:47
*** gouthamr has joined #openstack-infra13:47
*** thiagop has joined #openstack-infra13:48
*** r-mibu has quit IRC13:48
*** r-mibu has joined #openstack-infra13:48
*** asettle has joined #openstack-infra13:49
openstackgerritMerged openstack-infra/git-review: Use consistent formatting for deprecations
*** rbrndt has joined #openstack-infra13:50
*** gouthamr_ has joined #openstack-infra13:51
*** cshastri has quit IRC13:52
openstackgerritMerged openstack-infra/jenkins-job-builder: Adds matrix restriction to trigger-builds plugin
fungiazvyagintsev: well, you can perhaps test those two things separately? e.g., test that your software is able to boot a vm (something small like cirros), and then test your software that should run in that vm by installing it locally on the job node13:53
AJaegerfungi, do we have enough resources to take roughly 40 more repos on? We have around 20 new fuel repos and 15+ charm repos in the project-config queue..13:54
fungioh wow13:54
fungiwell, additional repos don't themselves necessarily consume any additional "resources"13:54
*** gouthamr has quit IRC13:55
openstackgerritMerged openstack-infra/system-config: Add a node for artifact signing jobs
AJaegerdisk space + jobs ;)13:55
fungithe majority of new repos add very few jobs13:55
openstackgerritMerged openstack-infra/jenkins-job-builder: Add "every online node" factory to Parameterized Trigger builder
AJaegernothing out of the normal from what I can see for each, just more repos than usual ;)13:55
*** gouthamr_ is now known as gouthamr13:55
*** sree has quit IRC13:56
*** sorantis has quit IRC13:56
azvyagintsevfungi: that an main problem- no i can't. i'm going to test murano-apps. for this i need openstack + murano , then i import murano-app =>  run it => app spawn 1-5 VMs, and so on13:56
* AJaeger hasn't reviewed in full detail yet, just did some glance over13:56
azvyagintsevso its require pretty huge(in tests meaning) resources13:56
fungiazvyagintsev: yep, sounds like a good case for a third-party ci system then. that's pretty far out of scope for our existing test infrastructure13:58
fungiazvyagintsev: our ci manages virtual machines directly, you want jobs that manage virtual machines instead. you're likely going to have some additional struggles with protecting credentials, so you likely need something like what nodepool does, but for creating and destroying tenants in your test cloud rather than creating and destroying individual nova instances14:00
*** mhickey has joined #openstack-infra14:00
*** eharney has joined #openstack-infra14:00
azvyagintsevah, so anyway thanks :)  for me- it's pretty clean how-to achieve it, but not clean - how i should  integrate my idea with infra. so, if you sure that it's fully for third-party ci...i don't have q for you  :) thx  one more time!14:01
fungiazvyagintsev: that way if a job leaks the keystone credentials for the tenant it's using, that doesn't matter because you're going to blow that tenant away as soon as your job completes and use a different tenant for the next job14:01
*** amitgandhinz has quit IRC14:01
azvyagintsevyeah, i think about tenant-per-job,  so, i don't want to recreate it each run, at least for now14:01
*** amitgandhinz has joined #openstack-infra14:02
*** Liuqing has quit IRC14:02
fungiazvyagintsev: if you reuse tenants, i wonder how you prevent a test for a random gerrit change from publishing the credentials for the tenant it uses14:02
*** tonytan4ever has joined #openstack-infra14:03
azvyagintsevyup, but as i saw - third-party CI usually use internal openstack, w\o external access14:03
fungiazvyagintsev: unless you consider that running untrusted changes from gerrit actually _is_ a form of external access14:04
*** yolanda has quit IRC14:04
*** esikachev has quit IRC14:05
*** yolanda has joined #openstack-infra14:05
funginot a ton of difference between "i can log into your environment and run something" or "i can post something to gerrit and get it to run in your environment"14:05
azvyagintsevsome kind. bit test tenant don't have admin access and so on...but yes, i guess you right. not its work on you'r side ? some pre-job hook ?14:05
azvyagintsev*but test14:06
*** Liuqing has joined #openstack-infra14:06
*** yamamoto has quit IRC14:06
azvyagintsevi mean, how exactly you create test-tenant, w\o sharing admin tenant creds in logs ?14:06
*** tonytan4ever has quit IRC14:07
*** tonytan4ever has joined #openstack-infra14:07
fungilike i said, one way to do it is with a separate service. like how we entrust keystone credentials to nodepool and have it create virtual machines, then zuul assigns jobs to those virtual machines. you could write a service which has your cloud admin credentials and creates tenants, which your job scheduler then assigns jobs to14:08
openstackgerritMerged openstack-infra/tripleo-ci: pingtest: don't show error if no images needs to be cleaned-up
*** david-lyle has quit IRC14:10
azvyagintsevcatch it. ok, thx for describing14:11
*** kgiusti has joined #openstack-infra14:11
*** daemontool has quit IRC14:12
azvyagintsevbtw, sounds like - we need another one tool, tenant-pool :D14:14
*** daemontool has joined #openstack-infra14:14
*** TaylorH has joined #openstack-infra14:19
*** Srinu has joined #openstack-infra14:19
SrinuAsselin_: hi14:20
SrinuAsselin_: can we use two jenkins master in the same network r not14:21
*** TaylorHuang has quit IRC14:22
*** yamamoto has joined #openstack-infra14:23
fungiazvyagintsev: it's also possible that would be a worthwhile extension to nodepool (the resource tracking framework itself could probably be entirely reused for that at least), but i don't know. it would be a pretty large scope change so i'm not sure how i feel about that. would want to hear feedback from other nodepool devs14:23
SrinuCan we use two jenkins master in the same network14:24
SrinuFungi: hi14:24
mordredfungi, azvyagintsev: I think it would not be a terrible thing to add to nodepool - and would work nicely even in upstream infra when our clouds grow domain support so that we could have nodepool have domain-admin14:25
*** links has quit IRC14:25
fungiSrinu: we used to have 8 jenkins masters in one network, tied to a single instance of zuul. so yes, that is possible. it's the reason we originally created the jenkins-gearman plugin and gearman-based zuul v214:25
mordredfungi, azvyagintsev: it would, however, absolutely require domain support in our clouds for infra to run such a thing14:25
azvyagintsevyeah, its perfectly for nodepool - just injecting some pre-generated operc file, and then connect it like env-file for slave. sounds good14:25
mordredfungi, azvyagintsev: although I recommend s/tenant/project/ in your brains :)14:26
fungimy brain will still always have tenant stuck in it, since that's what we public cloud providers called them way back in the beforetime, pre-openstack14:27
SrinuFungi: i have one jenkins master with gearman  enable and how can i add anther jenkins to it14:27
fungiSrinu: you just connect the additional jenkins to gearman the same way you did the first one14:28
*** links has joined #openstack-infra14:28
fungiSrinu: and if you're using nodepool, configure it with the new jenkins master as a second "target"14:28
fungiSrinu: and also as a second zmq-publisher14:29
funginodepool and zuul are designed to work with an arbitrary number of jenkins masters14:30
SrinuFungi:i installed 2 jenkins master with 2 different slaves. I enable gearman in both the jenkins with different zuul14:30
*** zhurong has joined #openstack-infra14:30
openstackgerritAlfredo Moralejo proposed openstack/diskimage-builder: Remove python-setuptools from yum excludes
fungiSrinu: why a different zuul? you should be able to have one zuul with multiple jenkins masters14:31
*** esikachev has joined #openstack-infra14:31
*** xyang1 has joined #openstack-infra14:32
clarkbAJaeger: fungi I guess we could merge it today if we want. we should send email explaining what new job is, details on how to add py35 tox env if necessary and how to make job voting if/when it passes14:32
clarkbI have not drafted that yet and wont be able to for a biy due to morning chores14:33
*** baoli_ has quit IRC14:33
*** baoli has joined #openstack-infra14:34
AJaegermorning, clarkb. I guess, in that case, we should draft - or even send - the email today and merge on Tuesday.14:34
*** salv-orlando has joined #openstack-infra14:34
*** Srinu has quit IRC14:34
AJaegerclarkb: then my offer stands: If merges earlier, I'll rebase the your python35 change.14:36
* AJaeger now needs to do some chores as well...14:36
clarkbAJaeger: ya dont worry about conflicting with it, I or you or frickler (thhanks!) can rebase as necessary14:36
AJaegerclarkb: my change removes 800 jobs...14:38
clarkbAJaeger: ooh I should review it then :)14:38
*** armax has joined #openstack-infra14:39
AJaegerclarkb: and some low-hanging fruits: and - that's all cleanup I came up with...14:39
AJaegernow really until later ;)14:39
*** Srinu has joined #openstack-infra14:40
*** salv-orlando has quit IRC14:40
SrinuFungi: can you suggest me links to do that14:41
*** slaweq has quit IRC14:41
mordredclarkb: the change looks good to me and already has another +2 - do you want me to hold off on approving so you can look at it?14:41
*** zz_dimtruck is now known as dimtruck14:41
apuimedodhellmann: mordred mentioned in the ml that the use of python packages can bring some issues and that you could tell me more about it14:42
mordredapuimedo: fungi might also remember the issue that oslo ran in to with namespace packages14:42
*** yamamoto has quit IRC14:43
apuimedoyes, they moved from oslo.config to oslo_config, right?14:44
*** snarwade has quit IRC14:44
*** baoli_ has joined #openstack-infra14:45
clarkbmordred: nope go for it14:46
SrinuClarkb: hi14:46
clarkbSrinu: hello14:46
SrinuClarkb: how can i make 2 jenkins to use single zuul to get the streams by enabling gearman14:47
*** baoli has quit IRC14:48
clarkbSrinu: configure them to talk to the same gearman server as zuul14:48
SrinuClarkb: i have two separate jenkins with different slaves..14:49
mordredSrinu: one gearman server. zuul talks to gearman. each jenkins talks to gearman.14:49
SrinuClarkb: i want to make to listen same zuul14:49
SrinuClarkb: i enable gearman in both jenkins14:50
*** chem has joined #openstack-infra14:50
*** xarses has quit IRC14:50
*** dingyichen has quit IRC14:53
SrinuMordred: i enable gearman in both jenkins masterp14:53
openstackgerritMerged openstack-infra/jenkins-job-builder: Properties: fix copyartifact permissions with several projects
*** pilgrimstack has quit IRC14:54
*** pilgrimstack has joined #openstack-infra14:55
*** arxcruz has quit IRC14:55
clarkbSrinu: just make sure that zuul and the jenkins masters use the same gearman server and it will do what you want14:56
*** infra-red has joined #openstack-infra14:56
*** yolanda has quit IRC14:56
*** yolanda has joined #openstack-infra14:57
openstackgerritMerged openstack-infra/project-config: Rework publishing jobs
SrinuClarkb:  in another jenkins master i have to use gearman ip to connect to first jenkins gearman right14:57
*** infra-red has quit IRC14:59
*** dizquierdo has quit IRC15:00
*** sputnik13 has joined #openstack-infra15:00
buntingHi, what could cause permissions of a folder to be diffrent between two diffrent functional tests? Since I am running into "Permission denied: '/var/cache/glance'"15:00
fungimordred: apuimedo: if memory serves, it's that they fight over which one controls the for the namespace15:01
clarkbSrinu: use the same server for both so should be the same ip15:01
*** links has quit IRC15:02
fungiapuimedo: has some details, it seems15:02
*** makowals has quit IRC15:02
clarkbbunting: possibly different writers setting different permissions15:03
*** aysyd has quit IRC15:03
clarkbbunting: I would start with figuring out what writrs the files and work from there15:03
*** piet has joined #openstack-infra15:04
*** aysyd has joined #openstack-infra15:06
buntingclarkb: This is the glance-api trying to create the folder. I'm looking at a glance_store functional test (which runs into the problem) vs a nova functioal test (which works).15:06
AJaegermordred: thanks for review15:06
clarkbbunting: well nova functional tests may not write to /var/cache at all15:07
*** jistr is now known as jistr|mtg15:07
clarkbbunting: you will need root perms tto create that dir15:07
*** pcaruana has quit IRC15:08
AJaegerfrickler, clarkb :I'm rebasing 336272 - the py35 patch - now15:08
SrinuClarkb: i installed one jenkins with gearman enable with one slave and another jenkins master with different slave but i am facing this issue when on jenkins master report sucess another jenkins master node is going offline15:10
*** _nadya_ has joined #openstack-infra15:10
*** salv-orlando has joined #openstack-infra15:10
*** xarses has joined #openstack-infra15:13
*** rcernin has quit IRC15:14
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python35 jobs
*** dimtruck is now known as zz_dimtruck15:15
clarkbSrinu: maybe they both ran jobs? Ehat you describe isnt necessarily an issue if both masters are activelyy running jobs15:16
*** ajmiller_ has joined #openstack-infra15:16
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI
*** salv-orlando has quit IRC15:17
fungijeblair: i guess you can un-wip (and approve?) 330698 after the zuul.o.o distro upgrade today15:18
*** hongbin_ has joined #openstack-infra15:18
SrinuClarkb: they are running jobs but when one reaches success another jenkins master slave node went offline15:19
*** ajmiller has quit IRC15:19
*** locust has joined #openstack-infra15:20
*** xarses has quit IRC15:20
*** xarses has joined #openstack-infra15:20
*** pradk_ has joined #openstack-infra15:20
AJaegerclarkb, what about splitting the layout.yaml change up - only add the new jobs to infra repos. That way we can guinia pig a bit...15:21
*** amitgandhinz has quit IRC15:21
*** hongbin_ has quit IRC15:21
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update ScriptTrigger plugin
*** jamesmcarthur has joined #openstack-infra15:22
AJaegerbut probably not worth it since it's non-voting and in addition - not replacing15:22
*** amitgandhinz has joined #openstack-infra15:22
*** hongbin_ has joined #openstack-infra15:22
clarkbAJaeger: we could do that, we can also split by db vs not db additions15:22
fungiSrinu: is there any chance you have your two jenkins masters sharing a slave node or something?15:22
clarkbbut ya concern is less about it breaking and more helping people understand and fix any issues15:22
fungijeblair: is it safe to go ahead and launch the new zuul.o.o instance now so that we can prep the firewall changes, or is there a risk that it will connect to gerrit and try to do things? granted its gearman will have no job registrations/workers/mergers available so it won't be able to do much more than make annoying comments if anything15:23
*** kushal has quit IRC15:23
*** amotoki has quit IRC15:23
fungioh, and leave bad votes of course15:24
fungiahh, but our puppet-zuul module doesn't auto-start the service!15:24
*** sputnik13 has quit IRC15:24
fungiso should be 100% safe right?15:24
jeblairfungi: i think there is a good chance of that, but yes, hopefully it will just wait for mergers to show up before actually saying anything.  so i think it should be okay if you keep a close eye on it15:24
*** Jeffrey4l has joined #openstack-infra15:24
jeblairfungi: (hopefully it configures it to start on boot though, and launch-node tells it to reboot)15:24
*** Jeffrey4l has quit IRC15:24
fungid'oh! right15:25
*** Srinu has quit IRC15:25
clarkbyou could disable puppet on old host, update key info to be temporarily invalid then make new host15:25
AJaegerclarkb: let's leave as is and I'm fine with merging completely - splitting it up is a boring split ;)15:25
fungii guess i can launch it, then ssh in immediately after the reboot and stop the zuul service15:25
clarkbAJaeger: ok15:26
*** Kiall has joined #openstack-infra15:26
*** esikachev has quit IRC15:26
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add plugin="junit" attribute for JUnit plugin
fungicollateral damage should be minimal as long as i stop it right away15:26
jeblairclarkb: yeah, but i think it's not going to actually do anything as fungi says, so i think the risk is low enough to just manually stop it asap15:26
*** vhosakot has joined #openstack-infra15:27
*** esikachev has joined #openstack-infra15:27
fungijeblair: oh, weird... we have /opt double-mounted on the current zuul.o.o15:28
*** daemontool has quit IRC15:28
fungidf lists it twice (with the same blockdev)15:28
fungithat's... odd15:28
*** nwkarsten has joined #openstack-infra15:28
*** tesseract- has quit IRC15:28
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update OpenShift Pipeline plugin
*** PollyZ has joined #openstack-infra15:32
*** zhurong has quit IRC15:32
*** amitgandhinz has quit IRC15:32
*** esikachev has quit IRC15:33
*** nwkarsten has quit IRC15:33
*** Kiall has quit IRC15:33
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update GitHub pull request builder plugin
fungijeblair: another related question... it looks like we have a 200gb cinder volume (10447e03-2bf6-4ea1-b7c0-d16ee90a673b) at xvdb but instead we're using the ephemeral disk at xvde for /opt15:34
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update bazaar plugin
fungii guess that's less of a question and more of a statement, to be followed by the question "do we need it?"15:34
fungialso i'm pretty sure we no longer need the tmpfs mounted at /var/lib/zuul/git since that predates the separate mergers15:36
fungiand /opt seems to be mostly filled with cruft15:37
locusthello, is there existing config for nodepool15:37
clarkbI think zuul foesnt need that now that mergers are separate15:37
locustI know there exists project-config-example15:38
openstackgerritMerged openstack-infra/system-config: Rename the eavesdrop site to be clearer
clarkblocust: yes though practice says they arent very reusable because clouds differ as do image needs and so on15:38
fungilocust: your question puzzles me. we run nodepool, so we have _a_ configuration (it's in openstacl-infra/project-config) but it's probably not usable as _your_ configuration15:38
locustfungi: hm15:39
jeblairclarkb, fungi: i think the cinder volume was used for the ginormous debug logs we needed a while back?  so i think the answer is no we don't need it anymore15:39
locustI am deploing third party CI, so thinking what to start with?15:40
fungilocust: does the puppet-openstackci module provide a templated one?15:40
fungino, i guess it just expects you to provide your own project-config15:41
locustas I see --- not15:41
*** ifarkas has quit IRC15:41
fungioh! it does15:42
fungithere's the openstack-infra/project-config-example repo which has a simple one in it15:42
*** thiagop is now known as thiagop-afk15:42
*** SumitNaiksatam has quit IRC15:43
*** yamamoto has joined #openstack-infra15:43
fungijeblair: also now that the memory leak is under control, it looks like we could easily get by with a 15gb flavor instead of 30gb?
*** david-lyle has joined #openstack-infra15:44
locustfungi: yes, but it is not working from the box15:44
fungilocust: nothing is going to work "out of the box"15:44
jeblairfungi: wow, i'm surprised that's so high15:44
*** salv-orlando has joined #openstack-infra15:45
jeblairfungi: we *should* need half that.  but i agree we appear to need more.15:45
jeblair(i think something is still wonky)15:45
fungilocust: it's a daemon for managing virtual machine resources, so without knowing your cloud provider, your image needs, et cetera there's no way to make it work "out of the box"15:45
*** markusry has quit IRC15:47
locustfungi: I thought this is just for creating images15:47
locustlike build some thign like bootable iso15:47
*** amotoki has joined #openstack-infra15:47
fungilocust: probably the best place to start is asking, what do you want it to do? (make images, but with what in them? boot virtual machines, but where and how many?)15:47
AJaegerare we still having a downtime at 20:00 UTC? Should we send a reminder?15:47
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Update OSA job skip for all jobs except docs/releasenotes
fungilocust: perhaps it's best to read and then come back with clearer questions15:48
*** roxanagh_ has joined #openstack-infra15:48
locustI want image of "All in one" deployment of openstack to test cinder driver15:48
*** karthik__ has joined #openstack-infra15:48
locustI do not expect to have lots of VM's may be one15:49
locustis it enoungh to support driver testing?15:49
fungilocust: you want an image of a devstack deployment to use for local testing of something?15:49
*** PollyZ has quit IRC15:49
fungii don't know what you mean by "driver testing"15:49
*** roxanagh_ has quit IRC15:49
*** amotoki has quit IRC15:49
*** amotoki has joined #openstack-infra15:50
*** karthik__ has quit IRC15:50
*** jistr|mtg is now known as jistr15:50
*** andreas_s has quit IRC15:50
locustfungi: I am making Cinder to be "Friendly" to some new Device15:50
fungiAJaeger: yeah, i was going to wait until closer, like maybe 18:00 utc for a 2-hour warning15:50
fungilocust: yeah, i know what a driver is, but please elaborate on _what_ about it you expect to be testing? are you just running some local tests you've devised to help you write that driver?15:51
locustfungi: and to provide commit of this updates I need to setup CI on my side, to test my code15:51
*** mandre has quit IRC15:52
*** salv-orlando has quit IRC15:52
locustfungi: no, I am using tempest to proof that my code is valid and to commit it to Cinder repo15:52
*** hashar has quit IRC15:53
AJaegerfungi, ok. Was expecting an email - but that can go out around 18:00 as well...15:53
fungilocust: are you saying you expect to test changes other people propose to cinder to make sure they don't break your driver? if so, you're not going to get by with just one vm, i don't think. you'll be booting a lot of them and not reusing them because you're giving someone else (by way of testing their proposed patches) root access on those virtual machines15:53
*** Nikolay_St has quit IRC15:53
*** pilgrimstack has quit IRC15:53
*** PollyZ has joined #openstack-infra15:54
* AJaeger needs a break now - if I don't come back online today, I wish everybody here based in the US a great 4th of July weekend!15:55
locustfungi: m... my first prioryty target is to test MY code15:55
locustfungi: in the whay it is acceptable by Cinder comunyty15:56
fungilocust: okay, in that case you're probably better off starting from installing devstack in a virtual machine. it's designed to be very easy to set up15:56
*** sree has joined #openstack-infra15:56
locustfungi: yes, I've already done this15:56
locustand my code is passing tests15:57
*** david-lyle_ has joined #openstack-infra15:57
fungilocust: then it sounds like you've met your stated goal?15:57
*** david-lyle_ has quit IRC15:57
*** eezhova has quit IRC15:57
*** mandre has joined #openstack-infra15:58
locustso now I am working on contributing to Cinder, and folks at Cinder will not accept my code just because it is avesome, and I have done local tests15:58
*** yamamoto has quit IRC15:58
locustfungi: they want to see logs, and see interractive code testing15:58
*** karthik__ has joined #openstack-infra15:58
fungilocust: right, as far as i know the cinder team wants _you_ to test _other_people's_ proposed patches to cinder to make sure they don't break your driver15:59
*** dtantsur is now known as dtantsur|afk15:59
*** Swami has joined #openstack-infra15:59
locustand this too15:59
*** jlanoux has quit IRC15:59
locustfungi: ok, yes16:00
*** sree has quit IRC16:00
*** david-lyle_ has joined #openstack-infra16:00
fungilocust: i think there must be some critical context missing either on my end or your end of this discussion. you say you _just_ want something to make a virtual machine image for you, but that's not the extent of what the cinder team is going to require you to have in place as far as i know16:01
fungilocust: the best place to start is probably talking to other people who are doing similar things with third-party ci for cinder drivers. here is a good place to discuss the tools we've written which you can assemble into a ci system, but how to assemble those components to best test changes to cinder against your driver/hardware isn't really something we can help with16:01
fungithat's more of a policy question16:02
*** tqtran has joined #openstack-infra16:03
locustfungi: hm, you are write, bu not all Cinder developers are familiare with Puppet, Nodepool16:03
fungilocust: there's a bunch of third-party ci operators who meet weekly on mondays (i don't know if they're planning to meet this coming monday because it's a holiday for a lot of the people who normally participate)
fungithat's a good opportunity to get to know other people who are doing the things you're talking about needing to do16:03
pleia2good morning16:04
*** david-lyle has quit IRC16:04
*** david-lyle_ is now known as david-lyle16:04
fungilocust: the cinder devs can tell you what they expect from you, and the other third-party ci operators testing cinder drivers can tell you what specific software and configuration they've put in place to meet those expectations (much of which might be software we've written)16:05
*** baoli_ has quit IRC16:05
*** Apoorva has joined #openstack-infra16:06
locustfungi: yes, but I thought that here I can find good clues of how to start with One simple VM running ubuntu with devstack16:06
*** Apoorva has quit IRC16:06
*** _nadya_ has quit IRC16:06
locustfungi: not solve all my problems but find a ground to start with16:07
fungilocust: you've already started with one simple vm running ubuntu with devstack. what we have is software to automate doing that on a mass scale16:07
*** PollyZ has quit IRC16:07
*** tqtran has quit IRC16:07
fungilocust: and that automation at scale expects you've met a variety of other prerequisites such as having a tenant/project in an openstack cloud available with some quota set16:08
fungithe sorts of configuration you'll need to plug into nodepool (we can't know what that will be, which is why there is no nodepool config that will work "out of the box")16:09
*** _nadya_ has joined #openstack-infra16:09
*** tphummel has joined #openstack-infra16:10
locustfungi: so for instance if I look at nodepool.yaml at the fild of elements of diskimages: evry folder in elements must be writen by my hands?16:11
locustfugi: like write my own automation of image building?16:11
*** mriedem has quit IRC16:11
fungilocust: to summarize, nodepool's designed to use diskimage-builder to create images, upload them to a cloud, boot a defined number of single-use virtual machines from those images and register them to a job worker (such as a jenkins), and to track demand expressed by zuul (a job scheduler) to determine how many instances to boot, and to delete those instances once their jobs complete16:12
*** mriedem has joined #openstack-infra16:12
*** mriedem has quit IRC16:12
jeblairclarkb: python error in 336311, otherwise solution lgtm16:13
*** edmondsw has quit IRC16:13
clarkbjeblair: woo, can I blame juggling babies when writing that change?16:14
clarkbwe are in the "we can almost walk now let me show you" stage16:14
locustfungi: and I need to write my own image killing and booting according to zuul output?16:15
fungilocust: the example config is reusable. it looks like it's designed to expect a local jenkins master running on the same machine as nodepool, and a local zuul server also running on that machine, and will communicate with an openstack cloud keystone endpoint specified in your clouds.yaml as "local_01" to build and upload a predefined ubuntu image and boot up to 2 virtual machines at a time from that16:15
jeblairclarkb: you should just blame the babies!  i bet they get boolean coercions wrong all the time!  :)16:15
*** ihrachys has quit IRC16:15
mordredjeblair: I find it hard to think that clarkb's babies would ever make a boolean mistake16:15
fungilocust: it looks like there is a readme in the project-config-example repo that expresses some of this
locustfungi: thank you16:17
pleia2I'm going to reply to my initial email about the Gerrit downtime today as a reminder, have there been any changes to that? (2000 UTC, 2 hour window)16:17
locustfungi: gona read this one16:17
fungipleia2: we're still on schedule afaik16:17
pleia2fungi: thanks, sending reminder then :)16:18
fungii was just trying to finish establishing parameters for the replacement zuul so i can know the new ip addresses and start writing up the necessary config changes16:18
*** john-davidge has quit IRC16:18
*** coolsvap has quit IRC16:18
jeblairoh good, i was just about to go try to find out what i had committed to :)16:18
fungijeblair: you committed to "being around" as far as i'm concerned, so however much or little you want to do is appreciated16:19
jeblairfungi: and now i know when to be around!  :)16:19
*** jpich has quit IRC16:19
jeblairclarkb: i dropped off the planet right after we wrote the zmq fixes -- did they get merged, and appropriate things restarted?16:20
clarkbjeblair: they did though jhesketh had a much more extensive change you should consider16:21
*** Apoorva has joined #openstack-infra16:22
clarkbI need to check if all my restarts completed last night16:22
jeblairclarkb: any hints as to which change that might be?16:23
clarkbit was linked to in comments of your change16:23
fungijeblair: so you're cool with using a 15gb flavor this time around, or should i stick to 30gb?16:24
jeblairoh, it's on the "zuulv25-tests" branch16:24
*** PollyZ has joined #openstack-infra16:25
*** amotoki has quit IRC16:27
jeblairfungi: 15g sgtm16:28
*** shashank_hegde has joined #openstack-infra16:28
jeblairclarkb: 317481; reviewed16:28
*** pfallenop has quit IRC16:29
*** watersoul has quit IRC16:29
fungijeblair: as far as i can tell, the existing zuul is only referenced anywhere by hostname, so no explicit ip addresses to patch in any of our repos16:30
fungiwe'll just need to reapply iptables rules in a few places after we change dns16:31
jeblairbtw, i consider zuul v2.5 finished and have switched back to working on v3.16:31
jeblairfungi: agreed16:31
fungijeblair: awesome16:31
*** daemontool has joined #openstack-infra16:31
fungiand also restart everything that talks to gearman (nodepool, zuul mergers, zuul launchers)16:32
*** roxanagh_ has joined #openstack-infra16:32
openstackgerritKaiyan Sheng proposed openstack-infra/project-config: Unlink monasca gate jobs from neutron testes
fungii'm making sure the a/aaaa records for zuul and static are 5 minutes16:33
*** gyee has joined #openstack-infra16:33
fungipretty sure i previously made certain all the virtual hosts served from static.o.o are just cnames16:33
fungibut i'll double-check that shortly16:34
fungiokay, i managed to jump on the replacement zuul as soon as ssh would let me in after the reboot and stopped the daemon16:35
fungiso less than a minute of it running16:35
pleia2I can check on cnames real quick if you'd like16:36
fungipleia2: that would be a huge help--thanks!16:36
pleia2on it16:36
*** ociuhandu has quit IRC16:37
*** hongbin_ has quit IRC16:38
*** asettle has quit IRC16:38
*** lucasagomes is now known as lucas-afk16:38
pleia2tarballs.o.o and are both A records16:38
fungizuul records were already 5 minutes, static records were 60 but i've lowered them to 5. the new ttl should be propagated everywhere with plenty of time to spare16:38
fungipleia2: thanks, i'll switch those to cnames now16:39
*** piet has quit IRC16:39
pleia2ci, docs-draft, security, governance, specs, summit, trystack and releases are all cnams16:39
*** shashank_hegde has quit IRC16:39
*** derekh has quit IRC16:39
pleia2do we have anything outside of static.pp?16:39
*** nwkarsten has joined #openstack-infra16:40
funginot to my knowledge, no16:40
*** hongbin_ has joined #openstack-infra16:40
pleia2alright, checked everything there16:40
openstackgerritKaiyan Sheng proposed openstack-infra/project-config: Unlink monasca gate jobs from neutron tests
fungihrm, tarballs will be slightly racy... i need to delete the a/aaaa records since cnames can't coexist with them16:40
patrickeasthey, anyone have any tips of troubleshooting never finishing on subnodes with devstack-gate? it looks like is running since that txt file exists, but its empty16:41
*** jlanoux has joined #openstack-infra16:41
*** Kiall has joined #openstack-infra16:41
*** amoralej is now known as amoralej|off16:41
patrickeasti can see running if i look at ps aux16:41
patrickeastbut dunno how to get its output to see whats going on16:41
*** pfallenop has joined #openstack-infra16:43
fungipleia2: will need to be changed at the same time we adjust dns for static.o.o since (for similar reasons) you can't cname the soa of a zone16:44
*** Apoorva has quit IRC16:44
pleia2fungi: ok, so just dropping the ttl for now?16:44
*** geekinutah is now known as nerdonmountain16:44
fungipleia2: yep. also is in that same boat16:44
*** Apoorva has joined #openstack-infra16:44
pleia2ah yes, I missed the serveraliases =>16:45
pleia2we also have serveraliases => ['*'],16:45
pleia2but I suspect that's just www.16:45
*** sputnik13 has joined #openstack-infra16:45
fungipleia2: yep, and is already a cname to static.openstack.org16:46
*** javeriak has joined #openstack-infra16:46
*** ccamacho|lunch has quit IRC16:46
fungiokay ttls for and records are 5 minutes now as well16:46
*** jamesmcarthur has quit IRC16:46
*** pfallenop has quit IRC16:48
*** jsavak has quit IRC16:49
*** hongbin_ has quit IRC16:49
*** salv-orlando has joined #openstack-infra16:49
*** jsavak has joined #openstack-infra16:49
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI
*** hongbin_ has joined #openstack-infra16:50 is an 8gb flavor right now. looks like it could get by reasonably on a lot less but its bandwidth utilization spikes up from time to time so i'd worry about possibly hitting a lower bw cap if we did16:50
*** kzaitsev_mb has quit IRC16:50
fungilooks like we don't need to size _up_ though so i say we stick with 8gb16:51
clarkbI have confirmed that all the restarts completed using
clarkbfungi: ++ especially for the bw16:51
clarkbbw gets halved as you go to each smaller flavor iir16:51
*** acoles is now known as acoles_16:52
*** ddieterly is now known as ddieterly[away]16:52
fungithe cpu/io/buffers are high from time to time, but that's because we've still not successfully disabled the mlocate cronjob16:53
*** bryan_att has quit IRC16:53
*** _nadya_ has quit IRC16:53
*** bin_ has quit IRC16:54
* fungi shakes fist at mlocate wanting to reindex terabytes of log data16:54
*** asettle has joined #openstack-infra16:54
*** pfallenop has joined #openstack-infra16:55
*** pfallenop has joined #openstack-infra16:55
*** salv-orlando has quit IRC16:56
jeblairclarkb: i wonder if mordred's async change to ansible would make that work16:56
openstackgerritgreghaynes proposed openstack/diskimage-builder: Spec for adding diskimage_builder module in-chroot
fungido we want to use or or or a new etherpad for this window?16:56
jeblairmordred: btw, what's the status of that?  did it get made into a pr?  did it merge?  will it be in next release?16:56
*** tonytan4ever has quit IRC16:57
*** tosky has quit IRC16:57
*** ddieterly[away] is now known as ddieterly16:58
*** _nadya_ has joined #openstack-infra16:58
*** sambetts is now known as sambetts|afk17:00
clarkbjeblair: it might, but its also unnecessary here17:00
clarkbsince we do one host at a time, keeping an ssh connection open is cheap17:01
jeblairclarkb: i thought we liked async for these things because we ran into ssh timeouts, or increased the risk of the connection dropping for some reason?17:02
*** PollyZ has quit IRC17:02
*** piet has joined #openstack-infra17:02
*** hongbin_ has quit IRC17:02
mordredjeblair: I do not think my patch would help with this - but we could potentially also do a patch to retry on spurious connection failures17:03
clarkbjeblair: my experience is the opposite, I wasn't really around when you all decided to switch to async wholesale17:03
clarkbjeblair: asnyc leads to consistent and frequent ssh errors between puppetmaster and the zuul launchers17:04
clarkbno asnyc worked 100% of the time in my small sample17:04
mordredI believe that async _should_ work between puppetmaster and the zuul launchers, and we should fix it if it doesn't. also, wait_for, since it has a timeout, should not need async.17:05
mordredit was my understanding that our use of async in z-l was to make timeouts work17:05
*** jsavak has quit IRC17:05
clarkbah ok that makes sense17:05
jeblairi think we are talking about 5 different things here17:05
*** hongbin_ has joined #openstack-infra17:06
*** jsavak has joined #openstack-infra17:06
jeblairzuul-launcher's use of async was my choice, and yes, it is the only ansible-native way of doing timeouts that i know of and that's why i chose it17:06
*** javeriak has quit IRC17:06
jeblairi did not choose any other use of async and can't speak as to why, but i head heard other people talk about ssh timeouts in the context of puppetmaster run-all17:07
clarkbjeblair: I think other people have begun to cargo cult your use of async even when it is not necessary17:07
clarkbunforutnately, this has led to reliability issues, possibly related to the same ones that zuul ran into and is trying to address with patches to ansible17:07
mordredright. it's actually a different but potentially related reliability issue17:08
mordredI'm looking right now to see if we can make a follow up that would handle connection issues17:08
mordredso that if we do need/want to use async for various things, we do not avoid it because of bugs17:08
mordredclarkb: any chance you have any captured errors/tracebacks from when it was giving you ssh problems?17:09
jeblairclarkb, mordred: i also used async in playbooks/restart_jjb for the same reason -- to get a timeout17:09
Shrewsmordred: i get the feeling we're going to end up rewriting most of the async code at some point17:09
mordredShrews: yah17:09
jeblairclarkb, mordred: and
clarkbmordred: I have already closed that screen session but I pasted the relevant bit of the error in here, let me find it in scrollback17:10
*** jsavak has quit IRC17:10
*** hongbin_ has quit IRC17:10
clarkbmordred: fatal: []: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."}17:10
mordredShrews: we may also want to chat with ansible core about playbook timeouts in general17:11
jeblairclarkb: have you spoken with pabelanger about this?17:11
Shrewsmordred: agreed. fest is a good time for that17:11
mordredjeblair: restart_jjb was for re-running jjb right?17:11
mordredShrews: agreed17:11
*** hongbin_ has joined #openstack-infra17:12
clarkbjeblair: yes he indicated I should run the playbook from my laptop, instead I disabled async and it just worked17:12
*** electrofelix has quit IRC17:12
jeblairmordred: typo: playbooks/run_jjb  and yes.17:12
*** hashar has joined #openstack-infra17:12
clarkbsince wait_for is used and has built in timeouts the async is not necessary17:12
clarkband will either succeed to ssh once then wait or fail and die eraly instead of failing an hour into an async poll17:13
jeblairclarkb: makes sense.  just thought it would be useful to find out what prompted him to use async.17:13
*** hashar is now known as hasharAway17:13
jeblairclarkb: but today is canada day.  so maybe we can follow up later.17:13
*** mhickey has quit IRC17:13
mordredjeblair: isn't every day canada day?17:14
Shrewso/` Oh, Canada o/`17:14
*** adrian_otto has joined #openstack-infra17:14
jeblairclarkb, mordred, Shrews: if we don't like async for zuul-launcher, we could have z-l pgkill the group (that is our fallback anyway -- i just wanted to try to do the right ansible-ish thing)17:14
mordredjeblair: I think async for zuul-launcher is correct usage17:15
clarkbI think it makes sense for zuul launcher17:15
*** tqtran has joined #openstack-infra17:15
clarkbparticularly since one of jenkins big scale issues was maintaining all of those ssh connections at once17:15
Shrewsasync just needs some polish, but is the right way17:15
jeblair(but also, i worry about timeouts on 'idle' ssh connections)17:15
mordredfor the restart playbook, wrapping the wait_for timeout in a second timeout just seems like overkill17:16
*** markusry has joined #openstack-infra17:16
jeblair(that's handwavey -- it's just something to potentially understand)17:16
mordredjeblair: totally... although I believe if wait_for can't deal with 'idle' connections that would also be a bug we should fix for sure :)17:16
jeblairyeah, i think clarkb's thing is good, assuming no ssh issues.  but maybe those were imagined or otherwise don't exist.17:16
openstackgerritDiana Whitten proposed openstack-infra/vinz: Vinz, the web app
mordredwoah. vinz changes17:17
Shrewsmordred: it would be kinda cool if timeouts were supported at the 'block' level17:17
mordredShrews: yup17:17
jeblairShrews: ++17:17
AJaegerteam, kite and python-kiteclient is dead - but there's a second +2A missing on to fully retire python-kiteclient. Is that something an infra-root should do - or should we ping the ex-kite folks again?17:18
openstackgerritDiana Whitten proposed openstack-infra/vinz: Vinz, the web app
*** ddieterly is now known as ddieterly[away]17:19
pabelangerohai! I opted for using async on zuul-launcher graceful restarts because I didn't think a single SSH connection idling for 3h10mins would be reliable. But agree the problem clarkb had was something I didn't initially think about17:19
pabelangerit would be great if async could simply ignore SSH failures and keep try connections until the timeout value was reached17:20
pabelangerrather then bailing on SSH connection failures17:20
clarkbpabelanger: or some total number of consecutive fails17:20
pabelangerclarkb: ya, that would be great too17:22
openstackgerritMerged openstack-infra/project-config: Update eavesdrop site name
*** hongbin_ has quit IRC17:23
jeblairclarkb, pabelanger: cool -- well, we have data for one side of the question.  i guess we'll see how reliable the really long ssh connection is now.  :)17:23
jeblairpabelanger: and shouldn't you be canadaing?17:24
*** javeriak has joined #openstack-infra17:24
pabelangerYa, I'm happy to try it without async17:24
pabelangersee what results we get17:24
*** salv-orl_ has joined #openstack-infra17:25
pabelangerjeblair: I have been! kids are down for naps and noticed IRC blinking at mt17:25
*** hongbin_ has joined #openstack-infra17:25
pabelangerwas about to drop off again to fetch a beer17:26
pabelangergood luck on the upgrades today17:26
jeblairpabelanger: thanks, and good luck on the beer!17:26
*** rkukura has quit IRC17:26
*** karthik__ has quit IRC17:26
clarkbpabelanger: does canada have a beer called "Canada"?17:26
openstackgerritMonty Taylor proposed openstack-infra/vinz: Update to use pbr and other infra standards
clarkbI recommend drinking beers named after your country on country day17:27
*** salv-orl_ has quit IRC17:27
mordrednibalizer, hurgleburgler: ^^ I was going to leave a -1 comment on some of the setuptools stuff in the previous patch, but then decided that would be rude and would be easier for me to just make a patch17:27
*** akshai has quit IRC17:28
mordredclarkb: eww. does that I mean I need to drink America on Monday?17:28
mordredthat can't possibly have good results17:28
clarkbmordred: I intend on drinking at least one can17:28
clarkbfor the lols17:28
clarkbmordred: also you are in texas which means you can only get America and Shiner neither of which I would drink in large quantities17:29
cineramalone star is the national beer of texas17:29
cinerama(it says so on the tin)17:29
clarkbclearly I am biased but still not anything I would drink in large quantities :)17:29
*** akshai has joined #openstack-infra17:29
clarkbin other news Gerrit GCing has picked up quite a bit recently, we should consider a restart17:30
clarkbmaybe when zuul is off?17:30
mordredclarkb: agree re: restart17:30
clarkbmordred: if you can find it Fremont Summer Ale out of seattle is great drink in quantity beer17:30
*** sdake has joined #openstack-infra17:30
mordredclarkb: however, I disagree re: shiner17:30
*** shashank_hegde has joined #openstack-infra17:30
persiacinerama: Yes, but one only drinks that on 2nd March17:31
*** PollyZ has joined #openstack-infra17:31
jeblairi wonder if i can get shiner around here17:31
mordredjeblair: I believe we export it now17:31
jeblair(not hipster enough?  too hipster?  i don't know!)17:31
mordredand I know I've had shiner in Palo Alto before17:31
AJaegerjeblair: there're two changes by ianw in project-config related to zuul-worker: (Don't use devuser for zuul-worker) and (Specify the ZUUL_USER_SSH_PUBLIC_KEY) - could you and others review them, please?17:31
clarkbI can get it at fred meyet17:31
mordredthat does not answer your questoin though17:31
nibalizermordred: thanks :)17:32
jeblairAJaeger: we're talking about *beer*!  ;)17:32
AJaegerjeblair: yeah, I know ;)17:32
*** sdake_ has joined #openstack-infra17:32
AJaegerjeblair: sorry, for having interrupted such an essential topic17:33
*** hongbin_ has quit IRC17:34
*** jamesmcarthur has joined #openstack-infra17:34
* AJaeger hides in the corner - and considers what kind of beer is in his cellar. Definitely neither America nor Germany ;)17:34
*** markus__ has joined #openstack-infra17:34
*** jsavak has joined #openstack-infra17:35
*** sdake has quit IRC17:35
*** fawadkhaliq has joined #openstack-infra17:35
jeblairoh neat, that's how you get data into dib17:35
TheJuliaIs gerrit throwing proxy errors for every other click for anyone else?17:35
*** markusry has quit IRC17:36
*** erlon has quit IRC17:36
greghaynesYep, that is the whole reason the extra-data.d phase exists - to copy data in to the image17:37
jeblairAJaeger: +1 from me -- i'd like to have + votes from greghaynes and pabelanger.17:37
* greghaynes adds to review queue17:37
hurgleburglermordred that patch is still very much in flux, so any suggestions that you want to make would be great17:37
AJaegerthanks, jeblair.17:38
jeblairTheJulia: clarkb mentioned increased GC activity which may be slowing it enough to cause what you report; he suggested maybe we should restart it in a few hours...17:38
mordredhurgleburgler: cool - I'm thrilled you're hacking on t17:38
AJaegerTheJulia: happens for me right now quite often as well - just not every other click ;)17:39
*** fawadkhaliq has quit IRC17:39
AJaegerjeblair, clarkb : Yeah, seems to be time again ;(17:39
TheJuliaAJaeger: okay, maybe once every three clicks :)17:40
*** markus__ has quit IRC17:40
*** markusry has joined #openstack-infra17:40
fungiyeah, we've got a maintenance window coming up at 20:00 utc anyway, so we'll kive it a kick while zuul et al are down17:42
*** mriedem has joined #openstack-infra17:43
*** signed8bit is now known as signed8bit_Zzz17:43
*** kzaitsev_mb has joined #openstack-infra17:44
*** jamesmcarthur has quit IRC17:44
*** jed56 has quit IRC17:45
*** dtardivel has quit IRC17:47
*** sdague has quit IRC17:47
*** kzaitsev_mb has quit IRC17:49
*** watersoul has joined #openstack-infra17:51
*** baoli has joined #openstack-infra17:51
*** baoli has quit IRC17:53
*** baoli has joined #openstack-infra17:53
*** rbrndt has quit IRC17:53
*** pradk has quit IRC17:54
*** pradk_ is now known as pradk17:54
*** sputnik13 has quit IRC17:54
*** sarob has joined #openstack-infra17:55
*** sarob has quit IRC17:55
*** signed8bit_Zzz is now known as signed8bit17:55
*** sarob has joined #openstack-infra17:55
*** salv-orlando has joined #openstack-infra17:58
*** trown is now known as trown|lunch17:59
*** asettle has quit IRC18:00
*** daemontool has quit IRC18:00
*** sdague has joined #openstack-infra18:02
*** bmjen has joined #openstack-infra18:03
notmorganoh look it's a hurgleburgler18:05
*** chem` has joined #openstack-infra18:05
*** salv-orlando has quit IRC18:06
notmorganmorning.. or afternoon...or whatever everyone! :)18:06
hurgleburglerAllo!  (◠‿◠✿)ノ18:06
*** chem has quit IRC18:07
*** jsavak has quit IRC18:08
*** baoli has quit IRC18:08
*** baoli has joined #openstack-infra18:09
*** sputnik13 has joined #openstack-infra18:09
*** jsavak has joined #openstack-infra18:09
*** akshai has quit IRC18:11
*** _nadya_ has quit IRC18:15
*** mgould is now known as mgould|afk18:15
*** _nadya_ has joined #openstack-infra18:17
*** Sukhdev has joined #openstack-infra18:17
EmilienMis it me or gerrit is super slow?18:18
*** rbrndt has joined #openstack-infra18:18
*** baoli has quit IRC18:18
*** baoli has joined #openstack-infra18:18
*** pvaneck has joined #openstack-infra18:19
*** ddieterly[away] has quit IRC18:19
*** baoli has quit IRC18:20
*** akshai has joined #openstack-infra18:20
*** baoli has joined #openstack-infra18:21
*** inc0 has quit IRC18:21
pleia2looks like the post job to update specs.o.o didn't run/work for the wiki spec yesterday, was that a known issue? I could use some help fixing the situation18:22
EmilienMgot 502 errors on gerrit18:22
EmilienMshould we restart it?18:23
pleia2EmilienM: it's been mentioned, the maintenance window is coming up in 90 minutes anyway, so it doesn't make sense to restart early18:24
pleia2EmilienM: go enjoy Canada Day :D18:24
EmilienMpleia2: how do you know? :)18:24
EmilienMpleia2: but yeah, I should leave18:25
pleia2my calendar told me (and I have a sister lives in Alberta)18:25
*** Liuqing has quit IRC18:26
sdagueI would second that gerrit is the sad right now18:27
*** pblaho has quit IRC18:27
notmorganpleia2: i like using simple presnet tense instead of present progressive when present progressive is appropriate18:28
*** jcoufal has quit IRC18:28
*** inc0 has joined #openstack-infra18:28
notmorganpleia2: :P18:28
pleia2notmorgan: writing is hard18:28
fungipleia2: it probably raced the post job for the other spec i approved at the same time18:29
fungithat's the usual cause18:29
pleia2fungi: ah, makes sense18:29
notmorganpleia2: right?!18:29
fungipleia2: if so, the next spec change which merges should solve it18:29
notmorganpleia2: i also realize... wow, i spent too much time in english classes / writing in college :P18:29
pleia2notmorgan: haah18:29
sdaguepleia2: except, the current 1 in 4 502 error rate on gerrit is making getting reviews done before the outage... difficult18:30
*** jlanoux has quit IRC18:30
*** bmjen has quit IRC18:30
notmorgansdague: yay 502s :(18:30
*** amrith has quit IRC18:30
mtreinishoh, that explains why my gertty sync is still going. I thought it was just something on my end18:31
notmorgani mean.. the opposite of that.18:31
*** sputnik13 has quit IRC18:31
*** amrith has joined #openstack-infra18:31
*** hurgleburgler has quit IRC18:31
*** salv-orlando has joined #openstack-infra18:32
*** bmjen has joined #openstack-infra18:34
*** xyang1 has quit IRC18:35
*** hurgleburgler has joined #openstack-infra18:36
*** _nadya_ has quit IRC18:38
*** ayoung has quit IRC18:39
*** rossella_s has quit IRC18:39
*** rossella_s has joined #openstack-infra18:39
*** salv-orlando has quit IRC18:40
*** xyang1 has joined #openstack-infra18:41
*** PollyZ has quit IRC18:41
funginibalizer: it looks like hasn't garnered much interest yet, though has suggestions from clarkb and ianw so i didn't approve it yesterday18:42
*** PollyZ has joined #openstack-infra18:42
nibalizerfungi: ok18:42
*** Sukhdev has quit IRC18:45
*** PollyZ has quit IRC18:45
*** karthik__ has joined #openstack-infra18:45
*** thiagop-afk is now known as TheJulia_18:46
*** TheJulia_ is now known as thiagop18:46
*** PollyZ has joined #openstack-infra18:47
*** HeOS has quit IRC18:53
mriedemwhen is the gerrit outage?18:53
mriedemoh nvm18:53
*** tpsilva has quit IRC18:58
*** nwkarsten has quit IRC19:04
*** PollyZ_ has joined #openstack-infra19:04
*** chem` has quit IRC19:05
*** PollyZ has quit IRC19:07
*** roxanagh_ has quit IRC19:09
*** PollyZ_ has quit IRC19:09
*** karthik__ has quit IRC19:09
*** PollyZ has joined #openstack-infra19:09
*** ddieterly has joined #openstack-infra19:09
*** karthik__ has joined #openstack-infra19:12
*** Sukhdev has joined #openstack-infra19:17
*** cody-somerville has joined #openstack-infra19:18
fungii'm accumulating server upgrade notes for zuul and static at
fungii've already got the replacement servers launched and identified there19:18
*** Sukhdev has quit IRC19:23
*** sputnik13 has joined #openstack-infra19:23
*** trown|lunch is now known as trown19:23
*** adrian_otto has quit IRC19:23
*** Apoorva has quit IRC19:24
*** sarob has quit IRC19:29
*** Ravikiran_K has quit IRC19:31
*** shardy is now known as shardy_afk19:32
*** harlowja_at_home has joined #openstack-infra19:33
*** yolanda has quit IRC19:34
openstackgerritAndreas Florath proposed openstack/diskimage-builder: package-installs: add list to arch and "not-arch" list
zigopabelanger: Hi there! Any progress on the reprepro job?19:35
zigoOr any hints to us, on how you want it made?19:36
zigoAny of the 2 would work ...19:36
*** pt_15 has quit IRC19:36
*** zz_zz_ja has quit IRC19:36
openstackgerritClark Boylan proposed openstack-infra/zuul: Make job registration with labels optional
clarkbjeblair: ^ that took entirely too long to get pushed today19:38
clarkbbut there we have it19:38
clarkbI am going to try sedning out the python35 email before the downtime stuff to19:39
clarkbwe are adding ~240 jobs for python35 which is far less than we hvae removed so far \o/19:39
*** jswarren_ has joined #openstack-infra19:39
*** zz_ja has joined #openstack-infra19:40
*** zz_dimtruck is now known as dimtruck19:40
*** bswartz has quit IRC19:40
*** xyang1 has quit IRC19:41
*** hrubi has quit IRC19:41
*** coreycb has quit IRC19:41
*** spiffxp has quit IRC19:41
*** boris-42 has quit IRC19:41
*** laurelm has quit IRC19:41
*** kambiz has quit IRC19:41
*** xyang1 has joined #openstack-infra19:41
*** spiffxp has joined #openstack-infra19:41
*** stevebaker has quit IRC19:41
*** tristanC has quit IRC19:41
*** andymccr has quit IRC19:41
*** SpamapS has quit IRC19:41
*** mgould|afk has quit IRC19:41
*** elarson has quit IRC19:41
*** itsuugo has quit IRC19:41
*** furlongm_ has joined #openstack-infra19:42
*** eharney has quit IRC19:42
*** yamahata has quit IRC19:42
*** liusheng has quit IRC19:42
*** gsilvis has quit IRC19:42
*** notmyname has quit IRC19:42
*** j_king has quit IRC19:42
*** jswarren has quit IRC19:42
*** fungi_lounge has quit IRC19:42
*** jtomasek has quit IRC19:42
*** furlongm has quit IRC19:42
*** thingee has quit IRC19:42
*** _alastor_ has quit IRC19:42
*** eglute has quit IRC19:42
*** jgriffith has quit IRC19:42
*** kevinbenton has quit IRC19:42
*** zeus has quit IRC19:42
*** uberjay has quit IRC19:42
*** lyz has quit IRC19:42
*** salv-orlando has joined #openstack-infra19:42
*** sflanigan has quit IRC19:42
*** gabriel-bezerra has quit IRC19:42
*** liusheng has joined #openstack-infra19:42
*** tristanC has joined #openstack-infra19:43
*** mgould|afk has joined #openstack-infra19:43
*** _alastor_ has joined #openstack-infra19:44
*** j_king has joined #openstack-infra19:44
*** boris-42 has joined #openstack-infra19:44
*** gsilvis has joined #openstack-infra19:44
AJaegerclarkb: and are further job removals19:44
clarkbAJaeger: noted19:44
*** kevinbenton has joined #openstack-infra19:44
fungizigo: it's canada day, so pabelanger is hopefully enjoying some time with his family. as to your question, i don't know the status on those at the moment though we're about to start a maintenance window for some server upgrades so i can't research it right now19:44
zigofungi: Thanks for the update.19:45
zigofungi: FYI, mordred wrote that pabelanger would work on it, which is why I asked.19:45
fungipleia2: should we irc a 10-minute warning?19:46
*** ayoung has joined #openstack-infra19:46
*** stevebaker has joined #openstack-infra19:46
*** itsuugo has joined #openstack-infra19:47
*** zeus has joined #openstack-infra19:47
*** elarson has joined #openstack-infra19:47
*** jgriffith has joined #openstack-infra19:47
*** notmyname has joined #openstack-infra19:47
*** andymccr has joined #openstack-infra19:47
*** coreycb has joined #openstack-infra19:47
*** kambiz has joined #openstack-infra19:47
*** eglute has joined #openstack-infra19:47
*** laurelm has joined #openstack-infra19:47
*** zeus is now known as Guest3196119:47
*** hrubi has joined #openstack-infra19:48
*** uberjay has joined #openstack-infra19:48
*** lyz has joined #openstack-infra19:48
pleia2fungi: yeah, I can do19:48
clarkbwould people here like to proof my python35 email? otherwise I am going to send it nowish19:49
*** sdake_ has quit IRC19:50
fungii'm in favor of you just sending it19:50
karthik__ianw_pto, AJaeger: I have made some changes to the devstack-gate for Openstack/cinder... Please will you be able to review it in your free time?19:50
*** salv-orlando has quit IRC19:50
mordredclarkb: I support the words you have in your brian19:51
mordrednot in your brian - that's weird19:51
clarkbI didn't know I had a Brian19:51
*** Nikolay_St has joined #openstack-infra19:51
*** Guest31961 is now known as zeus19:51
*** zeus has joined #openstack-infra19:51
pleia2#status notice The infrastructure team is taking Gerrit offline for maintenance beginning shortly after 20:00 UTC to upgrade the Zuul and servers. We aim to have it back online around 22:00 UTC.19:52
openstackstatuspleia2: sending notice19:52
clarkbok mail sent19:52
fungianyone helping on the maintenance, feel free to brainstorm things i've forgotten at
-openstackstatus- NOTICE: The infrastructure team is taking Gerrit offline for maintenance beginning shortly after 20:00 UTC to upgrade the Zuul and servers. We aim to have it back online around 22:00 UTC.19:53
fungii figure at the start of the window we'll stop zuul and gerrit, halt and then immediately update dns records19:53
* clarkb +2's AJaeger's change but doesn't approve due to ^19:53
*** SpamapS has joined #openstack-infra19:54
*** pt_15 has joined #openstack-infra19:54
*** Daisy has joined #openstack-infra19:54
*** sflanigan has joined #openstack-infra19:55
*** sflanigan has joined #openstack-infra19:55
openstackstatuspleia2: finished sending notice19:55
*** eharney has joined #openstack-infra19:55
clarkbI am impressed I think we removed about 1300 jobs in this job cleanup AJaeger has been doing19:56
*** jtomasek has joined #openstack-infra19:56
*** yamahata has joined #openstack-infra19:56
*** gabriel-bezerra has joined #openstack-infra19:56
jeblairclarkb, cinerama, mordred: look what i found!
* jeblair is prepared for maintenance!19:57
fungibest maintenance prep EVAR19:57
* fungi notes they have that at the grocery here too. probably just as terrible19:58
*** asettle has joined #openstack-infra19:58
*** asettle has quit IRC19:58
fungianybody want to claim steps on the plan at ?19:58
fungialso please add things i've forgotten if you know them ;)19:59
*** Nikolay_St has quit IRC19:59
*** Daisy has quit IRC19:59
*** harlowja_at_home has quit IRC19:59
*** piet has quit IRC19:59
*** piet has joined #openstack-infra20:00
jeblairfungi: are we halting static to try to avoid the interminable detaching state?20:00
fungijeblair: YEP20:00
*** inc0 has quit IRC20:00
fungimy clock says we're 60 seconds into the window now, btw. so free to start any time we like20:00
*** HeOS has joined #openstack-infra20:01
jeblairi claimed #2, so i'll start on that :)20:01
fungipleia2: want to do the status alert?20:01
pleia2right, on it20:02
jeblairthat horizon test looks like it's not particularly about to wrap up, so i will hard stop things now.20:02
pleia2#status alert Gerrit is offline for maintenance until approximately 22:00 UTC20:02
openstackstatuspleia2: sending alert20:02
fungii notice i forgot to add the reverse dns records on the replacement servers, so i'll add a step for me to fix that too20:03
*** ddieterly is now known as ddieterly[away]20:03
jeblairgerrit is stopping, i have saved zuul queues20:03
fungii'll halt static.o.o now20:03
* clarkb is around and can help as necessary20:03
jeblairgerrit and zuul have stopped20:03
fungiand working on dns updates now20:04
jeblairi'm going to stop launchers and mergers now20:04
-openstackstatus- NOTICE: Gerrit is offline for maintenance until approximately 22:00 UTC20:05
*** nwkarsten has joined #openstack-infra20:05
*** ChanServ changes topic to "Gerrit is offline for maintenance until approximately 22:00 UTC"20:05
*** weshay has quit IRC20:06
pleia2I can do 8, but I'll need some direction20:06
clarkbpleia2: I can walk you through it its basically ssh/ansible for loop to service iptables-persistent restart20:07
*** kgiusti has left #openstack-infra20:07
clarkbpleia2: you can run iptables -L -n before and after to see the ip addrs update20:07
* clarkb will work on a list of those hosts on the etherpad20:07
*** karthi___ has joined #openstack-infra20:07
*** Apoorva has joined #openstack-infra20:08
openstackstatuspleia2: finished sending alert20:08
pleia2clarkb: ah, easy enough20:08
*** karthik__ has quit IRC20:08
fungiforward dns done now (including and working on reverse dns20:09
jeblairclarkb, pleia2: i think we don't have many iptables to restart20:10
*** nwkarsten has quit IRC20:10
*** roxanagh_ has joined #openstack-infra20:10
clarkbjeblair: oh right because things connect to zuul not the other way around20:10
fungiyeah, the only one i could think of was graphite20:10
jeblairclarkb, pleia2: yeah -- i think it's only graphite?20:10
clarkbthe things I have listed all connect to zuul so should be fine20:10
pleia2even better20:10
*** achanda has joined #openstack-infra20:10
jeblairyou can totally still use ansible tho :)20:11
*** weshay has joined #openstack-infra20:11
*** _nadya_ has joined #openstack-infra20:11
*** Swami has quit IRC20:12
fungireverse dns done20:12
*** Sukhdev has joined #openstack-infra20:12
pleia2if it's just the one I might just play service restart20:12
fungii'm working on the static.o.o volumes now20:12
*** javeriak_ has joined #openstack-infra20:12
jeblairhrm.  we have no logs on the zuul mergers20:12
clarkbthey don't even run systmed/journald yet20:12
jeblairsorry, i meant service logs from z-m itself20:13
*** markusry has quit IRC20:13
*** javeriak has quit IRC20:13
*** EricGonczer_ has joined #openstack-infra20:14
*** PollyZ has quit IRC20:14
rm_workah rofl came to ask if anyone else was having gerrit issues. saw topic T_T20:15
achandagetting 503 from gerrit20:15
fungiachanda: see /topic20:15
jeblairrm_work: thanks!  it's been 5 years we've been doing this -- *someone* was bound to actually read the topic someday  ;)20:15
achandafungi, thanks, my bad!20:15
*** PollyZ has joined #openstack-infra20:15
rm_workhappened EXACTLY when i a was futzing with vpn / bastion / routes20:16
rm_worksuch timing20:16
jeblairrm_work: we try20:16
clarkbjeblair: for step 9 gear should reconnect on its own? maybe?20:16
fungiokay, all volumes detached successfully from old static.o.o. definitely should just halt servers before doing volume detach if we have the opportunity20:16
fungithis goes so much faster20:17
*** bswartz has joined #openstack-infra20:17
jeblairokay, i dunno what's up with the zm logs -- i will carefully restart them when we are done to try to further diagnose that20:17
*** salv-orlando has joined #openstack-infra20:17
fungivolume add is going a bit slower than volume remove. i suspect nova waits for the instance to settle or something between successive attachments20:18
jeblairclarkb: i think/hope it should do it on its own20:18
jeblairclarkb: it at least performs a new getaddrinfo every time it tries to connect20:18
jeblairclarkb: so it would have to be cached lower than that layer to fail20:18
fungii guess just wait on #9 and see if any of those don't dtrt20:19
jeblairthe only thing left trying to connect to zuul's geard is nodepool, right?  (i stopped the launchers and mergers -- mostly just because the mergers rarely get restarted)20:20
clarkbjeblair: yup should be20:21
fungiyay, all volumes (supposedly) attached successfully to the new static.o.o. i'm checking its work now ;)20:21
*** EricGonczer_ has quit IRC20:21
jeblairclarkb: does nodepool need a restart for any other reasons?  (pick up any recent changes?)20:21
fungigrr... is still showing "available"20:22
clarkbjeblair: not that I am aware of20:22
jeblairok, then i guess we can leave it be and see what happens20:22
jeblairi'm also going to take the opportunity to delete zl git dirs20:23
*** EricGonczer_ has joined #openstack-infra20:23
clarkbjeblair: zm too maybe?20:23
clarkbor is that what yo umeant?20:23
jeblairclarkb: that is indeed what i meant :)20:23
fungioh [expletive deleted]20:23
fungiwe have a configdrive20:24
jeblairso we lost a slot?20:24
fungi... yeah20:24
clarkbwe shouldn't need a config drive there20:24
jeblaircan we return the configdrive unused?20:24
jeblairi would like a refund20:24
clarkbjeblair: not without making a new instance I think20:24
jeblairclarkb: okay, but we can exchange our instance for one without a configdrive?20:25
clarkbyes we should be able to do that, I do not know why we are config driving now on those hosts20:25
fungii can solve this by detaching all these, attaching them back to the old server, starting it back up, using vgreduce to shuffle any used extents off main14, then detach them all again and attach the used 13 back to the new server20:25
fungibut that will eat up a lot of time20:25
jeblairfungi: i agree that's a solution -- clarkb's may be faster20:25
jeblairShrews, mordred: ^ possibly of interest20:25
fungiyes, i'd much prefer clarkb's solution first20:25
*** Swami has joined #openstack-infra20:26
* clarkb pulls up the launcer to see if it is configurable there20:26
*** salv-orlando has quit IRC20:26
clarkbyou might have to checkout an old launch node script20:26
*** ddieterly[away] is now known as ddieterly20:26
fungiis configdrive irreversible after the instance is created?20:26
jeblairfungi: i think that's what clarkb was telling me20:26
clarkbfungi: yes I don't think there is a way of telling nova to remove it20:26
fungias in do i need to detach all these and replace this instance and update dns again?20:26
clarkbsdague: or mriedem might be able to confirm20:27
clarkbit is configurable in launch-node.py20:27
jeblairclarkb: agreed, and defaults to true20:27
*** woodster_ has joined #openstack-infra20:27
fungiin that case, not a whole lot faster than the vgreduce option, but maybe still superior since it won't lose us a terabyte20:27
clarkb--config-drive=False or similar to turn it off20:28
*** shardy_afk has quit IRC20:28
fungiokay, so any alternative proposals? shall i press forward replacing this instance?20:28
jeblairfungi: (doesn't the vgreduce require at least one full fsck?  -- i was imagining it to take hours)20:28
fungijeblair: no fsck required. it moves extents online20:28
fungior at least _can_ move extents while online20:28
jeblairand our volume is smaller than 14tb?20:28
jeblairour volume and our fs?20:29
clarkbI am investigating if there is a way to remove the config drive20:29
fungisorry, lvreduce first, then vgreduce20:29
mriedemconfig drive is static on the instance once you've created it20:29
fungibut both work online20:29
jeblairfungi: i'm still +1 on the new server20:29
mordredI'm also +1 on the new server20:29
fungiokay, i'll detach these and so that20:29
mordredand also wow what a 'fun' fail20:29
jeblairfungi: (right -- but there's an ext4 filesystem that's 14tb, isn't there?)20:29
jeblairmordred: do you have any idea why we defaulted that to 'true' in launch-node?20:29
fungijeblair: oh, grr. you're right, need to resize2fs. that does require a fsck20:30
mordredjeblair: habit from elsewhere. no reason we need to in launch-node20:30
fungianyway, the new static.o.o is halted and i'm booting a replacement while i detach these volumes20:31
mordredjeblair: in the general case, having configdrive=True means you can use glean instead of cloud-init which means you can use the keypair extension without needing to ever hit the scales-poorly metadata service20:31
jeblair-                        default=False)20:32
jeblair+                        default=True)20:32
mordredjeblair: but we have not gotten to booting from our own base images yet20:32
jeblairthat was in e14471dcc6daa4f33bba84ac1d3e7c091ada10ea with no explanation in the commit msg20:32
jeblairmordred: ok, good to know20:32
clarkbmriedem: ty for confirming20:32
jeblairmordred: so config_drive may be the future for us -- we may need to adapt to having one less cinder slot20:33
*** kzaitsev_mb has joined #openstack-infra20:33
fungier, --config-drive doesn't take a value, and --no-config-drive doesn't seem to work?20:33
mordredjeblair: yah - it was forward-looking, but we haven't caught up with it yet. obviously running out of volume mounts early due to config-drive will change the design of the future a bit20:33
mordredjeblair: yah20:33
jeblairfungi: haha!20:33
jeblairfungi: store_true with default=true is a bad idea20:33
jeblairmordred: ^ :)20:33
mordredjeblair: it's certainly something we can't avoid dealing with20:33
mordredjeblair: yay!20:33
fungiyep, it's hilarious. i'll clone this and patch it20:33
mordredfungi: sorry20:34
*** xyang1 has quit IRC20:34
funginah, it's fine. really is sorta funny ;)20:34
jeblairlike, maybe argparse should give you a heads-up on that one, huh? :)20:35
*** javeriak_ has quit IRC20:35
*** PollyZ has quit IRC20:35
jeblair"I noticed you really seem to want this value to be true..."20:35
fungi"--no-config-drive" is now working for me ;)20:35
*** karthikp_ has joined #openstack-infra20:37
*** karthi___ has quit IRC20:37
*** iurygregory has quit IRC20:38
fungihere's a new helpful nova error20:40
*** piet has quit IRC20:40
*** cody-somerville has quit IRC20:40
*** jsavak has quit IRC20:40
fungiCannot 'detach_volume' instance 78711ec1-78ce-41ce-b8dc-7d0c1fd3b7f1 while it is in task_state powering-off (HTTP 409) (Request-ID: req-96de3c63-093c-4943-951f-00387cb5184b)20:40
*** jsavak has joined #openstack-infra20:41
fungiapparently sometimes nova _doesn't_ like that?20:41
fungii got through the first 3 before nova started complaining on the rest20:41
fungiguess i need to boot it back up again20:41
mordredfungi: so - to recap - detaching volumes when the server is running sometimes does not work. detaching volumes when the server is halted sometimes does not work20:41
*** _nadya_ has quit IRC20:42
fungii think you can reduce that problem statement considerably ;)20:42
fungiyay nova!20:42
fungiCannot 'reboot' instance 78711ec1-78ce-41ce-b8dc-7d0c1fd3b7f1 while it is in task_state powering-off20:42
fungisdague: that ^ look in any way familiar?20:43
*** eharney has quit IRC20:45
*** Sukhdev has quit IRC20:46
mordredfungi: he's on vacation probaly. mriedem ^^ ?20:46
fungioh, he had popped in a few minutes ago20:47
mordredfungi: he's theoretically half-here20:47
fungianyway, i seem to be successfully detaching again, so whatever was going on was temporary20:47
mordredfungi: oh good20:47
mriedemi'm here20:47
mtreinishmordred: it's after 4pm on a friday, he's gone :)20:47
mriedempowering-off is a task state20:47
mriedemsdague is going camping for 2 weeks or something20:47
mriedemso yeah he's gone20:47
fungioh, awesome. he should definitely be camping and not in here20:48
* mriedem is writing the soul crushing blueprint freeze email to the dev list20:48
mordredmriedem: just rename whatever emotion you're feeling "happy"20:49
fungi"your ideas are sub-par and will not be included in the next release. drown your sorrows in booze"20:49
*** rfolco has quit IRC20:49
fungiawesome. all 14 volumes are back to available finally20:49
mordredfungi: I feel like you were just suggesting to me personally that it's time to start drinking20:49
mriedemmordred: i don't feel emotion20:49
mordredmriedem: cool. so if you redefine that, you can constantly be "happy"20:50
mriedem"your face is telling me that my face should be smiling"20:50
fungiupdating dns now20:51
fungi)remembered to set reverse dns on the new server)20:51
*** salv-orlando has joined #openstack-infra20:51
fungialso, no configdrive this time, i confirmed20:52
fungijust the rootfs and the ephemeral disk20:52
*** EricGonc_ has joined #openstack-infra20:52
*** EricGonczer_ has quit IRC20:55
fungia/aaaa records for all 3 domains updated to the new-new addresses for static.o.o now20:57
fungistarting to attach volumes20:57
*** jsavak has quit IRC20:58
*** asettle has joined #openstack-infra20:59
*** salv-orlando has quit IRC21:00
*** raildo is now known as raildo-afk21:01
*** Sukhdev has joined #openstack-infra21:01
fungifinally--all 14 volumes attached successfully and showing in use this time21:02
fungigonna see if it can assemble the vg now21:02
mordredfungi: woot!21:03
fungiyep, we have volumes docs-draft, logs, mirror, static and tarballs present21:04
fungiadding to fstab now21:04
*** aysyd has quit IRC21:07
jeblairi cleaned up all the git repos on the mergers21:07
jeblairso that should be faster when we restart21:07
*** Sukhdev has quit IRC21:08
*** Sukhdev_ has joined #openstack-infra21:08
fungioh, er, ew... need to chown all this too21:08
fungiuids/gids are not the same as on the old server21:09
clarkbthat could take days?21:09
*** akshai has quit IRC21:09
clarkbmight be easier to change the uids/gids21:09
*** mriedem is now known as mriedem_away21:10
fungiyeah, i'm thinking so21:10
*** sdake has joined #openstack-infra21:10
*** psilvad has quit IRC21:11
fungiwe have a bunch of different uids/gids showing up though, so i'm going to try and start the old server now and see what they were before21:11
jeblairinteresting -- jenkins should be the only interesting one, right?21:13
clarkbjeblair: and root but root should be the same :)21:13
fungii would think21:13
jeblairclarkb: yeah, but what about *systemd-root*?21:13
* jeblair ducks21:13
nibalizerjeblair: booooo21:13
*** EricGonc_ has quit IRC21:14
nibalizermaybe we should just be putting uid in the puppet resources for our users21:14
nibalizerthat would be one way to get consistency, might be scary to roll out though21:14
fungii think it's just called "lennart" (uid ∞)21:14
jeblairnibalizer: we do that for our user-users...  it would be hard to do that for our services though21:15
*** ldnunes has quit IRC21:15
jeblairnibalizer: (in that we can't guarantee as the authors of the zuul module that your zuul user should be $uid and won't conflict with some site-local thing)21:15
nibalizerjeblair: yea and you have things like the postgres package adding a postgres user of uid <next uid>21:16
jeblairi suppose we could make that a configurable input, via hiera...21:16
jeblairnibalizer: right21:16
nibalizerand hopefully we don't end up having an ldap for <20 users21:17
jeblairyeah, we thought about it for afs, but then figured easier to just set the uids21:17
jeblair(that's the main reason we have it for our human users)21:18
fungireviewday:x:1000:1000 jenkins:x:1005:1006 releasestatus:x:1006:1007 recheck:x:1007:100821:18
fungi1000:1000 is currently ubuntu:ubuntu so i'm cleaning it out of any group memberships and deleting its homedir21:19
mordredfungi: such a useful user21:20
*** asettle has quit IRC21:24
fungiokay, old uids/gids restored through the use of vipw/vigr and the same with -s21:25
pleia2nice work, fungi21:25
*** jsavak has joined #openstack-infra21:25
fungii'm going to attempt to start apache now21:25
pleia2we routinely forget about u/gids :\21:25
mordredpleia2: u/gids are silly21:26
mordredpleia2: we should get rid of user accounts and just have code21:27
pleia2they're something21:27
pleia2mordred: run everything as root!21:27
pleia2wait, no21:27
fungii've pulled up an old log successfully, browsed tarballs via https...21:27
fungianything anybody else wants to test out on the new static.o.o?21:28
clarkbdocs draft21:28
pleia2releases.o.o is forbidden21:28
pleia2hm, might be my problem21:29
fungii'm getting forbidden on a recent docs-draft from my browser history too21:29
fungiYou don't have permission to access /82/335582/1/check/gate-bindep-docs/e355ca4//doc/build/html/news.html on this server.21:29
jeblairAH01630: client denied by server configuration: /srv/static/releases/newton/schedule.html21:30
pleia2yeah, it's not just me21:30
jeblairapache config?21:30
fungithat's what i'm thinking21:30
jeblairthe 'allow from' thing maybe?21:30
fungiprobably apache 2.4isms21:30
fungiif so, we should be able to work through those with everything else back online again21:31
jeblair    Satisfy Any21:31
jeblairhrm, that's the new form, isn't it?21:31
jeblairoh no21:32
jeblair"Require all granted" is the thing we want21:32
jeblairand yeah, it's missing21:32
jeblairso ++ bring up and push fixing changes through21:32
*** jsavak has quit IRC21:32
*** jsavak has joined #openstack-infra21:33
jeblairare we ready for me to start starting zuul?21:33
fungiso we're up to the starting zuul step i guess"?21:33
fungii think so--go for it21:33
fungishould have had more than enough time for the most recent round of dns updates to propagate21:34
*** zzelle_ has joined #openstack-infra21:34
*** tonytan4ever has joined #openstack-infra21:35
fungiobviously we shouldn't reenqueue anything until gerrit is back online though21:37
jeblairfungi: yeah, also, i forgot to copy the queue files over, so it will take a little extra21:38
pleia2yeah, lmk when I should bring gerrit back21:38
*** gouthamr_ has joined #openstack-infra21:38
jeblairokay, zuul and all its components appear to be running21:39
jeblair(i just spent some extra time checking that out)21:39
jeblairit even seems to be okay with starting without being able to connect to gerrit, which is nice21:39
*** jsavak has quit IRC21:39
*** jsavak has joined #openstack-infra21:39
jeblairthe launchers have grabbed nodes from nodepool21:39
pleia2ok, graphite iptables now?21:39
fungiyeah could have really done the graphite iptables update any time21:40
pleia2done, gerrit?21:40
fungigood to know we can safely restart zuul with gerrit offline21:40
fungii wondered about that bit21:40
jeblairpleia2: go for it21:40
fungiyep, gerrit time!21:40
pleia2gerrit starting up21:40
pleia2Starting Gerrit Code Review: OK21:41
*** gouthamr has quit IRC21:41
* fungi is so glad we conservatively budgeted 2 hours for this maintenance, leaving us an hour to deal with unforeseen challenges21:41
pleia2hopefully it's just java-ing now21:41
jeblair2016-07-01 21:41:45,753 INFO zuul.source.Gerrit: Updating <Change 0x7fde8c6ac7d0 336719,1>21:41
fungiit reconnects QUICKLY!21:42
pleia2yep, there it goes, web UI is up21:42
fungiwho wants to hack on apache config fixes?21:42
jeblairand running jobs21:42
jeblairoh, heh21:42
jeblairwe forgot to copy the zuul time database over21:42
jeblairwe could probably do that even now and it would be okay21:43
jeblairor we could just leave it to rebuild21:43
jeblair(i vote leave it -- it will rebuild quickly)21:43
fungii have no opinion. i'm fine with it just regenerating that on the fly from fresh samples21:43
pleia2I can do the apache changes real quick21:43
fungipleia2: awesome. i'm happy to review/approve those21:44
jeblairooh, grafana is in xenial21:44
fungihrm, python-ironicclient change in the gate with a ton of POST_FAILUREs21:45
pleia2good grief, this apache template file21:45
clarkbI think we can just rebuild it21:46
fungisahara-tests change exhibiting similar issues21:46
*** watersoul_ has joined #openstack-infra21:47
*** lascii is now known as alaski21:47
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add Apache 2.4 syntax for static.o.o
fungiyeah, i think all completing jobs are generating POST_FAILURE results21:48
jeblairi'll track down their launcher logs for the cause21:48
*** baoli has quit IRC21:48
*** puiterwi1 has joined #openstack-infra21:48
*** furlongm has joined #openstack-infra21:48
*** javeriak has joined #openstack-infra21:48
fungii have a feeling there's something not quite right with log uploads21:48
*** gouthamr_ has quit IRC21:48
*** puiterwijk has quit IRC21:49
jeblair"rsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]"21:49
*** puiterwi1 is now known as puiterwijk21:49
*** ddieterly is now known as ddieterly[away]21:49
mordredhost key?21:50
fungirsync is definitely installed on static.o.o21:50
fungioh, do they have the old host key cached?21:50
fungido we puppet that?21:50
*** nibalize1 has joined #openstack-infra21:50
fungiit's not in hiera/group/zuul-merger.yaml21:51
*** Adri2000_ has joined #openstack-infra21:51
jeblairyes, they cache the keys21:51
fungimaybe it just caches them on first use?21:51
*** david-lyle has quit IRC21:51
fungii can copy the old host keys over to the new server21:51
jeblairi will remove /home/zuul/.ssh/known_hosts from all launchers21:51
*** tristanC_ has joined #openstack-infra21:51
fungior do you want to have them re-cache the new one?21:51
fungiokay, that's probably better21:51
jeblairlet's use the new one21:51
*** gsilvis_ has joined #openstack-infra21:51
fungiyeah, generated with stronger defaults and whatnot21:51
jeblairi could restart zuul and re-enqueue again21:52
fungiyeah, that's probably less painful than telling everyone to recheck all of these21:52
*** afazekas|real has quit IRC21:53
*** harlowja has quit IRC21:53
*** markmcclain has quit IRC21:53
*** sigmavirus24 has quit IRC21:53
*** coreyob_ has quit IRC21:53
*** afazekas has joined #openstack-infra21:53
*** hasharAway has quit IRC21:53
*** furlongm_ has quit IRC21:53
*** Adri2000 has quit IRC21:53
*** tristanC has quit IRC21:53
*** nibalizer has quit IRC21:53
*** sc` has quit IRC21:53
*** coreyob has joined #openstack-infra21:53
*** markmcclain has joined #openstack-infra21:53
*** javeriak has quit IRC21:53
*** harlowja has joined #openstack-infra21:53
*** watersoul has quit IRC21:53
openstackgerritlynxis lazus proposed openstack-infra/jenkins-job-builder: docker-build-publish: add registry + server support
*** hashar has joined #openstack-infra21:54
*** javeriak has joined #openstack-infra21:54
*** tlian has quit IRC21:55
*** tlian has joined #openstack-infra21:56
*** sigmavirus24 has joined #openstack-infra21:57
*** sc` has joined #openstack-infra21:57
jeblairzuul restarted and re-enqueing21:57
fungijeblair: thanks!21:57
*** gsilvis has quit IRC21:57
jeblairnp.  i'll push up my playbooks in a sec :)21:57
fungii'm watching closely to see if stuff uploads this time21:57
*** nibalize1 is now known as nibalizer21:58
*** jsavak has quit IRC21:58
*** salv-orlando has joined #openstack-infra21:58
openstackgerritlynxis lazus proposed openstack-infra/jenkins-job-builder: docker-build-publish: add registry + server support
*** jsavak has joined #openstack-infra21:59
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add playbooks to start/stop all of zuul
*** vhosakot has quit IRC21:59
jeblairthose act on all 17(!) zuul hosts :)21:59
fungi17 _for_now_ ;)22:00
fungii guess that has the added benefit of killing running jobs sooner so nodepool can reclaim the available capacity?22:00
*** salv-orlando has quit IRC22:00
jeblairyeah, that's why i did it22:01
*** tonytan4ever has quit IRC22:01
fungistill getting POST_FAILURE on jobs22:01
*** tonytan4ever has joined #openstack-infra22:01
fungiit did recreate /home/zuul/.ssh/known_hosts22:02
*** rfolco has joined #openstack-infra22:02
fungiand that looks like the correct ip address in its entry22:02
fungiso presumably now failing on something else22:03
fungilots of exceptions while sending job completion packet22:04
*** zeih has quit IRC22:05
fungioh, those are form 10 minutes ago22:05
fungiso probably associated with the restart22:05
jeblair"rsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226)22:05
jeblairagain a helpful message22:05
jeblairzuul@zl01:/var/log/zuul$ /usr/bin/ssh -i /var/lib/zuul/ssh/id_rsa -S none -o StrictHostKeyChecking=no jenkins@static.openstack.org22:06
jeblairPermission denied (publickey).22:06
*** javeriak_ has joined #openstack-infra22:06
*** nwkarsten has joined #openstack-infra22:06
*** javeriak has quit IRC22:07
*** zzelle_ has quit IRC22:07
jeblairthe jenkins authorized_keys files on old/new static servers seem to match22:07
fungithis one's my fault. i changed jenkins's uid/gid22:08
jeblairfungi: what did you miss?  (i'm still not seeing it)22:08
fungihrm, no22:09
*** karthikp_ has quit IRC22:10
jeblairoh there it is22:11
*** nwkarsten has quit IRC22:11
fungiokay, fixed22:11
jeblairfungi: /home/jenkins uid is 300022:11
fungiit was _both_ :/22:11
jeblairfungi: yep, lgtm22:11
jeblairssh access from zl01 works22:11
jeblairi will do the restart dance again22:11
fungii had to rerun vipw and vigr (and -s) to get rid of the "extra" jenkins at 3000:300122:12
fungiand then re-chown ~jenkins22:12
*** ddieterly[away] is now known as ddieterly22:12
*** rfolco has quit IRC22:12
fungipuppet had created that entry but not the others for reviewday, recheck and releasestatus22:12
*** hashar has quit IRC22:13
fungiso i got overzealous in adding them and forgot to remove/replace teh jenkins account22:13
*** matt-borland has quit IRC22:14
*** javeriak has joined #openstack-infra22:15
*** javeriak_ has quit IRC22:15
*** rlandy has quit IRC22:16
jeblairzuul is restarted and reenqueing22:16
fungithanks jeblair22:16
fungipleia2: minor suggestion for 336722, i think we can reasonably simplify that addition22:17
pleia2fungi: thanks22:17
fungiif anybody is using our puppet class, an unexpected apache 2.4ism is the least of their worries22:18
fungiso i'd rather we didn't make those templates any more complex than they already are if we can help it22:18
fungiif it were in one of our other puppet modules i would be more inclined to agree with the additional level of caution22:19
*** berendt has joined #openstack-infra22:19 is still offline/not working, is this related to the maintenance of gerrit/static?22:20
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add Apache 2.4 syntax for static.o.o
fungiberendt: should have been up and available for the past ~40 minutes22:20
berendtAh, it is :)22:20
*** sheeprine has quit IRC22:21
berendtI think the review proposed by Elizabeth is related to this issue.22:21
fungiberendt: oh!22:21
fungiberendt: yep, it's working via https but not http22:21
fungiand i hadn't checked http22:21
fungiand yes, 336722 will get the http version working again too22:21
pleia2interesting that it's working via https22:21
berendtyes, little bit strange22:21
*** Douhet has quit IRC22:23
*** yamamoto has joined #openstack-infra22:24
*** Douhet has joined #openstack-infra22:24
fungipleia2: i went ahead and punted 336722,2 straight to the gate22:26
pleia2fungi: thanks22:26
fungiso we won't have to wait for resources in the check pipeline22:26
*** piet_ has joined #openstack-infra22:27
fungisince we've sort of blasted our quota with these several zuul restarts in a row now22:27
fungiyay job logs uploading again though!
*** salv-orlando has joined #openstack-infra22:31
*** jsavak has quit IRC22:32
*** achanda has quit IRC22:32
fungiahh, yep being broken has caused some jobs which rely on downloading things from it at runtime to break22:34
fungibut looks like the fix is merging in seconfs22:34
fungiseconds too22:34
fungihere it comes22:34
*** tlian has quit IRC22:35
*** jianghuaw has quit IRC22:35
fungii guess uploads for our apply jobs are slow22:35
openstackgerritMerged openstack-infra/system-config: Add Apache 2.4 syntax for static.o.o
fungithere it is22:35
*** javeriak has quit IRC22:36
*** jianghuaw has joined #openstack-infra22:36
fungimachinery exercised22:36
*** achanda has joined #openstack-infra22:36
*** asettle has joined #openstack-infra22:37
jeblairi will fix root's known hosts on puppetmaster22:38
jeblairneeds static/zuul entries removed22:38
fungioh, we still need to do that even with ansible launching?22:38
*** rossella_s has quit IRC22:39
fungiyeah, that explains what i was just starting to look into, which is why there's no puppet mentioned in teh syslog of our new static.o.o22:39
jeblairfungi: i don't think it removes old entries?22:39
fungiagreed. that seems to be the case22:39
*** rossella_s has joined #openstack-infra22:39
jeblairokay, done22:39
fungii thought it was connecting by ip address alone due to the fact that we can have multiples with the same name and some not in dns22:40
fungilooks like it just connected now, so seems to be the case that there's some complication there with conflicting host keys22:42
jeblairi also am running our single-host adhoc script22:42
fungioh, is that why i saw puppet fire up?22:42
fungithought it seemed quick on the draw22:42
*** asettle has quit IRC22:42
jeblairfungi: yeah, i think the run-all cron missed it this time22:43
jeblairso it would have just been the ad-hoc22:43
jeblairthough that appears not to have helped22:43
fungii'm guessing teh ad-hoc play doesn't pull system-config updates?22:44
fungiso it's still working off a git pull from before that merged22:44
jeblairprobably not, so it's probably not very useful for this sort of thing22:44
*** piet_ has quit IRC22:45
fungiand the existing run went a little long so we're missing the 22:45 pulse22:45
jeblairokay, i'll run the update puppet play and the adhoc again22:45
fungilooks like it was probably over by about 30 seconds22:45
fungi_now_ /optsystem-config/production is updated22:46
fungier, /opt/system-config/production22:46
jeblairi ran ansible-playbook -f 10 ${ANSIBLE_PLAYBOOKS}/update_puppet.yaml22:46
jeblairit's still doing something....22:47
jeblairTASK [command] *****************************************************************22:47
fungigot it, and that does the needed git pull i guess?22:47
jeblairTASK [git] *********************************************************************22:47
jeblairchanged: [localhost]22:47
jeblairthat already happened22:47
jeblairah, install_modules is the slow bit22:47
jeblairokay, running the adhoc on static now22:48
*** tonytan4ever has quit IRC22:48
bkerooof, those task names could use some love22:48
*** karthikp_ has joined #openstack-infra22:49
fungiupdated apache vhost configs that time22:49
fungirestarted apache22:49
fungii'm still getting 403 forbidden22:50
pleia2oh bother22:51
*** tonytan4ever has joined #openstack-infra22:51
fungiand i see why22:51
*** piet_ has joined #openstack-infra22:52
*** rbrndt has quit IRC22:52
fungilooks like /etc/apache2/sites-enabled/ only has directory blocks for /srv/static/election and /srv/static/governance under the http vhost22:52
fungiand /srv/static/tarballs under the https vhost22:52
fungithat's exceptionally odd22:53
jeblairwell, what about releases?  is that simpler?22:53
fungisame pattern. /srv/static/election and /srv/static/governance under the http vhost22:54
jeblairi get what you're saying.22:54
*** karthik__ has joined #openstack-infra22:54
jeblairthis looks either very auto-generated or very copy-pastad.22:54
fungii'm checking over the template it's using now22:55
*** ddieterly is now known as ddieterly[away]22:55
*** harlowja_ has joined #openstack-infra22:55
*** ddieterly[away] has quit IRC22:55
fungii wonder if someone got scope-happy and updated these somehow breakagey22:56
fungidirectories is not in the openstack_project::static scope it's supposed to be from the scope of teh vhost instantiating the template22:57
*** tristanC_ is now known as tristanC22:58
*** karthikp_ has quit IRC22:58
fungii think this was somehow a no-op on 2.222:58
fungithe same template expansion silliness can be observed on the old server as well22:58
fungibut i think it must have defaulted to permissiveness22:58
*** harlowja has quit IRC22:59
jeblairfungi: just for sanity, i manually edited the releases conf file and made the http section look like the https section23:00
fungijeblair: thanks. i'm hacking on a patch for the templates now23:00
jeblair(in an attempt to find any other lurking errors)23:00
jeblairand it seems to work23:00
*** zzzeek has quit IRC23:01
*** zzzeek has joined #openstack-infra23:02
*** ams__ has quit IRC23:04
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Fix variable scopes in static vhost template
fungijeblair: pleia2: ^23:05
*** zeih has joined #openstack-infra23:05
fungiinterestingly, the redirect vhost template got it right23:05
*** roxanagh_ has quit IRC23:06
pleia2ah, scopes23:07
fungioh, hold up, there's some duplication in there i also missed23:07
openstackgerritJef Mallal proposed openstack-infra/jenkins-job-builder: Add support for retry in multijob plugin
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Fix variable scopes in static vhost template
*** roxanagh_ has joined #openstack-infra23:09
fungiactually, no duplication, i just re-confused myself when writing the commit message, but now i've spotted a discrepancy in how the puppet class is defining these23:09
*** zeih has quit IRC23:10
*** zzzeek has quit IRC23:11
fungiit looks like actually didn't quite grok that these are shared templates being used across multiple vhosts23:11
fungiso 336736 is not the fix for this23:12
fungialiases and directories _are_ defined at the class level23:12
fungiso those are getting added to every chost23:13
fungier, vhost23:13
fungiworse, the if/else in there is causing it to apply to all vhosts because it's defined at the class level23:14
*** zzzeek has joined #openstack-infra23:14
fungii'm going to push a revert of the elections alias change for the moment, once i resolve the merge conflicts on it23:16
*** Apoorva_ has joined #openstack-infra23:17
*** Swami has quit IRC23:20
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Add election alias to"
*** Apoorva has quit IRC23:21
fungipleia2: jeblair: ^ that's the fix (for now)23:21
fungihopefully the explanation in the commit message makes sense in context23:21
pleia2it does, sorry for missing it when I reviewed the initial patch, that static server /o\23:22
*** Apoorva_ has quit IRC23:22
fungimeh, it's a confoundingly complex situation in there23:23
fungithe governance site should really just get its own vhost template23:24
*** javeriak has joined #openstack-infra23:24
fungithe shared template is intended for sites that do nothing unusual, so if you see a change adding something to that shared template for the benefit of a single site it's probably in error23:24
*** yamamoto has quit IRC23:24
fungithe only "sane" way to extend that with custom values that aren't already implemented in the apache vhost resource is probably to somehow subclass it or something23:26
*** tonytan4ever has quit IRC23:27
fungior maybe there's a way to add some lookup hash where you can add keys based on some value known from the vhost scope23:27
fungiso the hash is in the class scope but has keys with the site names and then the template checks the hash keyed by the site name23:28
fungianyway, complicated23:28
fungicustom template for that one site would be a simpler solution23:28
*** piet_ has quit IRC23:29
fungiwriting that replacement now23:32
*** gyee has quit IRC23:33
*** Sukhdev_ has quit IRC23:33
*** sdague has quit IRC23:33
pleia2we're pretty close, should status ok soon23:35
*** karthik__ has quit IRC23:35
*** ianychoi has quit IRC23:39
openstackgerritMerged openstack-infra/system-config: Revert "Add election alias to"
*** davideagnello has joined #openstack-infra23:40
*** davideagnello has quit IRC23:40
*** rfolco has joined #openstack-infra23:41
*** xarses has quit IRC23:42
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add election alias to
fungipleia2: jeblair: ^ i've resubmitted jhesketh's change more safely23:43
*** shashank_hegde has quit IRC23:44
pleia2fungi: want to add a ref: to the previous one in the commit message?23:45
fungisure, can do23:45
*** piet has joined #openstack-infra23:46
*** ajmiller_ has quit IRC23:47
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add election alias to
rm_workwhy am I seeing telnet://  links on zuul's status page for my changes that are in queue?23:50
rm_workis that normal? O_o23:50
rm_worki'm used to it linking me to jenkins23:50
fungirm_work: yes, we no longer run jenkins at all anywhere23:50
rm_workooooooookay then :P that is unexpected but would explain that23:51
fungirm_work: the telnet:// links are usable for now (zuul provides a tcp socket stream of the console logs for jobs while they're still running) and eventually someone will provide a websocket implementation to allow you to stream them in your browser as a webpage23:51
rm_workcool, so telnet follows the log23:51
fungirm_work: yeah, there were... announcements23:51
rm_workyeah i see23:51
rm_workah, mailing list?23:52
fungiso hopefully not a surprise to people who follow the mailing lists. yep23:52
rm_workyeah, i got kicked from the mailing list :/23:52
*** karthikp_ has joined #openstack-infra23:52
fungioh? what happened?23:52
rm_workit kept saying my email was sending too many bounces!? and i kept verifying my email with the link it would send... and eventually it gave me links with tokens that didn't work, so i gave up23:52
rm_workthe email for reference is my address, lol23:53
fungirackspace e-mail address by any change?23:53
clarkbWhen that happened to me I switched email providers23:53
rm_worki honestly barely read it anyway, so i gave up23:53
fungirackspace has (had?) some sort of misbehaving "fireeye" mail filter23:53
fungikept randomly bouncing e-mails for weeks if not months23:53
rm_workthere's so many hundreds of emails that i just filter out the ones with my project's name in the subject anyway23:54
*** rfolco has quit IRC23:54
rm_worki think that's literally everyone I know's config...23:54
rm_worknot that it's your fault/problem :P23:54
rm_workI just wouldn't be super surprised that no one sees announcements like that, heh23:54
fungihuh. i actually skim the general, -dev, -infra, -docs, -i18n, -stable-maint, -security and probably others i'm forgetting23:55
*** yamamoto has joined #openstack-infra23:55
fungithen again, i also don't have trouble keeping up with debian-devel and several other debian mailing lists too23:55
rm_workyeah, you would be the first i've talked to that actually does that ^_^23:55
fungiand the oss-sec ml, and some high-volume cryptography mailing lists23:56
rm_worki think i have 29,000 unread emails in my openstack-dev folder23:56
rm_workthat's over...23:56
*** thiagop has quit IRC23:56
fungioh, and python mailing lists23:56
rm_worki forget how long, but not THAT long T_T23:56
rm_workif i tried to deal with the MLs i'm on I'd go insane23:56
fungii learned long ago that keeping up with very high volumes of e-mail is only doable if you use mail clients that are engineered to make it convenient to deal with very high volumes of e-mail23:57
pleia2mutt <323:57
fungi'tis what i use too23:57
clarkbdoes mutt speak imap yet? seems like last I tried to use it was local mail only23:58
fungiuh, have you ttied it in the past 15 years?23:58
fungier, tried23:58
clarkbits been a long time23:58
clarkb9 years maybe23:58
pleia2yeah, I only use it with imap these days23:59
*** javeriak has quit IRC23:59
clarkbya 07ish iirc23:59
*** salv-orl_ has joined #openstack-infra23:59
pleia2fungi: I am losing track already, what needs to land before we can #ok ?23:59

Generated by 2.14.0 by Marius Gedminas - find it at!