Wednesday, 2015-04-15

*** dboik has joined #openstack-infra00:00
*** mriedem has quit IRC00:01
clarkbsdague: so far the job hsa been more reliable than nova >_>00:01
sdagueoh, with the live migration regressions?00:02
*** dboik has quit IRC00:02
*** mahito has joined #openstack-infra00:03
sdagueyeh, well in the real world most people doing that are using ceph00:04
sdagueso it's a slightly convoluted setup00:04
*** claudiub has quit IRC00:04
jogowe aren't getting a lot of runs at the moment00:05
sdagueI guess the patch flow is still in lull with the release00:05
jogosdague: yeah00:06
sdaguejogo: oh, that's gate only, yeh, what about check queue00:06
clarkbharlowja: you are likely going to need a gerrit admin to force push those commits in00:06
clarkbharlowja: how did we miss these when we sycned the repo in?00:06
sdagueclarkb: so, would you object if I changed the name to 'multinode' ?00:06
sdaguebecause, then people would know what it was00:06
clarkbsdague: no, I think that would be fine00:07
sdaguebased on the d-g code, I think we can do that in project config, and it will all be fine00:07
jogosdague: that is check queue the title is wrong00:08
clarkbsdague: ya as long as that job sets the mode to aiopcpu for d-g00:08
clarkbsdague: and don't forget to update zuul's node selection function to pick the 2-node env for "multinode"00:08
sdagueclarkb: the logic in d-g is actually inverted00:08
clarkbsdague: oh right its if !=aio00:08
sdagueclarkb: yeh, git grep is my fried00:08
clarkbso ya should just work00:08
Krinklejeblair: Thx, I'll re-triage a few things on our end and see to it that we have some actionable items for the next few days.00:09
*** julim has joined #openstack-infra00:10
openstackgerritSean Dague proposed openstack-infra/project-config: change name from aiopcpu to multinode
sdagueclarkb: there you go00:12
*** otter768 has joined #openstack-infra00:12
*** koolhead17 has quit IRC00:13
jogosdague: excellent00:14
sdagueok, have a good night folks00:15
jogosdague: only 3 multinode runs since the job was fixed00:15
*** otter768 has quit IRC00:17
*** baoli has joined #openstack-infra00:18
*** mbacchi1 has quit IRC00:22
*** r-daneel has quit IRC00:24
openstackgerritArmando Migliaccio proposed openstack-infra/devstack-gate: Follow up with some comments to clarify OVS commands
armaxsdague, clarkb: ^^^00:24
*** salv-orlando has joined #openstack-infra00:25
jamielennoxis there any facility in infra to rename a repo or do i need to essentially propose it again?00:25
openstackgerritSpencer Krum proposed openstack-infra/system-config: Use envassert for integration tests
clarkbarmax: I am wondering if we should be more specific. For the add port bit say something like "Create a GRE tunnel between remote_ip and local_ip uniquely identified by key00:29
clarkbjamielennox: its even more fun than that, we need to take a gerrit downtime to update databases00:29
armaxclarkb: can do, I was not sure about the level of verbosity needed00:29
jamielennoxclarkb: ... i think i'll just repropose :p00:29
clarkbjamielennox: if the project isn't created yet that would be easiest00:29
armaxclarkb: the command seems pretty self explainatory to me, or at least not any more convoluted than the equivalent ip/brctl syntax00:29
armaxclarkb: let me make the addition, since I need to fix a stupid syntactic error00:30
jamielennoxclarkb: it's created but there hasn't been a release or anything so it shouldn't be a history problem or anything, i can set upstream to the existing, and then later we remove the original project?00:31
*** wenlock has joined #openstack-infra00:31
jamielennoxclarkb: ? is does that make sense00:31
fungiwe batch up project renames, so we'd rather rename your project than have a dead copy of it on the servers in addition to your working one00:31
jamielennoxok, fun - how do i get a project on the rename queue?00:32
fungiwe have about a dozen scheduled to rename this friday, and could make it a dozen-plus00:32
jamielennoxoh - right, i thought i was doing something really unusual :)00:32
fungimake a change like the ones you see linked at
*** tiswanso has joined #openstack-infra00:32
*** salv-orlando has quit IRC00:32
fungii should have said "baker's dozen"00:33
fungioh well, the moment has passed00:33
clarkbjamielennox: right you cna't just repropose00:33
*** nelsnelson has quit IRC00:33
clarkbjamielennox: we have to gerrit downtime, update db, move files around etc00:34
openstackgerritArmando Migliaccio proposed openstack-infra/devstack-gate: Follow up with some comments to clarify OVS commands
fungifor the curious, our process is documented at
*** Krinkle is now known as Krinkle|detached00:34
lifelessclarkb: / fungi: I'd like to get a node held at some point00:35
fungilifeless: one running a particular job? one which hasn't run any jobs yet?00:35
*** dims_ has joined #openstack-infra00:35
lifelessit passes unit tests locally00:35
lifelessfailed in CI00:35
lifelessI need to rebase it but thats hygiene, not functional00:36
*** dims has quit IRC00:36
fungiis the pep8 failure at all related?00:36
*** nelsnelson has joined #openstack-infra00:36
jamielennoxclarkb, fungi: will put on the list - thanks00:36
lifelessits irrelevant00:36
openstackgerritlifeless proposed openstack-dev/pbr: Honour excludes
fungithe problem is pbr.tests.test_packaging.TestPackagingInGitRepoWithCommit.test_manifest_exclude_honoured(preversioned) in this case?00:37
*** arosen2 has joined #openstack-infra00:37
fungiso you're looking for a bare-trusty node. i guess even a used one will do as long as it's not still running a job at that point00:38
lifelessfungi: yes00:38
lifelessfungi: which works here :)00:38
*** arosen2 has left #openstack-infra00:38
fungisince the jobs which run on bare-.* nodes at least mostly don't ever do things with sudo00:38
lifelessfungi: I've just pushed up a rebase, which means we should get a node for this itself soonish00:38
fungiokay, i'll hold the one running its python27 job once it pops up00:38
*** arosen1 has quit IRC00:39
*** nelsnelson has quit IRC00:39
fungiwhich ssh key do you want authorized on it?00:40
*** emagana has quit IRC00:40
*** emagana has joined #openstack-infra00:41
lifelessthat was fast00:41
lifelesssee query00:41
fungilifeless: ssh jenkins@
*** jogo has quit IRC00:42
fungilet's make sure i pieced that pubkey back together right ;)00:42
*** tiswanso has quit IRC00:44
lifelessam in00:44
fungicool. let me know when you're done and i'll recycle it back into the pool00:44
*** tiswanso has joined #openstack-infra00:44
*** yamamoto has joined #openstack-infra00:46
lifelessok, so reproduced00:46
lifelessnow the hard bit - figuring out wtf00:46
fungiisn't that most of what we all spend our days doing?00:46
lifeless /wrists00:47
*** tiswanso_ has joined #openstack-infra00:47
lifelesslunch then diving in00:47
*** emagana has quit IRC00:48
*** tiswanso has quit IRC00:50
*** nelsnelson has joined #openstack-infra00:51
*** kmartin has quit IRC00:51
harlowjaclarkb i think it was mainly just the one that was on git that got synced from was out of date00:56
harlowja*wasn't resynced with bzr  before the sync event....00:56
*** nelsnelson has quit IRC00:56
openstackgerritMerged openstack-infra/devstack-gate: Add Neutron DVR aiopcpu support
clarkbharlowja ok, so ya you will likely need to construct the repo state you want then have a gerrit admin force push in00:59
harlowjathx; will see about that01:00
*** dimtruck is now known as zz_dimtruck01:00
*** tqtran has joined #openstack-infra01:01
*** mahito has quit IRC01:01
openstackgerritClark Laughlin proposed openstack-infra/puppet-jenkins: Change installation of the JDK to use the architecture of the host
openstackgerritDavid Stanek proposed stackforge/gertty: Removes the need to pass around the app object
openstackgerritDavid Stanek proposed stackforge/gertty: Add the initial CLI implementation
openstackgerritDavid Stanek proposed stackforge/gertty: Use a named argument for server
openstackgerritDavid Stanek proposed stackforge/gertty: wip: add sync CLI command
openstackgerritDavid Stanek proposed stackforge/gertty: Refactor: move getRepo out of the App object
openstackgerritJamie Lennox proposed openstack-infra/project-config: Rename keystoneclient-federation to saml2
*** yamamoto has quit IRC01:05
*** mahito has joined #openstack-infra01:06
*** ayoung has joined #openstack-infra01:09
*** achanda has quit IRC01:10
*** jogo has joined #openstack-infra01:10
morganfainbergwant to see something neat? not sure if we care to fix / disable this:01:10
openstackMozilla bug 667429 in Networking: HTTP "Shibboleth Replay Detection Failure" [Normal,Resolved: invalid] - Assigned to nobody01:10
morganfainberg*points* OMG see it.01:10
harlowjaclarkb is the changes that need to sync up, will talk to the other cloud-init folks to see what/when to do this...01:13
harlowjapita :-P01:13
*** marun has quit IRC01:13
fungimorganfainberg: fix/disable it where?01:14
morganfainbergfungi, not sure.01:14
morganfainbergfungi, in the puppet config for the bot that makes it talk to bugzilla01:15
morganfainbergfungi, we can probably make it only do LP01:15
* morganfainberg isn't sure it is worth caring about01:15
fungimorganfainberg: oh! i thought you meant fix/disable the apache issue mentioned there01:15
fungiwe as a project do actually care about red hat bugs from time to time01:15
morganfainbergwell that is a mozilla bug.01:16
fungier, right01:16
morganfainbergbut :P01:16
fungiless so about those ;)01:16
morganfainbergi wont worry, just surprised me to see it01:16
morganfainbergwas like OH HAI01:16
fungithough it does a lot of bug trackers, for example
openstackDebian bug 781027 in wnpp "ITP: nodepool -- nodepool a system to launch disposable OpenStack VM for testing purposes" [Wishlist,Open]01:16
*** sdake_ has joined #openstack-infra01:17
clarkbharlowja we sync what you told us01:18
clarkbharlowja we have very little way of knowing if it is correct01:18
*** sdake has quit IRC01:21
*** dboik_ has joined #openstack-infra01:21
*** tqtran has quit IRC01:23
*** esker has joined #openstack-infra01:24
*** Daisy has joined #openstack-infra01:25
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Tidy up generators into contained objects
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Add support for setting filters and views per file
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Allow the config to set the filter and view
jheskethclarkb: ^01:26
*** Swami_ has joined #openstack-infra01:28
*** sarob has joined #openstack-infra01:30
*** Daisy_ has joined #openstack-infra01:30
*** yamamoto has joined #openstack-infra01:30
*** Swami has quit IRC01:32
*** Daisy has quit IRC01:33
*** Daisy_ is now known as Daisy01:33
*** Sukhdev has quit IRC01:34
*** nelsnelson has joined #openstack-infra01:39
*** woodster_ has quit IRC01:40
*** nelsnelson has quit IRC01:41
*** miqui has joined #openstack-infra01:44
*** shardy_z has quit IRC01:46
*** sdake has joined #openstack-infra01:52
*** yamahata has quit IRC01:53
*** baoli has quit IRC01:53
*** baoli has joined #openstack-infra01:54
*** baoli has quit IRC01:54
*** sdake_ has quit IRC01:56
*** kaisers has joined #openstack-infra01:58
*** kaisers1 has quit IRC01:59
*** gyee has quit IRC02:02
*** ivar-lazzaro has quit IRC02:02
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Report memory usage low-point
*** jokke_ has joined #openstack-infra02:09
jokke_anyone here still awake?02:09
*** achanda has joined #openstack-infra02:10
*** amotoki has joined #openstack-infra02:11
ianwjokke_: it's midday somewhere02:11
jokke_ianw: fair point02:12
*** otter768 has joined #openstack-infra02:13
jokke_So We'd like to create stable brances for glanceclient and glance_store ... where/how I should get that done? I'd expect I should propose change on some of the infra configs, but not sure where.02:13
*** sarob has quit IRC02:13
jokke_I'd expect at least on the project-config acls02:14
*** changbl has joined #openstack-infra02:14
*** unicell has quit IRC02:16
*** achanda has quit IRC02:17
*** harlowja is now known as harlowja_away02:17
*** otter768 has quit IRC02:18
*** woodster_ has joined #openstack-infra02:19
*** dims_ has quit IRC02:21
*** fedexo has joined #openstack-infra02:23
*** spzala has quit IRC02:26
*** sabeen1 has joined #openstack-infra02:27
*** yamahata has joined #openstack-infra02:28
*** salv-orlando has joined #openstack-infra02:28
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Report memory usage low-point
*** tkelsey has joined #openstack-infra02:36
lifelesssomeday we'll be allowed to have nice things02:37
*** patrickeast has quit IRC02:40
*** salv-orlando has quit IRC02:41
*** davideagnello has quit IRC02:42
*** tkelsey has quit IRC02:42
*** julim has quit IRC02:43
*** sabeen1 has quit IRC02:43
*** SumitNaiksatam has joined #openstack-infra02:45
lifelessfungi: how can do I root things on this node?02:46
*** sabeen1 has joined #openstack-infra02:47
*** julim has joined #openstack-infra02:55
*** esker has quit IRC02:57
*** esker has joined #openstack-infra02:57
*** nijaba has quit IRC03:01
*** mriedem has joined #openstack-infra03:02
*** mahito has quit IRC03:03
*** mriedem1 has quit IRC03:04
*** otter768 has joined #openstack-infra03:05
*** nijaba has joined #openstack-infra03:06
lifelessfungi: thanks, node can be freed03:06
lifelessclarkb: ^03:06
openstackgerritlifeless proposed openstack-dev/pbr: Test that excludes work
*** mriedem has quit IRC03:08
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Report memory usage low-point
*** achanda has joined #openstack-infra03:13
*** sarob has joined #openstack-infra03:14
*** Sukhdev has joined #openstack-infra03:17
*** achanda has quit IRC03:17
*** Longgeek has joined #openstack-infra03:18
*** aswadr has joined #openstack-infra03:25
*** sputnik13 has quit IRC03:25
*** sputnik13 has joined #openstack-infra03:25
*** ashleighfarnham has joined #openstack-infra03:37
*** Longgeek has quit IRC03:41
*** achanda has joined #openstack-infra03:49
*** tiswanso_ has quit IRC03:50
*** ayoung is now known as ayoung_ZZZzzz03:52
*** aswadr has quit IRC03:54
*** amotoki_ has joined #openstack-infra03:54
*** ashleighfarnham has quit IRC03:56
*** SumitNaiksatam has quit IRC04:00
*** otter768 has quit IRC04:01
*** stevemar has quit IRC04:01
*** SumitNaiksatam has joined #openstack-infra04:01
*** ashleighfarnham has joined #openstack-infra04:04
*** sgordon has quit IRC04:04
*** SumitNaiksatam has quit IRC04:06
*** krtaylor has quit IRC04:07
*** mahito has joined #openstack-infra04:09
*** SumitNaiksatam has joined #openstack-infra04:09
*** sgordon has joined #openstack-infra04:10
*** krtaylor has joined #openstack-infra04:10
*** sdake has quit IRC04:11
*** ashleighfarnham has quit IRC04:13
*** patrickeast has joined #openstack-infra04:16
*** sarob has quit IRC04:16
*** sarob_ has joined #openstack-infra04:19
tchaypojokke_: I’d love to point you at the relevant docs in
tchaypobut they don’t seem to exist04:20
*** btran has quit IRC04:21
tchaypobut there *are* instructions for creating release branches and I’m assuming the process for creating a stable/ branch would be similar04:21
*** koolhead17 has joined #openstack-infra04:21
*** sks has joined #openstack-infra04:23
*** amotoki_ has quit IRC04:24
tchaypolifeless: I misread the “excludes” in that subject line as a verb and “work” as a noun.04:26
tchaypoenglish is delightful04:26
*** crc32 has joined #openstack-infra04:28
*** mgedmin has quit IRC04:28
*** salv-orlando has joined #openstack-infra04:29
*** mgedmin has joined #openstack-infra04:29
*** sputnik13 has quit IRC04:29
*** davideagnello has joined #openstack-infra04:30
*** Daisy has quit IRC04:31
*** davideagnello has quit IRC04:35
*** salv-orlando has quit IRC04:36
*** Longgeek has joined #openstack-infra04:37
*** luqas has joined #openstack-infra04:39
*** luqas has quit IRC04:43
*** Longgeek has quit IRC04:44
*** Swami_ has quit IRC04:47
*** achanda has quit IRC04:50
*** achanda has joined #openstack-infra04:50
*** xyang1 has quit IRC04:52
*** BharatK has joined #openstack-infra04:53
*** wenlock has quit IRC04:54
*** stevemar has joined #openstack-infra04:55
*** sabeen1 has quit IRC04:56
*** yamada-h has quit IRC05:00
*** unicell has joined #openstack-infra05:01
*** yamada-h has joined #openstack-infra05:02
yamamotocan anyone push this ?05:07
yamamotothe dependent patch has been merged05:07
*** deepakcs has joined #openstack-infra05:08
*** mrodden has quit IRC05:12
*** Daisy has joined #openstack-infra05:16
lifelesstchaypo: heh, no.05:16
*** tnovacik has joined #openstack-infra05:17
*** Sukhdev has quit IRC05:17
*** thingee has quit IRC05:29
*** ociuhandu has quit IRC05:29
*** resker has joined #openstack-infra05:32
*** esker has quit IRC05:35
*** ociuhandu has joined #openstack-infra05:38
*** patrickeast has quit IRC05:40
*** sputnik13 has joined #openstack-infra05:40
*** yamada-h has quit IRC05:44
*** samwan has joined #openstack-infra05:45
samwanis this the right place for devstack-gate query?05:45
samwandoes anyone know why devstack-gate does not support redhat?05:46
clarkbsamwan it should. we run tests on centos and fedora with it05:48
samwanno it does not support REDHAT05:48
*** Longgeek has joined #openstack-infra05:48
samwanthere's a function named start_timer in functions.sh05:49
samwanit only allows fedora,centos05:49
samwanbut not redhat05:49
clarkbcan you explain further? both centos and fedora are redhat related05:49
samwanfunction is_fedora {05:49
samwan    # note we consider CentOS 7 as fedora for now05:49
samwan    lsb_release -i 2>/dev/null | grep -iq "fedora" || \05:49
samwan        lsb_release -i 2>/dev/null | grep -iq "CentOS"05:49
samwanbecause of this05:49
samwanif is_fedora; then05:49
samwan        local ntp_service='ntpd'05:49
samwan    elif uses_debs; then05:49
samwan        local ntp_service='ntp'05:49
samwan    else05:49
samwan        echo "Unsupported platform, can't determine ntp service"05:49
samwan        exit 105:49
samwan    fi05:49
clarkbplease use a paste service for large pastes05:50
samwansorry for that05:50
clarkbbut looks like it just needs an updated check05:50
clarkbwe do not test on rhel because it costs money05:50
*** salv-orlando has joined #openstack-infra05:50
clarkbcentos is the alternative we do support05:50
samwani tested05:50
clarkbanyone can use that instead05:51
samwani think we should add redhat in the function05:51
samwanin is_fedora05:51
samwanwo should include grep -iq "redhat"05:51
clarkbfeel free to propose the addition05:51
samwanbut i don't know how to propose for devstack-gate05:51
clarkbits the same as any openstack project. make changes then use git review to push to gerrit05:52
samwanoh yeah05:52
samwanthanks clarkb05:52
*** mrodden has joined #openstack-infra05:55
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Invalidate user cache on user create
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for user_cache
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: MonkeyPatch time.sleep in unit tests to avoid wait
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test that deleting user invalidates user cache
*** ociuhandu has quit IRC05:56
*** mrunge has joined #openstack-infra05:57
*** yfried is now known as yfried|afk05:59
*** yfried|afk is now known as yfried05:59
*** salv-orlando has quit IRC06:00
*** otter768 has joined #openstack-infra06:02
*** unicell has quit IRC06:02
*** luqas has joined #openstack-infra06:04
*** mrunge has quit IRC06:05
*** garyk has joined #openstack-infra06:06
*** otter768 has quit IRC06:07
*** mrunge has joined #openstack-infra06:08
*** hdd has joined #openstack-infra06:09
*** yfried is now known as yfried|afk06:10
*** fedexo has quit IRC06:13
*** crc32 has quit IRC06:15
*** stevemar has quit IRC06:15
openstackgerritMerged openstack-infra/project-config: Enable DVR for Neutron aiopcpu job
*** alex_xu has quit IRC06:18
*** isviridov_away is now known as isviridov06:20
*** yfried|afk is now known as yfried06:20
*** miqui has quit IRC06:21
openstackgerritgreghaynes proposed openstack-infra/project-config: Add diskimage-builder functests
*** scheuran has joined #openstack-infra06:25
*** armax has quit IRC06:28
*** jlibosva has joined #openstack-infra06:28
*** jcoufal has joined #openstack-infra06:29
lifelessjd__: tchaypo: thanks for the reviews.. but deeper in the stack is what actually needs reviews :)06:29
jd__lifeless: yeah I'm reading :)06:30
*** alex_xu has joined #openstack-infra06:30
lifelessjd__: thanks :)06:30
*** yamada-h has joined #openstack-infra06:30
tchaypolifeless: starred for review later06:31
*** yfried is now known as yfried|afk06:32
tchaypolifeless: on which topic, 172691 is the latest gertty magic from jeblair - batches a bunch of small pushes up to the server. Upstream from that is a patch that avoids some duplicate tasks that were hitting the queue.06:33
jlibosva            /quit06:33
*** jlibosva has quit IRC06:33
tchaypoI’m not sure which one you’re at, but I’m having a much better experience running with the latest 17269106:33
lifelesstchaypo: I am not currently running gertty06:34
lifelesstchaypo: I didn't have time to fix it, and I didn't have time to use it06:34
*** jlibosva has joined #openstack-infra06:34
*** yamada-h has quit IRC06:35
tchaypoWould you be interested in me pinging you when those two things land?06:35
*** ildikov has quit IRC06:38
lifelesstchaypo: sure06:38
*** tkelsey has joined #openstack-infra06:39
StevenKlifeless: testtools will automatically pull in unittest2, or it needs some help?06:39
lifelessStevenK: define 'pull in'06:40
StevenKlifeless: import and use it. Trying to debug AttributeError: 'TestObject' object has no attribute 'assertRaisesRegex'06:41
lifelessStevenK: testtools has a versioned dep on unittest206:41
lifelessStevenK: whats the __mro__ for TestObject there?06:41
*** soren has joined #openstack-infra06:42
*** tkelsey has quit IRC06:44
StevenKlifeless: That's going to require patching it in, because docker. :-/06:44
*** unicell has joined #openstack-infra06:45
*** shardy has joined #openstack-infra06:45
lifelessStevenK: isn't there some way to get a debug session?06:45
lifelessStevenK: of course, if you're using docker, are you sure you're installing testtools from PyPI and not an ancient-ass-binary?06:46
StevenKlifeless: I am not installing testtools from pypi. It's coming from a local bootstrap archive and is
lifelessthats your problem06:47
*** achanda has quit IRC06:48
* StevenK grabs 1.7.1 from pypi and tries to package it up06:48
*** mwhudson has quit IRC06:48
*** woodster_ has quit IRC06:50
*** sushilkm has joined #openstack-infra06:52
*** sushilkm has left #openstack-infra06:52
*** yfried|afk is now known as yfried06:52
lifelessStevenK: you'll probably need to package linecache2, traceback2, unittest2 latest versions to go with06:53
*** rfolco has quit IRC06:55
*** sputnik1_ has joined #openstack-infra06:57
*** Ala has joined #openstack-infra06:59
*** sputnik13 has quit IRC07:00
*** sputnik13 has joined #openstack-infra07:01
*** yamada-h has joined #openstack-infra07:01
*** sputnik__ has joined #openstack-infra07:02
*** mrmartin has joined #openstack-infra07:03
*** samwan has quit IRC07:03
*** sputnik1_ has quit IRC07:03
*** yamada-h has quit IRC07:05
*** jeblair has quit IRC07:05
*** sputnik13 has quit IRC07:06
openstackgerritlifeless proposed openstack-dev/pbr: Fixup semver
openstackgerritlifeless proposed openstack-dev/pbr: Test that excludes work
*** dtantsur|afk is now known as dtantsur07:06
*** jeblair has joined #openstack-infra07:07
*** salv-orlando has joined #openstack-infra07:08
*** rwsu has quit IRC07:09
*** Guest87529 has joined #openstack-infra07:09
*** Guest87529 has quit IRC07:11
*** Guest87529 has joined #openstack-infra07:11
*** yfried is now known as yfried|afk07:12
*** markus_z has joined #openstack-infra07:12
*** yfried|afk is now known as yfried07:15
*** jistr has joined #openstack-infra07:24
*** sergsh has joined #openstack-infra07:28
*** sputnik13 has joined #openstack-infra07:31
*** yfried is now known as yfried|afk07:33
*** vlaza has joined #openstack-infra07:34
*** sputnik__ has quit IRC07:34
*** chlong has quit IRC07:35
*** yamada-h has joined #openstack-infra07:37
*** [HeOS] has quit IRC07:38
*** rfolco has joined #openstack-infra07:39
*** Guest87529 is now known as mwhudson07:40
*** jlanoux has joined #openstack-infra07:42
*** SumitNaiksatam has quit IRC07:44
*** SumitNaiksatam has joined #openstack-infra07:45
*** fhubik has joined #openstack-infra07:47
*** belmoreira has joined #openstack-infra07:47
*** mwhudson has left #openstack-infra07:47
*** yamada-h has joined #openstack-infra07:48
*** ildikov has joined #openstack-infra07:49
*** arxcruz has joined #openstack-infra07:50
*** hashar has joined #openstack-infra07:55
*** Somay has joined #openstack-infra07:58
*** e0ne has joined #openstack-infra08:02
*** otter768 has joined #openstack-infra08:02
*** e0ne has quit IRC08:05
*** otter768 has quit IRC08:07
*** Hal has joined #openstack-infra08:08
*** e0ne has joined #openstack-infra08:08
*** Hal is now known as Guest9325108:08
*** kiseok7_ has quit IRC08:08
*** davideagnello has joined #openstack-infra08:08
*** MaxV has joined #openstack-infra08:10
*** ildikov has quit IRC08:12
*** MaxV has quit IRC08:12
*** tkelsey has joined #openstack-infra08:13
*** davideagnello has quit IRC08:13
*** mpavone has joined #openstack-infra08:14
*** yfried|afk is now known as yfried08:17
*** e0ne has quit IRC08:19
*** Somay has quit IRC08:21
*** e0ne has joined #openstack-infra08:23
*** yamahata has quit IRC08:25
*** e0ne has quit IRC08:29
*** ociuhandu has joined #openstack-infra08:31
*** MaxV has joined #openstack-infra08:32
*** e0ne has joined #openstack-infra08:33
*** mahito has quit IRC08:34
*** cdent has joined #openstack-infra08:36
*** dtantsur is now known as dtantsur|brb08:37
*** vdo has joined #openstack-infra08:38
*** teran has quit IRC08:39
*** ildikov has joined #openstack-infra08:40
*** e0ne has quit IRC08:41
*** sarob_ has quit IRC08:46
*** otherwiseguy has quit IRC08:47
*** ihrachyshka has joined #openstack-infra08:49
*** otherwiseguy has joined #openstack-infra08:49
*** pelix has joined #openstack-infra08:50
*** ssam2 has joined #openstack-infra08:55
*** hichihara has quit IRC08:55
*** krtaylor has quit IRC08:57
*** pblaho has joined #openstack-infra08:59
*** amotoki has quit IRC09:01
*** pblaho has quit IRC09:02
*** pblaho has joined #openstack-infra09:02
*** krtaylor has joined #openstack-infra09:04
*** yfried is now known as yfried|afk09:06
*** yfried|afk is now known as yfried09:08
*** sdake has joined #openstack-infra09:10
openstackgerritAntoine Musso proposed openstack-infra/nodepool: Soften sqlalchemy and pyzmq requirements
*** fhubik is now known as fhubik_afk09:21
*** freerunner has quit IRC09:23
*** [HeOS] has joined #openstack-infra09:23
*** Daisy has quit IRC09:23
*** freerunner has joined #openstack-infra09:25
*** freerunner has quit IRC09:25
*** derekh has joined #openstack-infra09:25
*** freerunner has joined #openstack-infra09:26
*** freerunner has quit IRC09:26
*** freerunner has joined #openstack-infra09:27
*** ildikov has quit IRC09:28
*** freerunner has quit IRC09:29
*** freerunner has joined #openstack-infra09:29
*** fhubik_afk is now known as fhubik09:30
*** freerunner has quit IRC09:31
*** freerunner has joined #openstack-infra09:32
*** freerunner has quit IRC09:33
*** e0ne has joined #openstack-infra09:34
*** freerunner has joined #openstack-infra09:34
*** freerunner has quit IRC09:35
*** freerunner has joined #openstack-infra09:36
*** freerunner has quit IRC09:37
*** freerunner has joined #openstack-infra09:38
*** freerunner has quit IRC09:38
*** freerunner has joined #openstack-infra09:39
*** garyk has quit IRC09:39
*** garyk has joined #openstack-infra09:40
*** freerunner has quit IRC09:40
*** freerunner has joined #openstack-infra09:41
*** freerunner has quit IRC09:42
*** ZZelle has quit IRC09:44
*** freerunner has joined #openstack-infra09:44
*** fhubik is now known as fhubik_afk09:45
*** freerunner has quit IRC09:45
*** Guest93251 has quit IRC09:47
*** e0ne is now known as e0ne_09:49
*** liusheng has quit IRC09:51
*** freerunner has joined #openstack-infra09:51
*** jamielennox is now known as jamielennox|away09:52
*** freerunner has quit IRC09:52
*** freerunner has joined #openstack-infra09:54
*** freerunner has quit IRC09:54
*** jhesketh has quit IRC09:57
*** Somay has joined #openstack-infra09:58
*** jhesketh has joined #openstack-infra10:00
*** amotoki has joined #openstack-infra10:00
*** ildikov has joined #openstack-infra10:01
*** e0ne_ has quit IRC10:01
*** Longgeek has quit IRC10:02
*** Longgeek_ has joined #openstack-infra10:02
*** fhubik_afk is now known as fhubik10:03
*** e0ne has joined #openstack-infra10:03
*** Kennan2 has joined #openstack-infra10:03
*** otter768 has joined #openstack-infra10:03
*** Kennan has quit IRC10:04
*** ssam2 has quit IRC10:05
*** isviridov is now known as isviridov_away10:06
openstackgerritErno Kuvaja proposed openstack-infra/project-config: Add stable acls for python-glanceclient
openstackgerritErno Kuvaja proposed openstack-infra/project-config: Add stable acls for python-glanceclient
*** yamamoto has quit IRC10:08
*** otter768 has quit IRC10:08
*** teran has joined #openstack-infra10:09
*** dtantsur|brb is now known as dtantsur10:10
*** teran has quit IRC10:16
*** pc_m has joined #openstack-infra10:17
*** sarob has joined #openstack-infra10:19
*** ssam2 has joined #openstack-infra10:20
*** achanda has joined #openstack-infra10:22
*** dguitarbite has joined #openstack-infra10:24
*** sarob has quit IRC10:24
*** jcoufal has quit IRC10:27
*** achanda has quit IRC10:27
*** jcoufal has joined #openstack-infra10:27
*** tnovacik has quit IRC10:29
*** koolhead17 has quit IRC10:31
*** hashar is now known as hasharLunch10:33
*** ociuhandu has quit IRC10:34
*** claudiub has joined #openstack-infra10:34
*** MIDENN_ has quit IRC10:35
*** Somay has quit IRC10:36
*** Somay_ has joined #openstack-infra10:36
*** ssam2 has quit IRC10:38
*** e0ne is now known as e0ne_10:39
*** yfried is now known as yfried|afk10:39
*** amotoki_ has joined #openstack-infra10:40
*** teran has joined #openstack-infra10:41
*** e0ne_ is now known as e0ne10:42
*** amotoki has quit IRC10:43
*** mfmcdonagh has joined #openstack-infra10:44
*** yfried|afk is now known as yfried10:45
*** juggler has quit IRC10:45
*** sdake has quit IRC10:46
*** deepakcs has quit IRC10:46
*** Somay_ has quit IRC10:48
*** asselin_ has quit IRC10:49
*** ssam2 has joined #openstack-infra10:50
*** fhubik has quit IRC10:53
*** e0ne has quit IRC10:53
*** e0ne has joined #openstack-infra10:54
*** fhubik has joined #openstack-infra10:54
yolandahere, do you know if updating an upstream from an already existing project works? i amended one, but after running manage_projects, same content in the buggy repo10:55
*** dhritishikhar has joined #openstack-infra10:56
*** mattfarina has joined #openstack-infra10:58
*** arxcruz has quit IRC11:01
*** arxcruz has joined #openstack-infra11:02
*** amotoki has joined #openstack-infra11:02
openstackgerritIgor Belikov proposed openstack-infra/jenkins-job-builder: Add node-label parameter to trigger-builds builder
*** mattfarina has quit IRC11:04
*** amotoki_ has quit IRC11:05
*** zz_johnthetubagu is now known as johnthetubaguy11:07
*** dims has joined #openstack-infra11:08
*** ildikov has quit IRC11:09
*** BharatK has quit IRC11:09
*** mattfarina has joined #openstack-infra11:11
*** btully has quit IRC11:14
*** mrmartin has quit IRC11:16
*** btully has joined #openstack-infra11:16
*** e0ne is now known as e0ne_11:19
*** mwagner_lap has quit IRC11:21
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1441050 is fixed
openstackbug 1441050 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend" [Undecided,In progress] - Assigned to Bharat Kumar Kobagana (bharat-kobagana)11:22
*** dhritishikhar has quit IRC11:22
*** BharatK has joined #openstack-infra11:23
*** [HeOS] has quit IRC11:25
*** sergmelikyan has quit IRC11:25
*** [HeOS] has joined #openstack-infra11:26
*** e0ne_ has quit IRC11:29
*** Longgeek has joined #openstack-infra11:29
*** otter768 has joined #openstack-infra11:31
*** Longgeek_ has quit IRC11:31
*** nadya has joined #openstack-infra11:33
*** nadya is now known as Guest2717711:34
*** spredzy is now known as spredzy|afk11:34
*** Somay has joined #openstack-infra11:34
vladiskuzHi, folks! I've been playing with OAuth2.0 on and with User REST API on it. User API get following fields: name, family_name, nickname, picture, birthdate, gender, and email. Question: How can I distinguish between users? because in I can change my email and other fields11:34
*** Guest27177 is now known as _nadya_11:35
*** ociuhandu has joined #openstack-infra11:37
*** Mmike has quit IRC11:38
*** e0ne has joined #openstack-infra11:38
*** Mmike has joined #openstack-infra11:41
*** Mmike has joined #openstack-infra11:41
*** macjack has left #openstack-infra11:44
BobBallCan someone please remove the email address from the "XenProject CI" gerrit account so we don't get automatic emails from gerrit?11:44
*** davideagnello has joined #openstack-infra11:46
*** fhubik is now known as fhubik_afk11:47
*** hasharLunch is now known as hashar11:48
*** davideagnello has quit IRC11:50
*** doug-fish has joined #openstack-infra11:53
*** tnovacik has joined #openstack-infra11:54
*** otter768 has quit IRC11:56
*** dkranz has joined #openstack-infra11:56
*** freerunner has joined #openstack-infra11:57
*** freerunner has quit IRC11:57
*** freerunner has joined #openstack-infra11:58
*** freerunner has quit IRC11:59
*** isaic has joined #openstack-infra12:00
*** freerunner has joined #openstack-infra12:01
*** julim has quit IRC12:01
*** freerunner has quit IRC12:03
*** freerunner has joined #openstack-infra12:03
*** Hal has joined #openstack-infra12:04
*** notnownikki has quit IRC12:04
*** isaic has left #openstack-infra12:04
*** Hal is now known as Guest3238312:04
*** freerunner has quit IRC12:05
*** isaic has joined #openstack-infra12:05
*** isaic has left #openstack-infra12:05
*** ldnunes has joined #openstack-infra12:05
*** notnownikki has joined #openstack-infra12:05
*** freerunner has joined #openstack-infra12:07
*** mrmartin has joined #openstack-infra12:07
*** dprince has joined #openstack-infra12:08
*** freerunner has quit IRC12:08
*** sarob has joined #openstack-infra12:08
*** freerunner has joined #openstack-infra12:09
*** freerunner has quit IRC12:09
*** mwagner_lap has joined #openstack-infra12:09
*** freerunner has joined #openstack-infra12:12
*** freerunner has quit IRC12:12
*** sarob has quit IRC12:12
*** yamamoto has joined #openstack-infra12:13
*** laurelm has quit IRC12:13
*** freerunner has joined #openstack-infra12:14
*** e0ne is now known as e0ne_12:14
*** laurelm has joined #openstack-infra12:15
fungijokke_: check with ttx. the release managers have been coordinating creation of client stable branches so far from what i've seen12:15
fungitchaypo: ^ (for reference)12:15
mrmartinfungi, morning12:16
mrmartinhave you seen my email about openstackid-dev ?12:16
tchaypoThanks fungi12:16
*** freerunner has quit IRC12:16
*** fhubik_afk is now known as fhubik12:17
tchaypoI'd like to add something about this to the docs12:17
jokke_fungi: thanks ... yeah I noticed that the acl change needs to happen through the project-config repo, but I do not have rights to push the actual branch12:17
*** freerunner has joined #openstack-infra12:17
tchaypoIt sounds like, as a start, a pointer to the release manager email address (I'm assuming there is one) should be good?12:17
*** aysyd has joined #openstack-infra12:18
fungiyolanda: manage-projects intentionally ignores the upstream parameter except on initial import (when the project does not yet exist in gerrit), or for its somewhat obscure upstream tracking feature (where you keep upstream and a fork in separate branches)12:18
*** freerunner has quit IRC12:19
fungitchaypo: jokke_: probably best to ask ttx what he wants the documentation to recommend12:20
ttxwe'll be craeting stable branches for clients this afternoon12:21
fungimrmartin: thanks, fixing!12:21
vladiskuzFolks, have you seen my question above?12:22
jokke_ttx: glance_store would need one as well, thanks12:22
ttxyes, it's on the list12:22
jokke_ttx: gr812:22
*** bswartz has quit IRC12:22
ttxalso been discussing the client bump with stuart, we have that under control12:22
*** freerunner has joined #openstack-infra12:23
jokke_ttx: I know, that's why I tried to get the stable branches done12:23
*** yfried is now known as yfried|afk12:23
vladiskuzI've been playing with OAuth2.0 on and with User REST API on it. User API get following fields: name, family_name, nickname, picture, birthdate, gender, and email. Question: How can I distinguish between users? because on I can change my email and other fields12:24
mrmartinvladiskuz: yeap, it is a problem12:25
*** fhubik has quit IRC12:25
*** amotoki has quit IRC12:25
*** yfried|afk is now known as yfried12:26
vladiskuzIf I change my email than I get new openid with suffix .112:26
*** fhubik has joined #openstack-infra12:26
*** e0ne_ has quit IRC12:26
*** spredzy|afk is now known as spredzy12:26
mrmartinsure, we can ask smarcet, what he recommends here.12:26
fungimrmartin: fixed now12:26
*** amotoki has joined #openstack-infra12:27
*** vlaza1 has joined #openstack-infra12:28
*** vlaza has quit IRC12:28
mrmartinfungi: ack, works well.12:28
mrmartinthank you!12:28
*** cdent has quit IRC12:28
fungithanks for spotting my mistake!12:28
mrmartinanyway Sebastian plan to move the openstackid_id db too on dev.12:28
*** e0ne has joined #openstack-infra12:30
*** bknudson has joined #openstack-infra12:30
*** amotoki has quit IRC12:31
yolandafungi, thx, and if i did a mistake and create the wrong upstream, what's better? i remove the project and i recreate?12:32
*** koolhead17 has joined #openstack-infra12:33
fungiyolanda: elevate yourself to a member of whatever your equivalent of our project bootstrappers group is, and then push --force a new local repo in on top of it12:33
*** koolhead17 has quit IRC12:33
*** koolhead17 has joined #openstack-infra12:33
yolandaok, thx12:33
fungiyolanda: i assume you're talking about a gerrit you manage, and not review.openstack.org12:33
yolandaour gerrit12:33
fungimrmartin: oh, that's good to know. hopefully he'll give us a heads up when he's doing it this time12:34
*** eharney has joined #openstack-infra12:34
*** gordc has joined #openstack-infra12:34
mrmartinyeah, I asked him to announce *before* the change12:34
*** dkliban_afk is now known as dkliban12:35
*** ZZelle has joined #openstack-infra12:39
*** dkranz has quit IRC12:40
*** baoli has joined #openstack-infra12:42
*** dhritishikhar has joined #openstack-infra12:42
*** jaypipes has quit IRC12:42
*** otter768 has joined #openstack-infra12:43
*** julim has joined #openstack-infra12:43
*** Somay has quit IRC12:45
*** otter768 has quit IRC12:46
*** smccully has joined #openstack-infra12:46
*** rlandy has joined #openstack-infra12:46
*** arxcruz has quit IRC12:47
*** annegentle has joined #openstack-infra12:50
*** woodster_ has joined #openstack-infra12:50
*** mrmartin has quit IRC12:52
openstackgerritMerged openstack-infra/project-config: change name from aiopcpu to multinode
*** baoli_ has joined #openstack-infra12:54
openstackgerritMerged openstack-infra/system-config: Turn F20 min-ready down to zero before removal
*** garyk has quit IRC12:54
openstackgerritIgor Belikov proposed openstack-infra/jenkins-job-builder: Add node-label parameter to trigger-builds builder
openstackgerritMerged openstack-infra/system-config: Remove after using it in install_puppet
*** baoli has quit IRC12:56
*** jaypipes has joined #openstack-infra12:56
openstackgerritMerged openstack-infra/system-config: Rename summit server to design-summit-prep
openstackgerritJulia Kreger proposed openstack-infra/shade: Wrap ironicclient methods that leak objects
openstackgerritJulia Kreger proposed openstack-infra/shade: Change Ironic node lookups to support names
*** claudiub has quit IRC12:57
*** bswartz has joined #openstack-infra12:58
*** resker has quit IRC12:58
*** mrunge has quit IRC12:59
*** dhritishikhar has quit IRC13:00
*** ayoung_ZZZzzz is now known as ayoung13:06
*** mattfarina has quit IRC13:06
*** kgiusti has joined #openstack-infra13:06
*** miqui has joined #openstack-infra13:06
*** dboik_ has quit IRC13:08
*** mattfarina has joined #openstack-infra13:08
*** dhritishikhar has joined #openstack-infra13:10
*** adalbas has quit IRC13:12
*** dims has quit IRC13:13
eantyshevHello! Are there any dsvm jobs with centos7 images?13:14
*** fhubik is now known as fhubik_afk13:14
*** mjturek1 has left #openstack-infra13:14
*** dhritishikhar has quit IRC13:14
*** dims has joined #openstack-infra13:15
*** fhubik_afk is now known as fhubik13:16
*** dhritishikhar has joined #openstack-infra13:17
mordredjeblair: I have a strange request for gertty ... in unified diff mode - maybe skip printing the line numbers on the left-hand side in favor of a old,new line at the top of the section like diff -u does13:18
*** duran has joined #openstack-infra13:19
mordredjeblair: because in a normal 80 char xterm, unified diffs of openstack code should actually never wrap, since we're pedantic about that sort of thing13:19
mordredjeblair: but the line numbers make it wrap (this is also not really a problem and is easily fixable on my side - just musing outloud)13:20
*** david-lyle has quit IRC13:24
*** dustins has joined #openstack-infra13:25
*** dboik has joined #openstack-infra13:25
*** dhritishikhar has quit IRC13:26
*** dhritishikhar has joined #openstack-infra13:26
*** mriedem has joined #openstack-infra13:27
*** tiswanso has joined #openstack-infra13:29
*** peristeri has joined #openstack-infra13:30
*** wenlock has joined #openstack-infra13:30
*** r-daneel has joined #openstack-infra13:31
*** Daisy has joined #openstack-infra13:32
openstackgerritJulia Kreger proposed openstack-infra/shade: Basic test for meta method obj_list_to_dict
*** garyk has joined #openstack-infra13:33
openstackgerritRussell Bryant proposed openstack-infra/project-config: Add #openstack-neutron-ovn to gerritbot.
*** BharatK has quit IRC13:35
openstackgerritRussell Bryant proposed openstack-infra/system-config: Log #openstack-neutron-ovn.
openstackgerritMerged openstack-infra/shade: Add meta method obj_list_to_dict
spredzyif there is any core around, could I get a review . Thanks.13:38
*** yfried has quit IRC13:39
*** marcusvrn has quit IRC13:41
*** garyk has quit IRC13:41
*** yfried has joined #openstack-infra13:41
openstackgerritJulia Kreger proposed openstack-infra/shade: Add update_machine method and operator unit test substrate
*** e0ne is now known as e0ne_13:43
*** e0ne_ is now known as e0ne13:43
*** hdd has quit IRC13:46
*** mtanino has joined #openstack-infra13:48
*** markvoelker has joined #openstack-infra13:48
openstackgerritMerged openstack-infra/shade: Add API method create_subnet()
*** hdd has joined #openstack-infra13:50
openstackgerritMerged openstack-infra/shade: Add API method delete_subnet()
openstackgerritMerged openstack-infra/shade: Add API method delete_subnet()
openstackgerritMerged openstack-infra/shade: Add API method update_subnet()
*** dkliban has left #openstack-infra13:53
openstackgerritMerged openstack-infra/shade: Add get_subnet() method
sdaguefungi: our deleting servers get wedged?13:54
*** vlaza1 is now known as vlaza13:54
sdagueI see kind of steady growth over time there13:54
openstackgerritMerged openstack-dev/cookiecutter: Don't omit tests from coverage
*** marcusvrn has joined #openstack-infra13:56
*** sarob has joined #openstack-infra13:57
openstackgerritSteven Weston proposed openstack-infra/infra-specs: CI Dashboard Spec
*** fhubik has quit IRC13:57
*** cference_ has joined #openstack-infra13:57
*** cference_ has quit IRC13:57
sdaguefungi: yeh, we're down to about 100 working nodes13:58
fungisdague: looking to see if it's still rax having problems13:58
mordredfungi: you have a jeepyb change from ages ago with 2 +2s on it - about checking to see if a project uses github before using the pull request closer13:59
mordredfungi: any reasno I should avoid a +A?13:59
fungimordred: not to my knowledge. we can hold it up as proof that changes proposed by our core reviewers don't get fast-tracked though!14:00
*** dustins has quit IRC14:00
mordredfungi: hah. first +2 was in december14:00
*** amitgandhinz has joined #openstack-infra14:00
mordredscuse me - september14:01
*** sarob has quit IRC14:01
fungiso only 7 months. that's not so bad ;)14:02
*** dustins has joined #openstack-infra14:02
fungisdague: wow, that drop in building nodes is really something. i wonder if that was rax fixing itself14:03
fungioh, nevermind. that was a graph glitch14:03
openstackgerritMerged openstack-infra/jeepyb: Be more careful about when to close pull requests
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs
*** dizquierdo has joined #openstack-infra14:05
*** esker has joined #openstack-infra14:06
openstackgerritMerged openstack-infra/jeepyb: Replace 'javascript' method/parameternames
*** sabeen1 has joined #openstack-infra14:06
openstackgerritMerged openstack-infra/jeepyb: Rename upload_rss and move non-swift-relevant parts into the main method
*** bswartz has quit IRC14:06
dhellmannfungi, clarkb, jeblair, mordred: ttx and I are working on client stable branches this morning, and we're going to need some help with gerrit acls. See to start14:07
openstackgerritMerged openstack-infra/jeepyb: Add global flag to control replication
openstackgerritMerged openstack-infra/jeepyb: Add global flag to control replication
*** Kennan has joined #openstack-infra14:07
*** Kennan2 has quit IRC14:08
fungidhellmann: ttx: cool--should we also be cleaning up the refs/heads/proposed/* sections in some of those for better consistency? also what is our proposed/* branch control plan these days? i'd love to discuss dropping that section from the all-projects global acl and leaving it to the individual repo acls14:09
*** stevemar has joined #openstack-infra14:09
dhellmannfungi: I don't think we're using the proposed branch namespace for clients, but I'd want to verify with ttx14:10
dhellmannand I'd be happy to clean up the files I'm already touching. same patch, or a second one?14:11
*** zz_jgrimm is now known as jgrimm14:11
fungidhellmann: that was my thought too. let's do the same patch--this is a release management administrivia change for official projects, so making them consistent is very much on topic14:11
dhellmannfungi: wfm, I'll start updating14:11
* mordred stops reviewing patch14:11
dhellmannfungi: remove the "proposed" blocks entirely?14:12
fungidhellmann: yep, unless ttx objects14:12
fungii suspect they were just cargo-culted from the api server acls14:12
*** markvoelker_ has joined #openstack-infra14:13
fungisdague: yep, as i suspected rax-dfw is still in a bad way... 180 nodes (our entire quota there i think) is stuck in delete14:15
fungisdague: for reference
fungiseems to still be ongoing14:16
*** markvoelker has quit IRC14:16
*** scheuran has quit IRC14:17
*** tonytan4ever has joined #openstack-infra14:17
*** bswartz has joined #openstack-infra14:17
ttxfungi: you can drop proposed/* from libs ACLs14:18
mtreinishmordred: oh too bad, I was hoping you'd get to:
ttxI'd rather keep it in all-projects since that's a way to say "if you create a proposed branch, releasemanagement will own it"14:18
ttxbut we could certainly trickle that down if necessary14:19
*** achanda has joined #openstack-infra14:20
fungittx: well, part of the issue is that release management doesn't own it. there are projects overriding it in their acls to give control to their .*-milestone teams14:20
*** sigmavirus24_awa is now known as sigmavirus2414:20
mordredfungi: awesome. so I'm guessing that their folks would probably be less interested in runnign manual delete cleanups for us if they're in an incident14:20
*** ujuc_ has joined #openstack-infra14:21
*** ujuc_ has quit IRC14:21
fungittx: for example
ttxfungi: hmm, good point14:21
ttxI thought it was marked non-overfridable14:21
mordredjohnthetubaguy: just on the off chance you're around and feel like ignoring your employer and helping us14:21
fungittx: nah, whatever's in the repo-specific acl for a particular ref pattern overrides the equivalent pattern from all-projects14:22
*** dizquierdo has quit IRC14:22
*** marcusvrn1 has joined #openstack-infra14:22
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs
dhellmannfungi, mordred, ttx: updated ^^14:23
fungittx: looking at the nova acl, i think what they currently have is nova-core and nova-milestone granted control over refs/heads/proposed/* though that's worth confirming it's overriding in the direction i believe it's designed to14:23
dhellmannfungi: while you're reviewing my acl changes, I think I addressed your comments on the ones for oslo, too:
*** dkranz has joined #openstack-infra14:24
*** achanda has quit IRC14:24
mordredjohnthetubaguy: ^^ we have 180 nodes in deleting at rax14:24
mordredjohnthetubaguy: (remember a while back when I told you you were going to become our go-to guy for rax issues? :) )14:24
*** marcusvrn has quit IRC14:24
fungidhellmann: thanks. checking both now14:24
dhellmannfungi: ty14:25
fungittx: anyway, i don't think we need to change up the refs/heads/proposed/* section in all-projects for now, but post kilo release we should revisit streamlining this14:25
ttxdhellmann: +114:26
ttxfungi: ^14:26
dhellmannfungi: ++, I'm happy to help with that14:27
*** sarob has joined #openstack-infra14:27
fungittx: dhellmann: main reason being any special ref patterns we make exclusive in the all-projects global acl means that much more complication and confusion for stackforge and loosely-affiliated official projects who happen to have branch names colliding with that14:28
fungiand also gerrit's acl inheritance rules are a little opaque, so the less we expose people to that the better in my opinion14:28
johnthetubaguymordred: hey, I should try get some folks to look into things for you14:28
dhellmannfungi: that makes sense14:29
openstackgerritMerged openstack-infra/shade: Change Ironic node lookups to support names
johnthetubaguymordred: stuck in a meeting at this second, should be out soon14:29
ttxfungi: agreed14:29
fungijohnthetubaguy: is that meeting in dfw? are you sure it isn't just pending deletion since early yesterday? ;)14:29
johnthetubaguyfungi: I hurd some folks restarted things in DFW, I suspect thats related…14:30
fungicool--here's hoping14:31
*** armax has joined #openstack-infra14:31
*** dboik_ has joined #openstack-infra14:32
johnthetubaguyfungi: i am snug at home in the UK sadly, I guess its DFW where the issue is right?14:32
fungijohnthetubaguy: yeah. it's the issue posted at status.rackspace.com14:33
*** zz_dimtruck is now known as dimtruck14:33
dhellmannfungi, ttx: I wonder if we should consider encouraging core teams to think of stable-maint as their responsibility, too, and just stop having separate stable teams. in my (limited) experience, we usually end up with much the same teams anyway (that's why Oslo decided to go the route we chose)14:33
*** achanda has joined #openstack-infra14:34
ttxdhellmann: so the trick there is... we tried that and it failed14:34
ttxdhellmann: we need to check that core people actually understand the stable branch policy14:35
dhellmann"if it's everyone's job, it's noone's job"?14:35
ttxa surprisingly little number of them do14:35
dhellmannah, yeah14:35
sdaguedhellmann: yeh, the thing is, very different people care about stable 'products'14:35
fungidhellmann: i think part of the concern was that not enough of the core reviewers on some of the larger core teams paid attention to the branches changes were being proposed to, and didn't know (or want to know) the subtleties of the stable backport workflow and separate review criteria14:35
*** dkranz has quit IRC14:35
ttxI'm also all for splitting "core" as a single status14:35
ttxi.e. core reviewing != stable fitness judgment call14:36
dhellmannok, those arguments make sense14:36
ttxso the groups can certainly overlap14:36
ttxbut being the same is counterproductive14:36
fungiwe could rename them after the branches over which they have approval. "master reviewers" totally doesn't sound like an elite group to me ;)14:36
*** Ala has quit IRC14:36
*** emagana has joined #openstack-infra14:36
*** dboik has quit IRC14:36
ttxfungi: "HEAD reviewers" maybe :)14:36
*** markvoelker has joined #openstack-infra14:36
fungittx wins!14:36
ttx"trunk reviewers" sounds dirty14:37
*** dizquierdo has joined #openstack-infra14:37
fungii get that terminology matters, but i think we're going to just end up playing renaming games forever because the truth is that any time you have a specific set of permissions delegated to a specific group of people and the membership of those groups is published, they will inevitably be viewed as "elite" groups by external parties. no amount of calling everyone janitors and drudges will fix that14:38
*** markvoel_ has joined #openstack-infra14:38
*** russellb has quit IRC14:38
fungithough power to the janitors and all that14:38
*** hdd has quit IRC14:39
sdaguehonestly, the naming part isn't important here.14:40
*** markvoelker_ has quit IRC14:40
openstackgerritMerged openstack-infra/shade: Make cache key generator ignore cache argument
sdaguedhellmann: also, I think that the ability/desire of a team owning their stable branch largely has to do with how big the influx is on master14:40
*** markvoel_ has quit IRC14:41
fungisplitting various roles and responsibilities is always an option. we do it all the time in a lot of places in openstack already14:41
BobBallIf you're suggesting naming reviewers after the branch then surely 'stable reviewers; implies that the rest are unstable14:41
dhellmannyeah, I'm fine with leaving them split, I was just looking at that as a way to reduce the special casing14:41
*** russellb has joined #openstack-infra14:42
*** markvoelker has quit IRC14:42
*** marcusvrn has joined #openstack-infra14:45
*** sarob has quit IRC14:45
fungidhellmann: i think the big-tent scaling solution though is to not create more centralized groups. decentralize and let a thousand stars blossom, or something like that14:45
*** achanda has quit IRC14:45
*** sks has quit IRC14:45
fungidoesn't mean they can't be split roles to multiple groups, just should avoid making those centralized groups over multiple projects when avoidable14:45
*** otter768 has joined #openstack-infra14:46
*** marcusvrn1 has quit IRC14:46
dhellmannfungi: yeah, I'm trying to balance the self-service model with making sure stable-maint-core has permission to do things in repos in emergencies (or even just in cases like today)14:47
*** marcusvrn1 has joined #openstack-infra14:47
mordredso - since we can't encode that exactly in gerrit14:48
mordredwhat if we encode it as a unittest on things in the openstack dir of the acls dir?14:48
*** dboik_ has quit IRC14:48
mordredlike, "everything in openstack/ must include X team as a last-resort override team regardless of that team's policy"14:48
*** BharatK has joined #openstack-infra14:49
*** dizquierdo has quit IRC14:49
*** dboik has joined #openstack-infra14:49
*** marcusvrn has quit IRC14:49
mordredthat way people can define their own stable maint team, but we can have an uber team for dealing with things  like today14:49
fungimordred: we could probably add things to the current project-config-gerrit which does acl normalization checks14:49
fungier, what i mean is, we already do acl normalization checks in the project-config-gerrit job, so that seems like it would fit there and probably not be much extra code to implement14:50
mordredbecause there are times when a thing needs done and it's not a thing that needs 100 individual conversations14:51
*** otter768 has quit IRC14:51
fungidhellmann: ttx: both 173075 and 173892 lgtm now14:51
mordredand those times are few and far between - like, maybe one per cycle14:51
*** hdd has joined #openstack-infra14:52
openstackgerritMerged openstack-infra/shade: Invalidate volume list cache when creating
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: update to set defaultbranch
*** asselin has joined #openstack-infra14:53
*** annegentle has quit IRC14:54
*** patrickeast has joined #openstack-infra14:55
dhellmannfungi: thanks14:55
dhellmannmordred: I like that. if there's a way to just add stable-maint-core in the global acls file as fungi suggests, that would do it, too14:56
jeblairdhellmann: i finished about half of the change to do "file:" searching in gertty on the trip back from montreal14:56
dhellmannjeblair: nice!14:56
mordredlook! it's a jeblair!14:56
dhellmannjeblair: I tried to sleep on my flight, so no coding for me :-/14:57
jeblairdhellmann: (actually, i finished almost all of it, but i also think it needs to arrive with a fairly major schema change, otherwise the database has some major duplication, so i'm working on that now)14:58
*** kmartin has joined #openstack-infra14:58
*** Daisy has quit IRC14:58
dhellmannjeblair: typical developer. "It's done, I just need to make one more tweak..."14:58
jeblairdhellmann: i'm worried my database professor will find it and tell me it's not in the appropriate normal form14:59
dhellmannjeblair: all the cool kids are using denormalized schemaless databases these days, didn't you know?14:59
fungidhellmann: you mean filesystems?15:00
fungi(what's old is new again!)15:00
*** rwsu has joined #openstack-infra15:00
dhellmannfungi: you're giving away all the secrets that keep us ol' timers ahead of the curve15:01
*** dizquierdo has joined #openstack-infra15:01
* fungi puts his data in files. it's revolutionary!15:02
jeblairfungi: especially if the files are on spinning disk15:02
jeblairmordred: you use unified diff with gertty?15:02
*** jkraj has joined #openstack-infra15:02
mordredI was using it this morning, yes15:02
rcarrillocruzfungi: i notice the review manifest doesn't define mysql_user, i assume it uses some default15:03
mordred(still experimenting with how it works best for me)15:03
rcarrillocruz(hi btw)15:03
rcarrillocruzbut can't figure what that is15:03
rcarrillocruzgerrit2 as well?15:03
fungijeblair: i use our galaxy's rotation within the andromeda cluster to spin my storage these days. _really_ old-school15:04
*** duran has quit IRC15:04
jeblairmordred: i bet we could improve on what diff -u does.  we could probably have it always display the top line number in the 2nd line on the screen (where i put the filenames).  or even have it display the cursor line numbers as you move.15:04
fungircarrillocruz: you mean the openstack_project::review class?15:05
jeblairfungi: what do you think?  is it worth trying to get rid of the line numbers on the side in unified mode, so things really fit in 80 cols?15:05
rcarrillocruzyeah that one15:05
rcarrillocruzfungi: ^15:05
mordredrcarrillocruz: you want to look in templates/gerrit.config.erb15:05
mordredrcarrillocruz: in puppet-gerrit15:05
rcarrillocruzset in template15:05
mordredand yes - it's hardcoded because history15:05
*** xyang1 has joined #openstack-infra15:05
fungijeblair: i would love that, but it already manages to wrap gracefully on 70-something column lines and not wrap when it doesn't need to, so it's not terrible15:06
rcarrillocruzseeing so many hardcoded things lately in manifests, yeah15:06
openstackgerritDoug Hellmann proposed openstack/requirements: Remove OpenStack library caps for liberty
*** thedodd has joined #openstack-infra15:07
*** belmoreira has quit IRC15:08
openstackgerritTristan Cacqueray proposed openstack-infra/project-config: Remove stable-compat-jobs for projects that have stable branchs
DuncanTDoes anybody know Daniel Berrange's IRC nick, please?15:09
fungircarrillocruz: well, we do try to hard-code things that shouldn't need changing and avoid unnecessary proliferation of class parameters15:09
fungiDuncanT: that's the thing... he doesn't hang out in irc15:09
DuncanTfungi :-( Thanks15:09
DuncanTAnybody know gerrymander reasonably well?15:09
fungiDuncanT: he's said in the past that if it didn't happen in e-mail then it didn't happen as far as he's concerned15:09
*** flaviosr has joined #openstack-infra15:10
*** soren has quit IRC15:10
DuncanTfungi: I'll email him if needed, thanks15:10
fungiDuncanT: i think harlowja_away may have done some things with gerrymander... now i don't recall for sure15:10
DuncanTHe's got a commit in it, good point, cheers15:11
*** markvoelker has joined #openstack-infra15:12
rcarrillocruzspeaking of class parameters hiera and things15:13
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin
mordredfungi: funny, I consider that if it wasn't mentioned in IRC it didn't happen15:13
mordredrcarrillocruz: yah?15:13
rcarrillocruzwhat you think about using hiera as an ENC?15:13
rcarrillocruzsite.pp is becoming hard to manage15:13
rcarrillocruzwhy not putting on hiera what service classes belong to each node15:13
rcarrillocruzthat will make code cleaner15:13
rcarrillocruzand allow for a better experience on deploying new infras15:13
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin
rcarrillocruzand testing infra itself15:14
mordredrcarrillocruz: I'm not sure I can fully visualize what that looks like15:14
fungircarrillocruz: i think the plan is to get rid of the monolithic site.pp in system-config anyway15:14
rcarrillocruzfungi: and what's the alternative15:14
mordredbut yeah - once we've moved to puppet apply, I think there are several thigns that can get more flexible15:14
fungircarrillocruz: that site.pp is basically an artifact of using a puppetmaster15:14
mordredrcarrillocruz: one we're doing puppet apply, we could have a top-level manifest per type if we wanted15:14
fungiso, yeah, moving to masterless puppet makes it unnecessary15:15
mordredrcarrillocruz: that is drastically more confusing to me15:15
jeblairrcarrillocruz: also, i believe nibalizer was looking into moving more information into hiera after is implemented15:15
mordredbut public hiera is a great idea15:15
rcarrillocruzyes, we've been talking about an in-tree hiera in gozer for a while15:16
rcarrillocruzone hiera for public conf15:16
rcarrillocruzanother one for secrets15:16
mordredyes. that's right15:16
mordredI'd rather get that finished before we start thinking about larger structural changes like the ENC thing you mentioend15:16
*** markvoelker has quit IRC15:16
mordredhowever, I do want to say -that if I have to understand a magical hiera directory structure to be able to infer what classes are on a node, I'm going to be unhappy15:16
*** _nadya_ has quit IRC15:16
rcarrillocruzhiera is a good thing, and it improves dramatically downstream consumption of infra15:17
rcarrillocruzright now automating an infra deploy is nightmarish, because of the hardocded things around and highly coupled manifests15:17
nibalizeryea i havent worked on an in tree hiera dir in a while15:17
nibalizerwe shoukd return to tgat15:17
rcarrillocruznibalizer: well, we'll meet next week15:18
nibalizermordred: did i see you grump a bit about fabric yesterday?15:18
rcarrillocruzproviding gozer shares a lot of things with infra, we can discuss and get back to it on that spec15:18
*** bswartz has quit IRC15:19
*** david-lyle has joined #openstack-infra15:19
*** david-lyle_ has joined #openstack-infra15:19
*** david-lyle_ has quit IRC15:19
yolandafrom my side as well, i'm finding very difficult to be on sync with upstream due to that hardcoded bits, generating a huge drift15:20
*** hdd has quit IRC15:20
*** Krinkle|detached is now known as Krinkle15:20
yolandaso improving isolation, moving all possible logic to modules, and add parameters (even if upstream has its sane defaults to avoid mess), is the path to follow, from my opinion15:21
jeblairyolanda: that's what we're doing, right?15:21
yolandajeblair, yes, but honestly is not easy sometimes15:21
jeblairnope it's not.  it's difficult and slow15:21
fungiwe have a large and complex set of systems and lots to do15:22
yolandai'm filing changes and changes for that, as well as Ricardo and other ones15:22
jeblairyes, and we're reviewing them and merging them rather quickly since this is one of our priorities15:22
openstackgerritFabien Boucher proposed openstack-infra/system-config: Move server class call outside of jenkins*.pp class
yolandajeblair, but sometimes is not easy as we have that policy that system-config is not reconsumable. Even if it's not, that concept of avoiding hardcoded things, moving all logic to puppet modules, should be on top15:23
dhellmannsdague: can you take a look at this change to uncap our libs in master?
rcarrillocruzjeblair: i get sometimes feedback that system-config is not downstream consumable when i want to refactor stuff up. Un-hardcoding things doesn't mean i want to use system-config to deploy my own infra (we will always need our very own system-config branch and repo), it helps to test things in infra when we need to fix bugs that imply several components or automation workflows15:23
openstackgerritJulia Kreger proposed openstack-infra/shade: Wrap ironicclient methods that leak objects
openstackgerritJulia Kreger proposed openstack-infra/shade: Add patch_machine method and operator unit test substrate
*** dizquierdo has quit IRC15:24
fungircarrillocruz: yolanda: well, the answer to that is not to make system-config more complex so that downstream can consume it. it's to split system-config up and/or move more of what's there down into the individual service modules so that we can maintain a better defined api around those things15:24
yolandafungi, that's what were are trying in the background15:25
jeblairrcarrillocruz: right.  we're changing this system one step at a time.  first split out puppet modules.  make their api's better.  then start working on building those up into system units.15:25
rcarrillocruzun-hardcoding things is not making them more complex, is making them more flexible15:25
jeblairrcarrillocruz: the openstackci module is an example of that:
openstackgerritFabien Boucher proposed openstack-infra/system-config: Move server class call outside of jenkins*.pp class
yolandaat the end i  don't care if parameters come from site.pp, hiera or whatever, but having that parameters exposed is a necessary thing15:25
johnthetubaguyfungi: mordred: have you got some instance uuids you are having a problem with right now?15:25
rcarrillocruzinteresting jeblair15:26
rcarrillocruzso that's some sort of a minimal reference design for Openstack CI15:26
fungircarrillocruz: more flexible is basically _always_ more complex. choices add complexity. taking choice away reduces complexity15:26
jeblairrcarrillocruz: so let's focus on implementing the parts of the plan we've already discussed and approved, and then take the next step from there.15:26
*** jcoufal_ has joined #openstack-infra15:26
yolandahow is status on that spec btw? i wouldn't mind to help15:26
yolandawe are on the same boat15:27
fungiwe acknowledge that to make these things more reusable downstream they need to be more flexible, but the complexity that brings needs to go into places where we can corral it and wrap api stability expectations around it, rather than just making system-config more and more of a rat's nest15:27
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Add all Sahara images build to check pipeline as non-voting
rcarrillocruzi don't intend to use system-config for a prod CI15:28
fungijohnthetubaguy: sure, just a sec. getting an example now15:28
*** jerryz has quit IRC15:28
rcarrillocruzi'd be happy if i could just put my args in site.pp/hiera15:28
rcarrillocruzfire puppet15:28
rcarrillocruzand more or less have something functional15:28
johnthetubaguyfungi: thanks15:28
*** stevemar has quit IRC15:28
jeblairyes, the efforts already in progress are all about moving things into appropriate API layers and giving us the flexibility to reorganize things15:28
jrollfungi: chance that these are stuck from the thing I mentioned yesterday?15:28
jeblairrcarrillocruz, yolanda: so please do pitch in on the public hiera and openstackci efforts15:29
*** tonytan4ever has quit IRC15:29
fungijroll: my assumption is that they definitely are. the rackspace status page claims this incident is still ongoing15:29
*** jcoufal has quit IRC15:29
rcarrillocruzwe will15:29
jrollfungi: oh, interesting, did not realize it was still a thing15:29
yolandaasselin, count with us to help, can we have a talk and see how can we collaborate?15:29
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: fix
rcarrillocruzso asselin15:30
rcarrillocruzif i'm not mistaken , we work for the same company15:30
rcarrillocruzand you are EMEA based15:30
asselinyolanda, in a meeting-4....with catch up after15:30
yolandak, thx15:30
rcarrillocruzk, will ping you later then15:30
openstackgerritMerged openstack-infra/reviewstats: Make local cache more permanent.
fungijroll: so i guess it got resolved according to that?15:31
jrollfungi: yeah, looks resolved15:31
*** harlowja_at_home has joined #openstack-infra15:31
jrollfungi: but maybe some stuff got stuck during that time that isn't getting unstuck15:31
*** tonytan4ever has joined #openstack-infra15:32
*** russellb has quit IRC15:32
*** dangers_away is now known as dangers15:33
notmynamettx: now that we've got all the ec stuff onto master, there's an opportunity for some branch cleanup. feature/ec_review has been merged to master and can be deleted. feature/ec IMO should be tagged for history and then deleted15:34
notmynamettx: (for swift)15:34
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin
*** sdake has joined #openstack-infra15:35
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: upgrade ec2api jobs
yolandajeblair, will you have a sec to talk about nodepool queues today?15:35
fungijohnthetubaguy: jroll: here's 111 uuids for instances we've been trying to delete for ~20 hours
jeblairyolanda: sure, but i haven't reviewed your patch yet.  do i need to do that first?15:35
yolandayes, you should15:35
johnthetubaguyfungi: cool, thanks, I will pass that on15:35
jeblairyolanda: i'll ping you when i do then15:36
yolandait's following the 2 queues approach , one for create/delete, but i want to know your feedback15:36
ttxnotmyname: ok. Note that while I can create feature branches, I can't delete them (don't ask). So you should ask for infra help on that15:36
notmynamettx: ok. that's why I asked in here. I wasn't sure who does that15:36
*** russellb has joined #openstack-infra15:36
notmynamejeblair: ^^15:36
notmynamejeblair: now that we've got all the ec stuff onto master, there's an opportunity for some branch cleanup. feature/ec_review has been merged to master and can be deleted. feature/ec IMO should be tagged for history and then deleted15:37
*** dboik_ has joined #openstack-infra15:37
clarkbnotmyname might be best for you to tag feature/ec first then we can remove the branches (easier to tage the correct sha1) that way15:38
*** Sukhdev has joined #openstack-infra15:38
fungijohnthetubaguy: jroll: spot-checking that list against what nova list reports, some of those uuids don't seem to correspond to an existing instance at all, while others are showing in an active state. i'll try to force nodepool to clean up what it can and see what lingers15:38
notmynameclarkb: I wasn't aware that I could push tags to gerrit for swift (just swift client)15:39
clarkboh right15:39
*** sdake_ has joined #openstack-infra15:39
clarkbnotmyname what name should the tag be theb?15:39
notmynameclarkb: help me, clarkb, you're my only hope ;-)15:39
notmynameclarkb: erasure_code_dev_history15:39
*** sarob has joined #openstack-infra15:39
*** dboik has quit IRC15:40
*** samueldmq has quit IRC15:40
*** plaurin has joined #openstack-infra15:42
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin
plaurinHello, in a jenkin job, how to I configure a nodepool to make devstack-gate build a multi node environnement?15:42
*** sdake has quit IRC15:42
clarkbI initially thought feature-ec-eol to be like stable tags when they eol but I think that may confuse people into thinking the feature was end of lifed15:42
clarkbnotmyname: erasure_code_dev_history wfm15:43
*** spredzy is now known as spredzy|afk15:43
notmynameclarkb: ok15:43
clarkbplaurin if you grep for multinode in openstack-infra/project-config you will see examplea15:43
clarkbplaurin what are you trying to set up?15:44
*** marcusvrn has joined #openstack-infra15:44
plaurina netvirt "ping" test using robot framework, I am currently working in the opendaylight jenkins sandbox. I am able to run devstack gate, but I want to add two jcloud instances and make devstack-gate build compute nodes in them15:45
*** amotoki has joined #openstack-infra15:45
fungihrm, nodepool is basically our replacement for the jenkins jclouds plug-in. which are you using? nodepool or jclouds?15:46
*** marcusvrn1 has quit IRC15:46
clarkbfungi I think this may be third party ci15:46
fungisounds like it to me, yes15:46
clarkbbut yes we dont use jclouds for multinode envs, we assume nodepool15:47
plaurinjclouds, but I don't mind if it uses nodepool15:47
*** mbacchi1 has joined #openstack-infra15:47
*** flaviosr has quit IRC15:47
*** dboik_ has quit IRC15:48
ttxfungi, jeblair: we have an issue with our stable/kilo testing. It doesn't seem to fall back to proposed/kilo when no stable/kilo is available15:48
*** dboik has joined #openstack-infra15:49
ttxThat seems to point to stable/kilo requirements changes testing against trove master, according to sdague15:49
*** thinrichs has joined #openstack-infra15:50
plaurinhow is the multinode triggered?15:51
*** thinrichs has left #openstack-infra15:51
openstackgerritMerged openstack-infra/system-config: Create a list to coordinate internships
*** garyk has joined #openstack-infra15:51
clarkbplaurin zuul triggers the job for us15:51
fungittx: that may need a patch to devstack-gate. checking now15:52
jeblairfungi, ttx: er....15:52
ttxfungi, jeblair: so it is likely that the proposed/kilo testing falls back to stable/kilo when no proposed/kilo is around, but not the other way around15:52
jeblairfungi, ttx: can we just create a proposed/kilo branch?15:52
ttxfor requirements ?15:52
jeblairer, sorry, stable/kilo for requirements15:52
ttxstable/kilo for requirements is what we have15:53
ttxI think someone said that it wouldn't work if we cut proposed/kilo. Wouldn't sync to stable/kilo already-cut branches15:53
jeblairoh, sorry, so we'd need stable/kilo trove15:53
*** cdent has joined #openstack-infra15:53
*** annegentle has joined #openstack-infra15:54
ttxSo currently we have stable/kilo branches for libraries. proposed/kilo branches for all RCs. stable/kilo branch for requirements.15:54
ttxproposed/kilo branch testing seems to work fine and uses stable/kilo stuff where available15:55
*** thingee has joined #openstack-infra15:55
ttxstable/kilo branch testing on the other hand does not seem to use propsoed/kilo stuff if no stable/kilo is available ?15:55
fungijeblair: ttx: confirmed we don't have kilo branches handled in devstack-gate yet. i can add them unless anyone knows of a change in flight to do that already15:56
*** sarob has quit IRC15:56
*** deepakcs has joined #openstack-infra15:56
ttxfungi: so this has nothing to do with proposed/stable, but more to do with kilo ?15:57
jeblairfungi: what do you mean?  i don't think we should have to add anything to d-g.15:57
*** mrmartin has joined #openstack-infra15:57
ttxfungi: no such change that I know of15:57
fungijeblair: all the branch logic for e.g. grenade ordering?15:57
*** MarkAtwood has joined #openstack-infra15:57
*** FallenPegasus has joined #openstack-infra15:57
fungiis that stuff determined automagically outside of devstack-gate now?15:58
fungiit looks like features.yaml has been updated for kilo at least15:58
jeblairfungi: yes, grenade needs some help15:58
jeblairttx: i'm uncertain that there's any kind of stable<->proposed fallback logic in either direction.15:58
ttxjeblair: there certainly was in previous releases :)15:59
*** bswartz has joined #openstack-infra15:59
*** sdake has joined #openstack-infra15:59
*** tqtran has joined #openstack-infra15:59
fungidepends on where the bug is. if this is just a problem with requirements rewriting (i haven't looked at the logs yet) then it may be something we added between juno and now16:00
*** sarob has joined #openstack-infra16:00
* plaurin looking for multinode examples in openstack jenkins configs16:00
*** unicell1 has joined #openstack-infra16:00
fungioh, hrm, no devstack branch yet?16:00
krtaylorasselin, sorry we ran out of time16:01
*** unicell has quit IRC16:01
ttxfungi: apparently not16:01
fungittx: jeblair: dtroyer: sdague: mtreinish: that error looks like we need a stable/kilo branch in the openstack-dev/devstack repo16:01
sdaguefungi: normally we don't cut that until release16:02
jeblairfungi: can you link to the error?16:02
sdagueit's not typically needed16:02
asselinkrtaylor, no problem, hope that was helpful16:02
ttxsdague: makes sense to have it now that there are */kilo branches everywhere though16:02
fungittx: jeblair: dtroyer: sdague: mtreinish: my bad, thrown off by benign error messages in the log. still looking16:03
*** sdake_ has quit IRC16:03
*** zz_ja is now known as zz_zz_ja16:03
*** derekh has quit IRC16:03
*** tqtran_ has joined #openstack-infra16:03
plaurinfungi: I think DEVSTACK_GATE_TOPOLOGY is what I am looking for16:03
*** emagana has quit IRC16:03
clarkbplaurin yes16:03
sdaguettx: well, except it means a lot more backporting. We delay the branch for a reason to fix up last bits before the release16:03
*** jistr has quit IRC16:03
jeblairttx, sdague: i'm having trouble understanding exactly what the error is here.16:03
ttxsdague: ack16:03
jeblairttx, sdague: can somene point me to the thing they think is wrong?16:03
sdaguejeblair: propose a change to stable/kilo requirements16:03
sdagueit is testing against master for the projects16:04
sdaguenot against proposed/kilo16:04
*** jistr has joined #openstack-infra16:04
jeblairsdague: i agree that is expected behavior for the current code.  why is there a stable/kilo for requirements instead of a proposed/kilo?16:05
fungittx: sdague: jeblair: are you sure?
openstackgerritMerged openstack-infra/devstack-gate: Set standard "swappiness"
*** jlanoux has quit IRC16:05
sdaguefungi: the clients have stable/kilo branches16:06
fungi"Switched to a new branch 'stable/kilo'" (for openstack/python-novaclient)16:06
sdagueyeh, see my link16:06
sdaguenova is running off liberty16:06
sdaguejeblair: right, that might be the fix16:06
jeblairsdague, ttx, fungi: so the new thing here is that we have cut stable branches for clients, and we have done so without stable branches of other things.16:06
fungioh, got it, so it's testing stable/kilo requirements changes against stable/kilo of anything that has that branch or master of anything which doesn't (even if it has a proposed/kilo branch)16:07
sdagueI'm just saying the current branch structure doesn't do the right thing16:07
ttxjeblair: yes16:07
*** luqas has quit IRC16:07
ttxjeblair: I'm fine calling it proposed/kilo requirements instead if that would still get the library tested correctly16:07
*** marcusvrn1 has joined #openstack-infra16:07
sdaguettx: you're also going to have to do that for client libraries, because otherwise it will fail the other way, right?16:07
ttxdepends on where and in what direction the stable <-> proposed fallback is done16:07
ttxbecause there is one16:08
ttxthere was always one16:08
fungiwondering if we've got a logic error in test-matrix.normalize_branch()16:08
sdaguefungi: could be16:08
jeblairttx, sdague: we have one level of fallback support in branch checkouts, so we can do either stable/foo->master or proposed/foo->master, but we have no support for stable/foo->proposed/foo->master.16:08
jeblairtest-matrix does not control this.16:08
sdaguejeblair: yeh, that's what I thought16:08
ttxweird that it worked before then16:09
*** jistr has quit IRC16:09
jeblairwe had not imagined that we would have all three in play at a single time; we previously only either had proposed or stable in use, but not both at once16:09
*** marcusvrn has quit IRC16:09
fungiokay, so changes proposed to stable/X branches of projects won't be tested with proposed/X branches of other projects, but the reverse is true16:09
*** emagana has joined #openstack-infra16:10
ttxjeblair: ack16:10
*** carl_baldwin has joined #openstack-infra16:10
jeblairfungi: i don't think the reverse is true...16:10
jeblairfungi: i don't think we'll end up with stable/foo and proposed/foo branches of different projects in the same test in any situation16:10
ttxfungi: if you're right, creating proposed/kilo requirements should solve it16:10
sdaguettx: during this window of release time we actually rely on being lucky a lot and that patch rate is slow16:10
*** harlowja_at_home has quit IRC16:10
ttx(which is what we might have done in juno)16:10
*** carl_baldwin has quit IRC16:11
jeblairsdague: and that "*/foo" and master are still mostly compatible16:11
fungiwhich does also complicate the suggestions we'd been making to the driver/plugin maintainers to just use stable branches when the servers they're integrating with are in rc on proposed branches16:11
*** carl_baldwin has joined #openstack-infra16:11
anteayait would be nice if what ever suggestions we make to driver/plugin maintainers was something they could rely on for a bit16:12
anteayawhatever those suggestions end up being16:12
fungisure. first we have to sort out what we're suggesting to ourselves i think16:12
anteayayes, that is what it sounds like to me16:12
*** markvoelker has joined #openstack-infra16:13
*** koolhead17 has quit IRC16:13
anteayais an inchannel conversation sufficient for that?16:13
*** dosaboy has quit IRC16:13
*** unicell1 has quit IRC16:14
*** markus_z has quit IRC16:14
ttxSo it looks like we have 3 options16:14
*** dosaboy has joined #openstack-infra16:14
fungiso... if we switch the requirements branch from stable/kilo to proposed/kilo we're going to get proposed/kilo branches of projects that have them tested with proposed/kilo requirements and master of projects that already have stable/kilo branches. conversely, changes to stable/kilo branches of the projects that have those will get tested with master requirements and master branches of projects which16:14
fungiare part of the integrated release rc model16:14
*** jistr has joined #openstack-infra16:15
* dhellmann waits to hear ttx's 3 options16:15
ttxstill processing what fungi just said16:15
*** sergsh has quit IRC16:16
ttxmight be only two options16:16
ttxoption 1 - switch to using stable/kilo everywhere16:16
jeblairfungi: i believe that is correct16:16
dhellmann1. Take up goat herding.16:16
dhellmann2. ?16:16
fungiproposed/kilo changes get tested with kilo requirements and liberty clients/libs. stable/kilo changes to clients/libs get tested with liberty requirements and api servers16:16
clarkbdhellmann make cheese16:16
dhellmannclarkb: ++16:16
*** annegentle has quit IRC16:16
ttxoption 2 - fix tooling so that it just ignores the * in */kilo16:16
dhellmannttx: yeah, I'm strongly in favor of your option 116:17
ttxMy original option 3 was to cut a proposed/kilo req branch, but apparently that won't solve it if I read fungi correctly16:17
anteayamorning zaro16:17
jeblairttx, fungi, sdague: if we are going to have stable and proposed branches live at the same time, i think perhaps we do need both stable and proposed branches on the requirements repo.16:17
dhellmannttx: that is also what I understood jeblair and fungi to be saying16:17
jeblairttx: ^ i think that's your option 316:17
*** ihrachyshka has quit IRC16:17
*** jistr has quit IRC16:17
ttxjeblair: oh, havbe /both/16:18
*** Swami has joined #openstack-infra16:18
clarkbjeblair how do you keep them synced?16:18
jeblair(i had typed it in but was reading fungi's reply; the prose doesn't make sense now, forgive me)16:18
fungii've been in favor of option 1 since we went through this about a year ago, and then six months ago. the problem with it is that we'll need to make control group changes if we want different people controlling that branch during the rc period than after release16:18
ttxOK, let's rule out option 2 as way too risky and painful16:18
jeblairclarkb: hard work.16:18
dhellmannyeah, syncing sounds like it would be a real pain16:18
clarkbwhy not just do stable -> proposed -> master fallback16:18
*** dboik_ has joined #openstack-infra16:18
ttxThe historical two issues with option 1 are:16:18
anteayaclarkb: I believe gerrit doesn't allow that option16:18
clarkbessentially the */kilo ignores *16:18
dhellmannfungi: I think having to change acls is ok, if we think we actually have to change acls16:19
jeblairso i just about have my head wrapped around the tech here, but i don't actually know why we have stable client branches now?16:19
ttxclarkb: that's option 216:19
clarkbttx right16:19
anteayaclarkb: if I read jeblair's notes from above correctly16:19
ttxclarkb: I fear that would be a bit complex to track everywhere16:19
*** markvoelker has quit IRC16:19
dhellmannjeblair: because we're capping requirements in stable branch test environments now16:19
clarkbjeblair because we use clients in servers and backward compat for that is hard16:19
ttxso 1a- we need different acls for stable/kilo and stable/* (* not being kilo)16:19
*** tqtran_ has quit IRC16:19
fungii guess the question is "why now rather than on release day?"16:20
jeblairfungi: yeah16:20
ttx1b- we convey the wrong meaning (the branch is not stable yet) -- we can survive that one16:20
clarkbjeblair so capping reqs with mappings to releases for the clients16:20
dhellmannwell, we need to get the caps in place before we release the other projects16:20
ttx1c- release tooling needs to change16:20
fungibasically doing it on release day means we're telling clients to freeze development between now and then, i guess16:20
dhellmannyeah, and we're already getting some push back because the oslo requirements in master are capped still, so I'd like to undo that16:21
fungibecause client/library master branches will need to remain compatible with anything that's using proposed/kilo until kilo release day16:21
jeblairi sort of imagined capping requirements would happen _after_ release and the branch was cut.16:21
*** dkranz has joined #openstack-infra16:21
ttxjeblair: unfortiunately some people want to release liberty libs before kilo final release16:21
*** dboik has quit IRC16:22
ttxsince liberty is "open" you know16:22
ttxso we can't freeze them16:22
asselinyolanda, rcarrillocruz hi16:22
fungithis may also just be a case of "sorry, that's too complicated for us to support, don't do that, wait your turn"16:22
dhellmannfungi: that's fair16:22
ttxLooks like the simplest is to just use stable/kilo everywhere16:22
*** emagana has quit IRC16:22
asselinyolanda, rcarrillocruz I'm in california. There are tasks in storyboard to track the effort:!/story/200010116:23
ttxDoes everyone agree that would solve the current issue and future ones ?16:23
dhellmannttx: what do we lose by doing that? we have proposed separately for some reason, but I don't think I ever got it?16:23
yolandahi asselin16:23
fungithe other minor objection to that option, as i recall, is some people might think we released kilo already if we have a stable/kilo branch in existence. but i'm not terribly concerned about that16:23
dhellmannttx: I think it does solve this problem16:23
ttxdhellmann: see 1a 1b 1c above16:23
*** dtantsur is now known as dtantsur|afk16:23
clarkbttx yup I think that solves it16:23
*** mpavone has quit IRC16:23
anteayaall I need is something to tell the plugin/driver maintainers, and acl docs to reference myself when reviewing patches16:23
*** jlibosva has quit IRC16:24
yolandaah, nice, i see them16:24
anteayaI don't know enough about release management to offer an opinion on the direction itself16:24
yolandawe are in the same direction as you, so let's work together16:24
fungidhellmann: proposed.* was and is currently used to make it so we don't have to switch who controls approval of changes to the branch at release time16:24
asselinyolanda, rcarrillocruz I have the log server done. Needs reviews really. This is the first one, so hopefully once everyone is on board and it's gets merged, the other parts should follow16:24
dhellmannttx: why different acls? to give $project-core greater access before the release?16:24
asselinyolanda, +116:24
ttxdhellmann: lesser access16:24
yolandagoing to check that reviews16:24
ttxyou wouldn't believe what people do with +As the day before release16:24
asselinyolanda, that's why we share the same topic: downstream-puppet :)16:24
ttxproposed/* is restricted to pTL + release managers16:25
ttxwhile stable/* is stable-maint team16:25
rcarrillocruzasselin: will check thx16:25
rcarrillocruzasselin: what timezone are you based btw16:25
yolandaasselin, yes, but i've also been focusing on improving system-config direclty16:25
ttxit was a risk management tool. We can emulate it using ACLs jumping16:25
rcarrillocruzah nm16:25
rcarrillocruzdidn't read16:25
ttxi.e. specifying stable/kilo specific ACLs and remove them on release day16:25
asselinrcarrillocruz, yolanda, so the spec focuses on the modules needed by 3rd party ci. That's really to limit scope & a starting point.16:25
dhellmannttx, fungi : we can control that with a single change in the global acls file, right? by adding a stable/foo that sets it to just stable-maint-core? or do we actually want it to be $project-stable-maint?16:25
fungidhellmann: yeah, we had something like the first thing, and now it's the second thing16:26
anteayacan acl files reference an acl file? can you nest them?16:26
*** dustins has quit IRC16:26
fungidhellmann: we _could_ control it by changes to group membership, or by changes to per-repo acls16:26
ttxdhellmann: no we want release managers + ptl to control the +A16:26
ttxfungi: feels like it's the simplest option16:27
fungianteaya: they can "inherit" from another specific acl16:27
dhellmannfungi, ttx: this seems like something we could script, to make toggling the settings easier next time16:27
anteayafungi: usually from the global acl file, yeah?16:27
ttxalthough we have a boatload of proposed/kilo changes in gerrit16:27
jeblairttx, clarkb: we _can_ add multi-level fallback to the git checkouts if we need to.  let's make the decision based on what we think is best for humans -- wether we think it's better to have the multi-tier system or not.16:27
*** marcusvrn has joined #openstack-infra16:27
asselinyolanda, yes, and some of those improvements are needed to even be able to split out16:27
rcarrillocruzasselin: that's ok, providing we have something working and tested we can then build things on top of it16:27
fungittx: on the up-side, if we want to do it via group membership, there's an api which can be used to update the groups en-masse but we'd probably need to switch to a more consistent group name for all of them to support that easily16:28
yolandaasselin, looking at logserver, so from here we also have another needs, that is , pointing to private git instead of git.openstack16:28
fungianteaya: yep16:28
yolandawill you accept to parameterize that?16:28
fungianteaya: but you can specify a different acl, and chain them that way16:28
asselinyolanda, yes, as a follow-on patch16:28
clarkbI was always pro single stable/kilo branch but appreciate the acl issues with that16:28
anteayafungi: oh16:28
*** ociuhandu has quit IRC16:28
*** dannywilson has joined #openstack-infra16:28
yolandaok, don't want to block these ones with -1, so let's push them and we can start working based on that16:29
asselinyolanda, I'm trying to keep the original as close to the original.16:29
yolandait's really cool16:29
ttxjeblair: multilevel fallback would be proposed->stable->master and stable->proposed->master16:29
ttxto cover all the cases16:29
rcarrillocruzasselin: i've been working on a tool to bring up a system-config from scratch, it does the provisioning instances with ansible, and then orchestrates the config management bits running puppet16:29
rcarrillocruzwhat you think about the scope of this?16:29
openstackgerritKhai Do proposed openstack-infra/project-config: Make gerritbot report changes from Gerrit 2.10.2 branch
*** marcusvrn1 has quit IRC16:29
asselinyolanda, yes, it's easier from a review point of view to separate them out16:29
rcarrillocruzwould you say the provisioning is a separate project, or you think it would be useful to put in on yours16:29
fungittx: jeblair: clarkb: sdague: i wonder if we could simplify it by special-casing stable/ and proposed/ and assuming that any given repo can only have one at a time but not both?16:30
jeblairclarkb, fungi, ttx: yeah, so maybe in an ideal world, we have a single stable/branch and hide the acl complexity with a tool?  does that make things easiest for humans to understand?  it violates ttx's 1b -- sends the wrong message about the release, but i think most folks think that's minor?16:30
asselinrcarrillocruz, I don't know enough about provisioning to comment.16:30
ttxfungi: that is for sure. Never both16:30
jeblairttx: i would expect only "stable->proposed->master".  do we need "proposed->stable->master" ?16:30
*** _nadya_ has joined #openstack-infra16:30
anteayajeblair: it sounds to me like if we have the multi-tier system, we can deal with many things in different states, which I believe would be easier for us than trying to get and keep everyone in the same state16:31
rcarrillocruzprovisioning in openstack itself btw, leveraging the new openstack ansible modules from mordred, shrews, TheJulia and others16:31
jeblairttx: oh. i see.16:31
dhellmannjeblair: yeah, I think the message issue isn't as big of a deal16:31
ttxwe can survive the message thing16:31
yolandaasselin, so i've also been working to move features out from system-config to modules, this could even simplify the openstackci module16:31
ttxespecially with us using stable/* now everywhere16:31
fungiso (proposed|stable)->master16:31
fungiif we can somehow work it so that the logic treats proposed/X and stable/X as equals16:32
rcarrillocruzasselin: do you use a particular topic on reviews for this work, so I can look at reviews and push changes with that same topic?16:32
ttxfungi: the question is when a change is proposed on requirements stable/kilo and pulls projects proposed/kilo16:32
*** emagana has joined #openstack-infra16:32
asselinyolanda, +116:32
ttxfungi: +116:32
asselinrcarrillocruz, downstream-puppet16:32
yolandasame as ours16:32
rcarrillocruzlet me see then16:32
ttxproposed/X and stable/X should be the same thing that can have two names16:33
jeblairttx: that seems harder to express16:33
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Make multinode first class with ansible
ttxPlan A is to renbame all branches to stable/kilo, somewhow move existing proposed/kilo reveiws to stable/kilo, fix all the ACLs to have stable/kilo ACL equivalent to proposed/kilo16:33
asselinrcarrillocruz, here are the one's I proposed:,n,z16:34
ttxPlan B is to somehow express in the tooling that proposed/kilo and stable/kilo are the same thing16:34
ttxChoice of plan depends how hard and risky it is to do plan B16:34
dhellmannwhat tool actually handles that checkout fallback behavior?16:34
jeblairi think we need further design on plan b if we want to pursue it16:34
dhellmannis that in a script, zuul, gerrit, something else?16:35
jeblairi don't think i fully understand it or agree that it would work16:35
ttxalso in plan A: have to rewrite release tooling, rewrite release doc, redirect everyone to stable/kilo testing16:35
ttxPlan A: No rocket science, just pain16:35
jeblairdhellmann: it's two things: shell script in devstack-gate and python code in zuul-cloner.  they do the same thing.  zuul-cloner will replace the shell script in devstack-gate shortly after kilo releases.16:35
fungidhellmann: devstack-gate in most cases (not sure if anything's using zuul-cloner for this purpose yet)16:35
ttxPlan B: potential rocket science16:36
*** vdo has quit IRC16:36
ttxjeblair, fungi: how about you evaluate the feasibility of plan B and we come back here to make a decision based on that ?16:36
dhellmannttx: ok, after I get some food I can help with the tooling changes for plan a16:36
ttxPlan A is defintely doable16:36
yolandaasselin, what about project-config, to bring up all the projects and jobs, do you have any thoughts on how to incorporate that?16:36
ttxso if plan B is to complex / risky / impossible, let's do plan A16:37
*** thingee has quit IRC16:37
asselinyolanda, nothing great16:37
jeblairttx: the thing i need to work out with plan b is whether we can implement it as a hierarchy, or if we need to do the (A|B)->master thing.16:37
fungiyeah, i need to go get lunch but can refresh my memory on how setup_project() in devstack-gate's is doing this16:37
asselinyolanda, right now, for 3rd party ci, I copy & override16:37
jeblairfungi, ttx: the implementation is not hard.  the logic is.16:38
fungiright, we have a lot of subtlety around it there already16:38
asselinyolanda, for example:
yolandawe also need some more services like a puppetmaster at this moment, some static, git servers...are we ok to propose there?16:38
fungiand we've seriously broken ourselves there in the past16:38
*** EmilienM is now known as EmilienM|afk16:38
clarkbasselin: yolanda IMO that module should be strictly what is required to run tests16:39
jeblair16:32 < ttx> fungi: the question is when a change is proposed on requirements stable/kilo and pulls projects proposed/kilo16:39
clarkbso zuul, nodepool, jenkins, logs16:39
ttxjeblair, fungi: Let's give it a bit more thought... I'll be back in a couple of hours to discuss that16:39
jeblairttx: ^ what do you think should happen in that example?16:39
ttxjeblair: I expect a change in */foo to be tested with */foo whatever that is16:39
ttxif none available, master16:40
fungiwe already have a change to requirements updating to look for proposed/X if stable/X doesn't exist for a given project, not sure if it's been merged yet... checking16:40
jeblairttx: but stable/foo is always preferred to proposed/foo, right?16:40
johnthetubaguymordred: fungi: I am seeming most of those instances as deleted now, has it resolved its-self?16:40
ttxjeblair: doesn't matter, they don't exist at the same time.16:40
ttx(for a given project)16:40
clarkbright so .*/kilo will only match a single branch on any project at one time16:41
dhellmann*that* says to me that using the name to indicate the status is the real issue16:41
fungijohnthetubaguy: yeah, it looks like explicitly deleting those has worked now16:41
ttxso proposed/foo -> proposed/foo, */foo, master16:41
ttxclarkb: yes16:41
*** patrickeast has quit IRC16:41
asselinyolanda, rcarrillocruz dropping off for a bit16:41
fungijohnthetubaguy: i suspect it was api calls with unexpected responses or something while the systems in that region were having trouble16:41
johnthetubaguyfungi: do you keep retrying delete if delete fails?16:42
*** asselin has quit IRC16:42
fungijohnthetubaguy: we do16:42
johnthetubaguyfungi: I thought so, hmm its odd it took so long to heal16:42
fungijohnthetubaguy: so in this case i'm not sure why nodepool didn't successfully retry deletes for these, but don't have time to look into it right now16:42
jeblairttx, clarkb, fungi: we need to express this in a generalized way.  we can't actually encode things like "stable/" and "proposed/" in the tooling itself.  so if we go this route, we need to define the rules that we want to have happen, then find a way to express those rules in a generalized way.16:42
johnthetubaguyfungi: ah, no worries16:43
*** dannywilson has quit IRC16:43
johnthetubaguyfungi: I suspect it was 500 errors, we can check the logs on our side if thats helpful when you need to dig into that16:43
*** unicell has joined #openstack-infra16:43
clarkbjeblair: yes I think the rule is that we have (potentially arbitrary) branch prefixes that prefix a unique release suffix16:43
ttxjeblair: I think we can. */foo -> */foo if available, master if not16:43
fungijeblair: well, we encode stable and proposed into so could in theory call out to that?16:44
*** markvoelker has joined #openstack-infra16:44
jeblairfungi: no, this has to be able to happen entirely within zuul-cloner16:44
*** dhritishikhar has quit IRC16:44
fungioh, right. devstack-gate is oldthink16:44
clarkbany prefix/suffix pairs will be treated as matching other prefix/suffix pairs if the suffix matches16:44
ttxclarkb: ++16:44
clarkb(this will likely also help cloud and gozer with their >1 kilo branches)16:45
jeblairif we can encode a hierarchy into that (even if we don't use it), that's probably easier...16:45
fungialpha order?16:45
ttxif you really want a hierarchy, make stable the "best" choice16:45
*** hughhalf has quit IRC16:45
ttxreverse alpha order16:45
*** unicell has quit IRC16:46
*** Guest32383 has quit IRC16:46
*** unicell1 has joined #openstack-infra16:46
fungiuser-defined ordering?16:46
ttxso that you could use prefix for versions and use the "latest"16:46
jeblairso 'match (.*)/(.*)'; try: 'stable/$2', 'proposed/$2', 'master'16:46
ttxjeblair: that works too.16:46
*** sarob has quit IRC16:47
*** sarob has joined #openstack-infra16:47
ttxgot to run, be back in 2 hours16:47
*** kgiusti has quit IRC16:47
*** hughhalf has joined #openstack-infra16:47
clarkbjeblair: ya I think that generally works. The one potential issue is feature branches16:47
*** otter768 has joined #openstack-infra16:47
*** afazekas has joined #openstack-infra16:47
clarkbbut if a feature branch overlaps with a release name then likely they should be treated the same16:47
*** dhritishikhar has joined #openstack-infra16:48
anteayado feature branch have consistent naming?16:48
clarkbotherwise it iwll fall back onto master which is how we currently test feature branches16:48
clarkbanteaya: they are all prefixed with "feature/"16:48
ttxyeah, retrospectively we should have done kilo/proposed16:48
ttxor foobar/feature16:48
jeblairif we're changing it anyway... :)16:48
*** MaxV has quit IRC16:49
* ttx runs away sceraming16:49
fungimatch (.*)/(.*) and from all listed branches $b matching (.*)/$2 try $b/$2 in a defined sort order (reverse alpha?) using the first match found, else use master16:49
ttxand screaming16:49
*** MaxV has joined #openstack-infra16:49
ttxfungi wins16:49
*** adreznec has quit IRC16:49
*** mrmartin has quit IRC16:50
*** mfink_ has quit IRC16:50
*** markvoelker has quit IRC16:50
*** tqtran_ has joined #openstack-infra16:50
fungithis assumes that, e.g., our local clone is up to date enough to know what remote branches are valid for that repo16:51
anteayaI am looking for a way to work the wilhelm scream into something:
fungi(or that we have some other source of information providing a list of valid branches for that project)16:51
* dhellmann goes for food, too16:51
jeblairthe etherpad has the current fallback logic16:52
*** otter768 has quit IRC16:52
jeblair(which is, of course, slightly more complex than we've been talking about.  we have the fallback branche, then we have overrides, then we have per-project overrides)16:52
*** koolhead17 has joined #openstack-infra16:52
*** dhritishikhar has quit IRC16:54
*** tqtran_ has quit IRC16:54
jeblairfungi: are you proposing that we examine the git repos before calling zuul cloner, or encode the sorted(refs/origin) in zuul-cloner?16:55
fungijeblair: examine the git repo to build the list of expected branch names16:55
*** e0ne has quit IRC16:56
jeblairfungi: but then we have to do git operations before calling zuul cloner, and it is supposed to do all of those for us16:56
fungijeblair: which seems like the only way around this if we explicitly insist on not hard-coding any prefixes into the tooling anywhere16:57
fungijeblair: i meant as a feature _of_ zuul-cloner16:57
jeblairfungi: i'm not proposing we encode the fallback logic in zuul-cloner, i'm saying we should create a way so we can specify it external to the tool16:57
*** david-lyle has quit IRC16:57
*** dustins has joined #openstack-infra16:57
fungioh! sure then your proposal sounds fine. it at first sounded like you were opposed to encoding these anywhere at all16:57
jeblairfungi: i put a sample command line in the etherpad16:57
clarkbjeblairs proposal makes sense to me and will accomodate feature branches too16:58
fungiyep, i see it there. i was confused as to why you were encoding branch prefixes into your sample command line since you seemed not to want any16:58
jeblairfungi: so i'm suggesting we have zuul-cloner accept arguments to match the branch, and then you can add additional arguments to specify the fallback branches using that match16:58
jeblairfungi: sorry, i meant i want the tool to be general, but to accept our project-specific logic as input16:58
fungii'm fine with that example syntax16:58
fungii thought you wanted devstack-gate itself to also not have to encode any branch prefixes16:59
*** bswartz has quit IRC16:59
fungibut putting them in devstack-gate's invocation of zuul-cloner seems reasonable16:59
anteayaI think I even understand what is going on16:59
jeblairyeah, and possibly later with a zuul cloner jjb macro (or zuulv3 equivalent) so you can use it without having to think too hard17:00
fungii need to go get lunch, but will catch up on the rest of the conversation when i get back17:00
anteayahow does this translate to what driver/plugin maintainers should do, or are we too early in the conversation for this17:01
anteayashould we wait until this is built?17:01
pc_mlifeless: hi17:02
anteayaas a for instance a split out neutron driver testing itself with latest neutron17:02
clarkbanteaya: if they use zuul-cloner then they can use either proposed/foo -> stable/foo or just go straight to stable/foo and it should just work for them17:02
*** unicell1 has quit IRC17:02
*** unicell has joined #openstack-infra17:02
anteayaclarkb: can I help with documenting that anywhere as a series of steps?17:03
clarkbanteaya: I think we should have something more concrete first (like a zuul-cloner change)17:03
anteayafair enough17:03
clarkbotherwise our docs may not be very correct17:03
* anteaya goes back to ansible things17:03
jeblairsdague: as the designer of half of the current fallback logic, can you look at and make sure it looks sane?17:06
*** bswartz has joined #openstack-infra17:06
*** dhritishikhar has joined #openstack-infra17:06
jeblairsdague: it's in zuul-cloner terms, but "--project-branch tempest=master" == "OVERRIDE_TEMPEST_PROJECT_BRANCH=master" in devstack-gate terms, and likewise "--branch stable/icehouse" == "OVERRIDE_ZUUL_BRANCH=stable/icehouse"17:07
*** andrey-mp has joined #openstack-infra17:07
*** tonytan4ever has quit IRC17:07
*** dboik_ has quit IRC17:08
*** dboik has joined #openstack-infra17:08
*** david-lyle has joined #openstack-infra17:09
clarkbnotmyname: I am going to grab breakfast then will look at the swift tagging17:09
clarkbttx: are you happy with erasure_code_dev_history as the tag for the feature/ec branch that will be deleted?17:09
*** ianw has quit IRC17:10
*** amotoki has quit IRC17:10
*** david-lyle has quit IRC17:11
*** david-lyle has joined #openstack-infra17:11
*** andrey-mp has quit IRC17:11
*** samueldmq has joined #openstack-infra17:11
notmynameclarkb: thanks17:12
*** MaxV has quit IRC17:12
*** adalbas has joined #openstack-infra17:13
cineramaugh, still having problems with openstackid-dev.o.o. i can't use openstackid.o at the moment because my patch hasn't been applied there yet (i think)17:14
*** AJaeger_ has joined #openstack-infra17:15
anteayacinerama: I can't help you but posting the url of the patch you are referencing might help the next person who can17:16
*** dhritishikhar has quit IRC17:16
*** ssam2 has quit IRC17:17
cineramaanteaya: hey that's a good idea...let me see if i can find it17:17
cineramathis is the change:
*** sigmavirus24 is now known as sigmavirus24_awa17:19
anteayaokay so when fungi or clarkb comes back from eating or jeblair returns from his task that helps them evaluate if that code is on the server17:21
*** asselin has joined #openstack-infra17:23
*** dhritishikhar has joined #openstack-infra17:24
marcusvrnanteaya: hi! Is there a problem with gerrit? My Zuul stop watching Gerrit events...then I tried it manually (ssh -p 29418 gerrit stream-events) and it doesn't return anything... I can ping review.openstack.org17:25
anteayamarcusvrn: hi17:25
*** Krinkle is now known as Krinkle|detached17:25
anteayalet's walk through this one at a time, is there a problem with gerrit? not to my knowledge17:26
anteayayour zuul stopped watching gerrit, okay so you are asking about your third party ci account17:26
marcusvrnanteaya: yeah, sry! hehe17:27
anteayayou tried to run stream-events manually using your third party ci credentials and were unsuccessful17:27
*** sputnik13 has quit IRC17:27
anteayayou have tried pinging gerrit and were successful17:27
anteayalet's first look at the possiblity that your third party ci account credentials are incorrect or your account is disabled17:28
anteayawhat is the name of your account?17:28
*** davideagnello has joined #openstack-infra17:28
*** Krinkle|detached is now known as Krinkle17:28
*** tnovacik has quit IRC17:28
marcusvrnanteaya: I think it's not disabled, because I can "connect" via ssh ->
anteayapaste the output of running the stream-events command17:31
anteayayour wikipage was last updated today by you:
*** hdd has joined #openstack-infra17:32
marcusvrnI have updated the wikipage yesterday17:32
anteayaah sorry yesterday17:32
*** sputnik13 has joined #openstack-infra17:33
marcusvrnand about stream-events command, nothing is returned...17:33
anteayano mention of hds on infra, dev or third-party-announce mailing lists for this month17:33
anteayait just hangs?17:33
anteayamarcusvrn: don't control-C it17:34
anteayait if is open it means it is working and no events are being broadcast17:34
*** dhritishikhar has quit IRC17:34
anteayawe are in release candidate stage17:34
anteayathe only patches you should be seeing are bug fixes and you shouldn't see many of them17:34
*** achanda has joined #openstack-infra17:35
marcusvrnhmm.... but I have tried to recheck a patch, and nothing was shown in zuul/debug.log17:36
*** tonytan4ever has joined #openstack-infra17:36
*** kgiusti has joined #openstack-infra17:36
anteayain your zuul/debug.log?17:36
*** harlowja_away is now known as harlowja17:36
armaxfungi, sdague, dtroyer, jeblair is there a plan to switch to using/storing local.conf instead of localrc in the OpenStack infra job?17:36
anteayamarcusvrn: what patch?17:37
armaxor any dsvm jobs for that matter17:37
*** deepakcs has quit IRC17:37
*** e0ne has joined #openstack-infra17:37
*** sputnik13 has quit IRC17:37
*** Sukhdev has quit IRC17:38
adam_ghi, a failing job accidentally slipped into stable/juno gate pipelines earlier this week, anyone have a second for a trivial project-config review that removes it and unblocks them?
anteayamarcusvrn: I don't see that jenkins picked it up either17:39
anteayamarcusvrn: has your stream-events posted any output back to you yet?17:40
AJaeger_anteaya: I already +2ed 173319...17:40
anteayaAJaeger_: does this have anything to do with what I am talking about with marcusvrn?17:41
AJaeger_anteaya: no - that was about adam_g. Sorry for interrupting.17:41
*** SumitNaiksatam has quit IRC17:41
anteayaAJaeger_: oh okay17:41
marcusvrnanteaya: nothing yet17:41
ttxclarkb: happy17:42
anteayamarcusvrn: okay, so stream-events I can help with or try to, I don't run zuul myself so I have no idea what should or should not be in the logs17:42
adam_gAJaeger, thanks for that.17:42
anteayamarcusvrn: I jsut started stream-events myself and also have nothing so far17:43
*** SumitNaiksatam has joined #openstack-infra17:43
marcusvrnanteaya: ok, if I solve the stream-events problem, I guess zuul will work too17:44
harlowjaclarkb so talked to smoser , what needs to happen to get to get pushed into stackforge (only on the 0.7.x branch)17:44
harlowja' Able to merge. These branches can be automatically merged. ' (so shouldn't be a problem)17:44
marcusvrnanteaya: I can recheck another patch to check if stream-events will posted some output17:45
*** duran has joined #openstack-infra17:45
mordredharlowja: one of us will need to merge it in to a repo locally and force-push it up to gerrit17:45
harlowjamordred cool; hopefully not to much of a pita :-/17:46
mordredharlowja: I can help with that ... but before I do, the other branches are good, right?17:46
*** [HeOS] has quit IRC17:46
*** _nadya_ has quit IRC17:46
harlowjamaster/head is just gonna be the home of 2.0 so its fine17:46
harlowjaand there are no other branches, so ya :-P17:46
harlowjaall good afaik17:46
anteayamarcusvrn: okay, when you leave a recheck is recheck the very first word in the first line starting in the first space of the comment?17:47
marcusvrnanteaya: just rechecked and nothing happens (
*** signed8bit has joined #openstack-infra17:47
jeblairanteaya, marcusvrn, fungi, clarkb, mordred, pleia2: i think all our ssh stream threads are stuck again17:47
*** markvoelker has joined #openstack-infra17:47
anteayajeblair: ah17:47
AJaeger_yeah, something is strange with gerrit - there are no new events going to the gate queue currently17:47
anteayain which case thank you marcusvrn for alerting us to that17:47
*** johnthetubaguy is now known as zz_johnthetubagu17:48
jeblair has merged17:48
jeblairi would like to deploy it so that we can debug this problem17:48
marcusvrnanteaya: np17:48
anteayajeblair: I am in favour of that action17:49
*** marun has joined #openstack-infra17:49
mordredharlowja: you want the 0.7.x branch to be what is in the contents of your 0.7.x-fixed branch, right?17:49
mordredjeblair: awesome17:49
mordredjeblair: I am also in favor of that action17:49
harlowjamordred ya17:49
*** e0ne is now known as e0ne_17:50
*** SumitNaiksatam has quit IRC17:50
*** erlon has joined #openstack-infra17:50
*** Longgeek has quit IRC17:51
*** adalbas has quit IRC17:51
*** markvoelker has quit IRC17:52
*** mpaolino has joined #openstack-infra17:52
jeblairclarkb, fungi, mordred, pleia2, zaro:
clarkbmordred: harlowja no, cloud init needs to merge it then give us the repo to push imo17:53
clarkbmordred: I don't want to be responsible for figuring out what they want their git tree to look like17:53
harlowja ?17:54
harlowjaor do u want more than just a branch?17:54
harlowjawhich i can do also17:54
jeblairclarkb, mordred, zaro: that build includes
mordredharlowja: nope. that's fine17:54
jeblairwhich, if i understand zaro correctly from the meeting on tuesday, means we're about to break launchpad integration17:54
mordredclarkb: they did. it's harlowja's branch17:55
clarkbmordred: then what needs merging?17:55
jeblairmordred, harlowja, clarkb: any chance we could shelve this conversation for a minute17:55
harlowjaseems like nothing anymore17:55
harlowjamordred i think did it?17:55
clarkbjeblair: sure, I +2'd the war update17:55
mordredjeblair: yah17:55
clarkbjeblair: that brings the change you link and better connection logging which sounds good to me17:55
*** sarob has quit IRC17:55
jeblairclarkb: yeah, i'm saying that if i follow zaro correctly, will break gerrit-lp integration in jeepyb17:56
mordredjeblair: there is a correspondong jeepyb change which adds consumption of those parameters ...17:56
jeblairmordred: has it merged?17:56
clarkbwouldn't --change-owner just be ignored by argparse?17:56
mordredI just reviewed it this morning17:56
clarkbor is argparse going to complain?17:57
mordredI did not raise it as important because it mentioned 2.10 and not 2.817:57
clarkbI guess I don't know what argparse does in that situation17:57
mordredso it seems landing the jeepyb change before any gerrit changes would be safe17:57
*** ivar-lazzaro has joined #openstack-infra17:57
*** rfolco has quit IRC17:57
clarkbmordred: I agree, I can approve 173571 now17:58
*** andreykurilin has joined #openstack-infra17:58
clarkbexept it depends on which is wip17:58
*** openstackstatus has quit IRC17:58
clarkbzaro: ^ can you remove the dep there?17:58
mordredclarkb: argparse does error on unknown arguments17:58
mordredclarkb: it is?17:59
mordredclarkb: it does not show as dependent to me17:59
sdaguejeblair: I think  makes sense to me17:59
clarkbmordred: its a depends on17:59
mordredah - well, we can pull that from the commit message ourselves17:59
*** openstackstatus has joined #openstack-infra17:59
*** ChanServ sets mode: +v openstackstatus17:59
zaroclarkb: was getting this in the log:
mordredyes - argparse will error on unknown params18:00
jeblairzaro: i don't think we shoud have merged that change to 2.8.418:00
*** carl_baldwin has quit IRC18:00
jeblairi don't really know why it was necessary18:00
sdaguejeblair: are we sure it will stack up branches in order and keep them correctly? I think that's my only concern with fallback branch as multi option instead of comma18:00
clarkbmordred: +218:00
jeblairsdague: i need to focus on gerrit things for a few mins, will get back to you18:01
jeblair#status alert Gerrit has stopped emitting events so Zuul is not alerted to changes.  We will restart Gerrit shortly to correct the problem.18:01
openstackstatusjeblair: sending alert18:01
sdaguejeblair: no worries18:01
mordredin any case, the depends-on was backwards18:01
mordredthe jeepyb change was to allow jeepyb to not break when the new params exist, then the gerrit change adds the params ... zaro for next time you have a similar thing18:02
zarojeblair: the purpose was so we could pass owner info back to LP18:02
zarojeblair: i believe there was a request in LP for that.18:02
-openstackstatus- NOTICE: Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem.18:02
*** ChanServ changes topic to "Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem."18:02
mordredharlowja: your 0.7.x branch should be correct now18:03
*** vlaza has left #openstack-infra18:03
jeblairmordred: we need jeepyb deployed before we restart gerrit
*** patrickeast has joined #openstack-infra18:03
mordredjeblair: hrm. so - want me to manually deploy a copy of it with that patch applied?18:03
*** signed8bit is now known as signed8bit_ZZZzz18:04
*** annegentle has joined #openstack-infra18:04
jeblairmordred: i think you need to force-merge that change, and i need to force-merge the gerrit change, and then perhaps if we run ansipuppet after checking out system-config on puppetmaster it will do both?18:04
mordredjeblair: ok18:04
mordredjeblair: I have force-merged the jeepyb change18:04
clarkbjeblair: yes, if you don't use the script then you need to update system-config directly18:05
*** e0ne_ is now known as e0ne18:05
openstackstatusjeblair: finished sending alert18:05
mordredjeblair: I've got ansipuppet when you're ready18:05
*** signed8bit_ZZZzz is now known as signed8bit18:05
jeblairmordred: okay, will you make sure the system-config repo is up to date?18:05
jeblairmordred: i have force-merged the puppet change18:05
jeblairmordred: so all yours18:05
*** ashleighfarnham has joined #openstack-infra18:06
mordred    Upgrade production gerrit to
mordredis the latest there now18:06
zaromordred: the 'kind' param is not in 2.8 will jeepby ignore that?18:07
mordredzaro: yes18:07
*** bswartz has quit IRC18:07
mordredzaro: when you add it to argparse, it will be fine if it doesn't recieve it18:07
mordredzaro: because you did18:08
mordred    parser.add_argument('--kind', default=None)18:08
fungiokay, back full of fish and catching up18:08
mordredthat means args.kind will be None18:08
mordredwhich is fine, because we don't consume it anywhere18:08
*** _nadya_ has joined #openstack-infra18:08
*** mpaolino has quit IRC18:09
fungicinerama: i got openstackid-dev.o.o back on track this morning (tipit moved one of its databases) and was able to authenticate to it successfully thereafter. what specifically is it doing for you?18:09
cineramafungi: let me try again18:09
clarkbmordred: how is puppet doing?18:10
*** sigmavirus24_awa is now known as sigmavirus2418:10
*** david-lyle has quit IRC18:10
cineramafungi: heh. now it works.18:10
mordredclarkb: running18:11
*** mayurig has joined #openstack-infra18:11
jeblairmordred, clarkb: from inspecting the filesystem, the new gerrit war is in place.18:11
_qbhello, I'm trying to estimate how much hardware I need to throw at an environment to do third party testing.  it's daunting.  anyone want to help?18:13
mordredjeblair, clarkb: should be done18:13
*** dosaboy has quit IRC18:13
*** amitgandhinz has quit IRC18:13
anteaya_qb: we are dealing with a maintainance issue at the moment18:13
jeblairmordred: i see '--change-owner' in "update-bug --help"18:14
*** rfolco has joined #openstack-infra18:14
*** dosaboy has joined #openstack-infra18:14
jeblairmordred: ready for me to restart gerrit?18:14
*** tkelsey has quit IRC18:15
*** dannywilson has joined #openstack-infra18:15
mordredjeblair: yes18:15
fungiokay, caught up now. i seem to have missed all the excitement (hopefully!)18:15
jeblairoh, er...18:15
jeblairdoes puppet restart gerrit?18:15
anteayafungi: all but the cheering18:16
fungii have not seen puppet restart gerrit normally, though it may start it if it was stopped?18:16
clarkbI don't think it does18:16
* clarkb reads some puppet18:16
jeblairgerrit2   7803  0.0  0.0   4440   652 ?        Ss   18:06   0:00 sh -c /etc/init.d/gerrit stop; /usr/bin/java -jar /home/gerrit2/review_site/bin/gerrit.war init -d /home/gerrit2/review_site --batch --no-auto-start; /usr/bin/java -jar /home/gerrit2/review_site/bin/gerrit.war reindex -d /home/gerrit2/review_site18:16
fungiperhaps that's what you meant by restart though18:16
jeblairi think it has stopped it and is running a reindex18:16
jeblairso, er, i guess gerrit is down for the next 10 minutes :)18:17
fungiahh, yep18:17
anteayaoh my, that takes how long? 30 minutes?18:17
anteayaoh just 1018:17
*** mayurig has quit IRC18:17
jeblairwell, it's 10 minutes into it, so if it takes 15 we have 5 left18:17
clarkbit will run a /etc/init.d/gerrit start after it installs a new war18:18
clarkbbut I don't see it doing a stop18:18
anteaya_qb: can you join #openstack-dev and I can talk to you there18:18
clarkboh the stop in in the "init" exec18:18
clarkbthats not very intuitive18:18
clarkbjeblair: so yes it iwll restart gerrit for us18:18
jeblairit seems to have finished the index and not restarted18:19
jeblairit is possible i broke it with my service gerrit stop18:19
*** teran has quit IRC18:19
jeblairbut i also don't understand how ansipuppet could have returned while that was still runinng18:19
*** teran has joined #openstack-infra18:19
jeblairsomeone should look into this18:19
clarkbif the init script returned early that could happen18:19
jeblairon a test node18:19
jeblairstarting gerrit18:20
zarojeblair: it will take a while to start after the reinfex18:20
*** dosaboy has quit IRC18:20
jeblair2.8.4-17-ge391a2d is running18:20
clarkbits up an appears to be happy for me18:20
anteayawhat are we confirming, launchpad login?18:21
clarkbnotmyname: you were saying we should delete feature/ec_review as well? should that branch get a tag too?18:21
*** dosaboy has joined #openstack-infra18:21
clarkbanteaya: I opened a change and looked at my changes dashboard18:21
*** _qb has left #openstack-infra18:21
anteayaI can login with launchpad in chrome18:22
*** EmilienM|afk is now known as EmilienM18:22
zaroanteaya: that a change in gerrit puts an LP bug to 'in progress'18:22
*** SumitNaiksatam has joined #openstack-infra18:22
*** dosaboy has quit IRC18:22
zaroanteaya: but i think the bigger one is gerrit events are working again18:23
notmynameclarkb: ec_review has been merged to master, so it is accessible from there18:23
clarkbnotmyname: I see18:23
*** teran has quit IRC18:23
jeblair#status ok Gerrit has been restarted.  New patches, approvals, and rechecks between 17:30 and 18:20 UTC may have been missed by Zuul and will need rechecks or new approvals added.18:23
openstackstatusjeblair: sending ok18:23
*** dosaboy has joined #openstack-infra18:24
*** _nadya_ has quit IRC18:24
*** dosaboy has quit IRC18:24
*** dosaboy has joined #openstack-infra18:24
jeblairsoooo many git push tasks18:24
anteayamarcusvrn: open stream-events again and tell me what your experience is18:24
clarkbnotmyname: I am tagging ff7f7f474b634a921561aae1d0093a2881efcde4 with name erasure_code_dev_history and will push that momentarily18:25
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source"18:25
-openstackstatus- NOTICE: Gerrit has been restarted. New patches, approvals, and rechecks between 17:30 and 18:20 UTC may have been missed by Zuul and will need rechecks or new approvals added.18:25
clarkbnotmyname: can you double check that commit for me (it is head of feature/ec)18:25
marcusvrnanteaya: now it's working!! :)18:25
notmynameclarkb: ack ff7f7f474b634a921561aae1d0093a2881efcde418:26
jeblairsdague: yes, i think if you use 'append' with argparse you get the arguments in order18:26
*** tonytan4ever has quit IRC18:26
jeblairsdague: i think i like that because then we don't have to parse an argument that might already be a complex expression itself18:27
anteayamarcusvrn: wonderful, thank you for sharing your experience and alerting us to the problem, my gratitude18:27
openstackstatusjeblair: finished sending ok18:27
*** bswartz has joined #openstack-infra18:28
*** signed8bit is now known as signed8bit_ZZZzz18:28
*** sambetts has quit IRC18:29
sdaguejeblair: ok, sure, might be good to ensure that it's documented, because wrong order is going to cause very subtle bugs.18:29
jeblairsdague: ++18:30
jeblairincidentally, i think this solves a wikimedia problem where they don't have master branches18:30
*** e0ne has quit IRC18:30
jeblair(so the hardcoded 'master' fallback didn't always make sense to them)18:30
marcusvrnanteaya: you're welcome! thank you all for solving the problem so fast! :)18:31
clarkbnotmyname: tag pushed18:31
anteayamarcusvrn: :)18:31
notmynameclarkb: thanks18:31
clarkbmordred: I need to remove myself from project bootstrappers should I remove you?18:31
clarkbnotmyname: deleting feature/ec and feature/ec_review now18:31
notmynameclarkb: ack. looks good (tag)18:32
*** sambetts has joined #openstack-infra18:32
mordredclarkb: yes18:33
jeblairclarkb, fungi, ttx: okay, so i think plan B from earlier is doable as described in, and we can probably land the changes for it this week if we are in a rush.18:33
clarkbnotmyname: branches are deleted, you should be set18:33
notmynameclarkb: thanks18:33
clarkbmordred: done18:33
jeblairclarkb, fungi, ttx: for actual implementation, we can land the devstack-gate version of it first, just so long as we have a plan for zuul-cloner18:33
*** thingee has joined #openstack-infra18:34
clarkbjeblair: that seems reasonable since d-g is primarily what will affect kilo18:34
fungithat seems like a pragmatic path18:34
fungino objection18:34
AJaeger_jeblair: totally different topic, please put it in your queue and answer later. I'm signing of here but will read backlog tomorrow:18:34
AJaeger_jeblair: anteaya, fungi and myself discussed yesterday gerrit permissions a bit on this etherpad: . Should I clean this up and propose a patch somewhere? Question is where exactly...18:34
jeblairclarkb, fungi, ttx: so i think we can make the decision based on what we _want_ to do.  technically we are able to execute plan B with what we wrote in the etherpad, and we can execute plan A with tooling around changing acls/groups, etc.18:34
*** annegentle has quit IRC18:35
fungiAJaeger_: it should probably wait until after kilo since we're discussing shaking some of that up anyway18:35
*** annegentle has joined #openstack-infra18:35
fungiAJaeger_: though i'm okay with documenting it and then changing it in a few weeks18:35
*** eharney has quit IRC18:36
*** andreykurilin has quit IRC18:36
*** duran has quit IRC18:36
clarkbI do have a preference for a single branch for a release18:36
*** mrmartin has joined #openstack-infra18:36
clarkbbut I am not the person juggling acls and branches typically so defer to ttx's desire to have reviewer control around that18:36
fungijeblair: clarkb: ttx: i do still find the simplicity of not having a proposed/* branch attractive, but will defer to ttx on that18:37
jeblairclarkb: me too18:37
AJaeger_fungi: I'm happy either way, just want some guidance ;) Thanks!18:37
jeblairttx: so we might have a room full of people that would prefer planA but are happy to implement planB if it makes release people happy :)18:37
fungiAJaeger_: for example, _right now_ we're discussing possibly getting rid of one of the items you described there entirely18:38
*** ivar-laz_ has joined #openstack-infra18:38
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Make multinode first class with ansible
*** notnownikki has quit IRC18:38
AJaeger_fungi: ah, ok ;) Missed the start of it18:38
*** andreykurilin has joined #openstack-infra18:40
anteayaAJaeger_: and thanks for writing up the etherpad, very valuable for me, thanks18:40
*** ivar-lazzaro has quit IRC18:42
anteayaI have no opinion on what the thing is as long as ttx and dhellmann are happy and I know what to tell neutron driver/plugin maintainers as well as how to review acl patches18:42
*** tonytan4ever has joined #openstack-infra18:43
*** bswartz has quit IRC18:44
*** bswartz has joined #openstack-infra18:44
harlowjamordred thx much for the cloud-init repo /merge/push/update18:44
AJaeger_anteaya: I've learned a lot as well - and like to have a reference that I can point others to.18:44
anteayaAJaeger_: ++18:44
*** isviridov_away is now known as isviridov18:45
*** jcoufal_ has quit IRC18:45
*** isviridov is now known as isviridov_away18:45
*** sarob has joined #openstack-infra18:46
mordredharlowja: sure nuff!18:47
harlowjathe other question that smoser had that idk if he asked was is there anyway for to automatically push to (or gerrit could do this?)18:47
harlowjaso that cloud-init/cloud-init  is mirroring stackforge18:47
harlowjasmoser so there's your answer :-P18:48
smoserharlowja, thanks18:48
*** otter768 has joined #openstack-infra18:48
harlowjamordred do people usually setup there own little cron-jobs that do this then?18:49
fungijeblair: clarkb: ttx: also, as a thought experiment on simplifying the shift of power between rc and release, what if the people approving backports from master to fix rc bugs are the same people who have a vested interest in the state of the stable branch which emerges from that, and follow relatively similar guidelines as to what's worth backporting to it in both states?18:49
*** tqtran is now known as tqtran_afk18:49
mordredharlowja: I don't know of anyone who uses a second github repo that is different than the github repo that gerrit replicates to18:50
*** amitgandhinz has joined #openstack-infra18:50
harlowjamordred kk18:50
*** kgiusti has quit IRC18:50
harlowjasmoser ^18:50
dhellmannfungi: how does that differ from what we do now?18:52
*** andreykurilin has quit IRC18:52
*** duran has joined #openstack-infra18:52
*** otter768 has quit IRC18:53
*** hdd has quit IRC18:53
fungidhellmann: right now there are different groups of people approving changes to proposed/kilo branches vs those who will be approving backports to stable/kilo after release day18:53
fungidhellmann: i'm just suggesting that perhaps there might actually be a lot of overlap between those groups, and that maybe we can come up with one acl which covers both the rc period and post-release18:54
dhellmannfungi: ok, I guess I didn't realize that because in oslo it's largely the same set of people.18:54
openstackgerritRussell Bryant proposed openstack-infra/project-config: Add #openstack-neutron-ovn to gerritbot.
dhellmannfungi: like $project-stable-maint ? :-)18:54
fungidhellmann: sounds like i already have a data point in favor of my conjecture!18:54
ttxfungi: one aspect is that stable people didn't sign up for that task at all though18:54
*** ivar-laz_ has quit IRC18:55
dhellmannfungi: yeah, though we are trying to go the other way in oslo and say that oslo-core replaces oslo-stable-maint and we don't have 2 teams -- if the tools are made simpler by us implementing that with group membership in gerrit, we can do that, too18:55
*** tnovacik has joined #openstack-infra18:55
fungittx: sure, though how many of them are the same people?18:55
fungithat's more or less what i'm asking18:55
ttxSo whiel I could introduce the idea to them, i don't want to change the process to match the tooling18:55
dhellmanncould we figure that out by looking at the gerrit db?18:55
*** ivar-lazzaro has joined #openstack-infra18:55
funginot talking about .*-core vs .*-stable-maint, but rather .*-milestone vs .*-stable-maint18:56
*** baoli_ has quit IRC18:56
fungiyeah, we could run some numbers to identify overlap. i worry that it's all a bit ad-hoc so far though and might be hard to draw actual conclusions from across projects18:57
dhellmannyeah, we don't have a milestone team in oslo so I always forget about that18:57
ttxRC handling is completely different set of processes and procedures though. We can't just assume people inherit knowledge magically18:57
ttxIt's also more time-sensitive and therefore risky18:57
ttxwhich is why the current set of people is so restricted18:57
ttxcatching up on backlog18:57
*** ivar-lazzaro has quit IRC18:58
fungifair enough. sounds like it really does need to be a separate set between rc approvers and and stable approvers18:58
*** tsg_ has joined #openstack-infra18:58
*** ivar-lazzaro has joined #openstack-infra18:59
ttxfungi, jeblair, clarkb: if the plan takes until end of week to land, I think plan A is faster, and time is a factor here18:59
ttxfungi: at least for this release yes18:59
*** [HeOS] has joined #openstack-infra19:00
*** bswartz has quit IRC19:00
fungiso for plan a (there is no proposed, only zuul^H^H^H^Hstable), we need to decide if we want to go with new groups and/or twiddle acls or group membership at release time19:02
*** tqtran has joined #openstack-infra19:02
ttxI'm drafting plan A at bottom of
*** bswartz has joined #openstack-infra19:03
openstackgerritJulia Kreger proposed openstack-infra/shade: Add Ironic maintenance state pass-through
*** sabeen1 has quit IRC19:03
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: fix query for bug 1285323
openstackbug 1285323 in grenade "Services fail to shut down on the old side of Grenade" [Undecided,In progress] - Assigned to Sean Dague (sdague)19:04
ttxfungi: is step 2 of the plan doable ?19:04
*** eharney has joined #openstack-infra19:04
ttx(see bottom of
fungii think they'd need to be reproposed19:05
fungiit _might_ be doable with some database modifications during a gerrit outage, but i wouldn't guarantee it (or feel safe about doing it without a lot of testing)19:06
ttxWe have about 24 patches up at,n,z19:06
*** tqtran has quit IRC19:06
dhellmannI can spend some time resubmitting those when we have the right branches19:07
ttxI guess they can be manually reproposed19:07
*** soren has joined #openstack-infra19:07
fungiyeah, unique changes take target branch into account, so if you just try to push a new patchset but change the branch, you'll get an entirely separate change19:07
openstackgerritRussell Bryant proposed openstack-infra/reviewstats: Catch exception on exec_command()
ttxfungi: it's just that we can't remove proposed/kilo until all review there are abandoned, right ?19:07
*** luqas has joined #openstack-infra19:08
fungiafaik there's no gerrit feature for moving a change to another branch, nor for renaming an existing branch19:08
dhellmannare we talking about renaming the existing branch, or making a new one?19:08
ttxso if it takes us 4 hours to repropose stuff, both branches coexist19:08
ttxdhellmann: same thing19:08
ttxdhellmann: you can't rename a branch, you create a different ref19:08
ttxand delete the old one19:09
dhellmannok, that's what I thought19:09
dhellmannso we can make a new branch, resubmit against it and abandon them one at a time19:09
ttxI don't feel very comfortable with the idea of those branches coexisting for a long time, and I can't sign up to repropose them all in record time19:09
openstackgerritJulia Kreger proposed openstack-infra/shade: Update register_machine to use tasks
openstackgerritJulia Kreger proposed openstack-infra/shade: Update unregister_machine to use tasks
ttxIf we have volunteers to do the reproposing job in the coming hours, I guess plan A is doable19:10
lifelesspc_m: hi?19:10
ttxI'd like to close most of this today if possible. Traveling tomorrow19:10
ttxand meeting all day19:10
ttxfungi: if we abandon them and remove branch, can we still check out the proposed change as a step to repropose it ? Or is it gone ?19:11
dhellmannttx: my idea was to abandon them after they had been resubmitted19:11
fungiyou can still check them out and repropose them to another branch even if already abandoned19:12
fungibut whichever19:12
dhellmannok, well, that works too19:12
*** luqas has quit IRC19:12
fungiokay, i can propose the acl changes to add a refs/heads/stable/kilo section to each repo which duplicates the permissions from refs/heads/proposed/*19:12
fungiif this is the path forward19:12
ttxSounds like the simplest way19:12
ttxI'm +1 on it19:12
dhellmannI'll start pulling these patches down19:12
ttxdhellmann: ?19:12
openstackgerritJulia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through
dhellmannttx: ++ let's go with plan A19:12
fungijeblair: clarkb: any last-minute objections?19:12
*** stevemar has joined #openstack-infra19:13
*** vlaza has joined #openstack-infra19:13
ttxfungi: assuming we are 99.9% sure that will work :)19:13
dhellmannttx: should we try one project with a small number of patches first?19:13
ttxdhellmann: don't know how to test that "things are operating correctly"19:13
ttxsince the test that doesn't operate correctly still reports success19:14
dhellmannwell, sdague saw that the test job was checking out the wrong branch, right?19:14
fungii am 100% sure the change reproposal part will work. i _expect_ the stable/kilo integration testing to work because, well, it will need to after release and there's no reason it shouldn't also work now19:14
ttxhe has eagle eyes indeed19:14
ttxok, let's move to #openstack-relmgr-office for execution of plan19:15
*** mrmartin has quit IRC19:17
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: Adding hipchat publisher
*** jaypipes has quit IRC19:18
sdaguettx: you just need to read logs :)19:19
*** otherwiseguy has quit IRC19:20
ttxsdague: I already don't read them when they FAIL19:20
ttxnow I have to read them when they SUCCEED!19:20
*** spzala has joined #openstack-infra19:21
*** jaypipes has joined #openstack-infra19:21
anteayaha ha ha19:23
*** peristeri has quit IRC19:24
*** ashleighfarnham has quit IRC19:25
*** dboik_ has joined #openstack-infra19:25
openstackgerritMerged openstack-infra/reviewstats: Sync infra projects to governance repo list
*** dboik has quit IRC19:29
*** otherwiseguy has joined #openstack-infra19:29
*** david-lyle has joined #openstack-infra19:30
*** e0ne has joined #openstack-infra19:30
lifelessmordred: around? can has +2+A on ?19:31
*** armax has quit IRC19:31
*** vlaza has quit IRC19:35
*** vlaza has joined #openstack-infra19:35
jeblairfungi: no, but i don't see the plan in the etherpad anymore19:38
*** signed8bit_ZZZzz is now known as signed8bit19:38
fungii think ttx may have deleted it?19:38
*** crc32 has joined #openstack-infra19:38
fungiit's been integrated into
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: remove query for bug 1285323
openstackbug 1285323 in grenade "Services fail to shut down on the old side of Grenade" [Undecided,In progress] - Assigned to Sean Dague (sdague)19:39
*** signed8bit is now known as signed8bit_ZZZzz19:39
*** e0ne has quit IRC19:41
*** koolhead17 has quit IRC19:42
*** alex_xu has quit IRC19:48
*** alex_xu has joined #openstack-infra19:49
*** markvoelker has joined #openstack-infra19:50
*** spzala has quit IRC19:50
*** kmartin has quit IRC19:51
*** Rockyg has joined #openstack-infra19:51
elarsonnot sure if this is the right place to ask, but using jenkins job builder, I get an error when I configure a scm to do a clean up with scm: clean: after: true19:52
elarsonI noticed the docs mention that flag as an extension, but I didn't see anything outside verifying the jenkins git plugin is installed.19:53
elarsonapologies if this isn't the best channel to ask about jenkins job builder. it's a shot in the dark19:54
anteayaelarson: this is the best channel to ask about jenkins job builder19:54
anteayaelarson: have you a stacktrace you can paste to or another paste service?19:54
*** markvoelker has quit IRC19:55
elarsonanteaya: sure19:57
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add Artifactory plugin support.
anteayaelarson: what command produced this stacktrace?19:59
elarsonanteaya: jenkins-jobs --conf /jjb/jenkins_jobs.ini --ignore-cache update /jjb/designate --delete-old19:59
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin
anteayathe after: true seems to be the issue, is that part of what you created, where did you come up with that key: value pair?20:00
*** dustins has quit IRC20:00
*** smccully has quit IRC20:01
anteayacan you paste the part of the file where that code occurs? with a line or two before and after20:01
elarsonanteaya: I didn't write it originally, but I believe is where it came from (down a bit in the extensions section)20:01
elarsonanteaya: here is the section that breaks
*** mrmartin has joined #openstack-infra20:03
*** crc32 has quit IRC20:05
anteayaelarson: that syntax appears to match the expected syntax from the test:
anteayaI don't know where to go from here to offer assistance20:07
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Temporarily copy proposed/* ACLs to stable/kilo
anteayazaro: are you about?20:07
mtreinishanteaya: if you get a sec mriedem was bugging me about:
mtreinishalthough besides a +1 there wasn't much I could do for him20:08
*** crc32 has joined #openstack-infra20:09
*** dustins has joined #openstack-infra20:09
elarsonanteaya: I think we found it. it is deprecated. I can use wipe-workspace: true instead20:09
elarsonanteaya: thanks so much for looking into it!20:09
anteayaelarson: good work20:11
anteayawe have a chorus line of +1's there20:11
*** teran has joined #openstack-infra20:12
anteayayou wrote all 3 patches I see20:12
anteayawhy is ceph gating at all?20:13
anteayathey are a third party job20:13
anteayawhich are open source so happen to be running by us, and I have no problem with them voting in check, but why are they in the gate?20:14
anteayaI am very against third party anything in the gate20:14
anteayamriedem: ^20:14
clarkbanteaya its not third party20:15
anteayaI thought it was20:15
anteayathat is what I have been told20:15
clarkbits us running it, cinder and nova wanted it to vote20:15
anteayaceph and sheepdog and glusterfs20:15
fungiceph is not an openstack project, but that doesn't mean we can't gate on it20:15
fungimysql and postgresql are not openstack projects either, but we gate on whether or not our code works with them20:16
anteayaI have been told that they are being treated as being in the same category20:17
mriedemanteaya: so...if you want the dirty details it's in the cinder meeting logs from last week when we talked about making the ceph job voting on nova/cinder/glance changes in the gate20:18
mriedemit's not 3rd party, and it's a job that's already hitting all 3 projects and is the only project in infra that is testing nova with shared storage20:18
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs
* anteaya goes to read the dirty details in the cinder meeting logs from last week20:18
mriedemand it's also the top used cinder backend according to the user survey20:18
anteayaI'm just gathering resources when I get hammered by third party operators who yet again want to make the case that they should be in the gate20:19
anteayawe go through this often20:19
clarkbanteaya they have to make their case to the projects20:19
clarkbceph did so20:19
clarkbalso its open source and doesnt require special hardware20:19
mriedemfor closed source vendor backends the response is easy20:19
anteayagreat, then we just tell them to hammer the ptls20:19
anteayamriedem: please state it for the record20:20
anteayaclarkb: and the ptls and me are getting tired of being hammered20:20
*** tkelsey has joined #openstack-infra20:20
mriedemstate that infra isn't going to run a storwize cinder backend?20:20
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin
mriedemor db2?20:20
clarkbthen stop drmanding everyone test their drivers... the ptls made that decision...20:20
mriedemif it's closed source and the distros don't have packages to run the stuff, infra shouldn't be required to run the jobs20:20
anteayamriedem: thank you20:21
*** adreznec has joined #openstack-infra20:21
*** mfink has joined #openstack-infra20:21
clarkb(also it doesnt have to be hammering the ptl, it could be a ml discussion or a meeting agenda item as was in this case)20:22
mriedemis it too late for me to run for ptl? :)20:22
fungimriedem: almost a week too late20:22
fungimriedem: though there's still time to make a bid for tc!20:23
fungimriedem: or you can get a head start on your campaign for october20:23
mtreinishmriedem: well you've used a storewize right? It's something I don't have fond memories of...20:23
AJaeger_anteaya: change was
mriedemmtreinish: i haven't20:23
mriedemfungi: too much work probably20:24
mtreinishmriedem: aren't you already ptl for qa?20:24
*** tkelsey has quit IRC20:24
clarkbI think we should not discourage testing when we are able to address a need20:25
*** kmartin has joined #openstack-infra20:25
fungiclarkb: jeblair: mordred: the new world release order cordially requests your review of these stylish and sweeping acl changes...
*** dboik_ has quit IRC20:26
*** dboik has joined #openstack-infra20:26
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin
fungiclarkb: jeblair: mordred: oh, and perhaps also
clarkbthough we should consider rolling ceph testing into an existing job (I didnt want to make that a req before release since we are a bit time sensitive)20:27
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin
*** dizquierdo has joined #openstack-infra20:28
*** hdd has joined #openstack-infra20:28
openstackgerritRyan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin
*** vlaza has quit IRC20:28
*** vlaza1 has joined #openstack-infra20:28
*** bswartz has quit IRC20:28
*** dboik has quit IRC20:28
*** dboik has joined #openstack-infra20:29
*** AJaeger_ has quit IRC20:29
*** vlaza1 is now known as vlaza20:29
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/project-config: Enable python3 jobs for oslo.messaging
*** baoli has joined #openstack-infra20:30
*** whoops has joined #openstack-infra20:30
openstackgerritTimo Tijhof proposed openstack-infra/zuul: Status: Don't raise "abort" as error to the user
anteayajust to clarify from the log 16:11:02 <thingee> I think Ceph is unique situation. It's integrated in a variety of projects and maintained by infra.20:32
anteayathe ceph job is run by the openstack infrastructure, ceph isn't maintained by infra20:33
clarkbI think he means the job is maintained upstream20:33
clarkbit was initially written by russellb iirc20:34
anteaya16:13:42 <thingee> xyang2: I think the idea is eventually CI's would vote. they're just not usually stable20:35
russellbactively maintained by someone else now20:35
anteayaI have consistently said that ci's won't be in the gate20:35
anteayasince it comes up all the time20:35
anteayathey all want to be in the gate20:35
asselinwhich ci's are in the gate? they should at most be in 'check'20:36
*** spzala has joined #openstack-infra20:36
clarkbI think any test we can run can gate if prohects are happy with that20:36
anteaya16:15:41 <e0ne> imo, any _stable_ ci must bt voting20:37
*** sarob has quit IRC20:37
fungijobs being run on someone else's (not upstream openstack's) infrastructure shouldn't be voting in the gate pipeline. jobs being run on openstack's infrastructure can do so if the projects they're voting on want that20:37
*** sarob has joined #openstack-infra20:37
*** mattt has joined #openstack-infra20:37
anteayano idea who e0ne is20:37
clarkbI dont tell cinder and nova what to gate one we merely make sure that the jobs run20:37
anteayajust saying it will open a flood gate20:37
fungithe problem is that the term "third party ci" has become ubiquitous and people are referring to jobs for optional drivers/plugins as "third-party" now even when they're not running in a third-party's ci infrastructure20:38
*** tonytan4ever has quit IRC20:38
fungithe ceph job is not "third party ci" it's "an openstack ci job"20:38
anteayathat is not how it is percieved in cinder20:39
* anteaya continues to read the cinder meeting logs20:39
fungii don't care if they paint the sky green personally, doesn't change reality20:39
*** palendae has joined #openstack-infra20:40
anteayaI agree20:41
anteayahowever that is not how they see things20:41
*** stevemar has quit IRC20:41
anteaya16:22:36 <DuncanT> thingee: I'm not totally opposed, as long as (a) the goal is to do this with other drivers too (b) we can turn off the voting easily if the job gets unstable20:42
anteayaDuncanT fully expects this to be the start of a direction, not the exception20:42
fungiyep, so long as "other drivers" are, like ceph, something that we can test upstream, then that's entirely possible20:43
anteaya16:23:47 <DuncanT> thingee: If the plan is not to allow other stable CIs to also vote with similar criteria for stability (whatever we pick), then I'm very much opposed20:43
anteayafungi: I'm not feeling that caveat in DuncanT's assertion20:43
clarkbit came up later in that discussion20:44
clarkbthat has never been negotiable20:44
*** gyee has joined #openstack-infra20:44
anteayaand thingee is in agreement with him: 16:24:07 <thingee> DuncanT: I think that would be the goal eventually to allow others. That's the point of this discussion20:44
*** gyee has quit IRC20:44
anteayaclarkb: I know that but it appears they don't20:44
openstackgerritDoug Hellmann proposed openstack-infra/project-config: oslo.messaging: enable Python 3.4 check
anteaya16:26:12 <thingee> So one thing being opposed to drivers for me is we have around 45 drivers (counting all supported protocols)20:45
fungiwell, anyway, there's nothing about that ceph job that's particularly different from anything else we currently have voting in the gate. our testing incorporates all manner of software we don't write. if there are jobs we can run upstream and projects want them voting, then that's pretty much the end of it20:46
anteayaokay so having to head off the herd beliveing this is the start of 100+ cis in the gate isn't really filling me with warm thoughts20:46
anteayafungi: right20:47
anteayaexcept that cinder has expectations that are unrealistic20:47
*** belmoreira has joined #openstack-infra20:47
fungiyeah, them and lots of other people20:47
*** dustins has quit IRC20:48
*** baoli has quit IRC20:48
*** baoli has joined #openstack-infra20:49
*** otter768 has joined #openstack-infra20:49
anteayaright, but I'm just evaluating my actions in terms of their unrealistic expectations at the moment20:49
fungithis has come up many times before (docker, glusterfs, et cetera). openstack infra is willing to assist free software authors in devising jobs which integrate their components if those are things projects want jobs for. and when those jobs cease to be desirable they can be made non-voting again or removed as needed20:49
*** thedodd has quit IRC20:49
*** spzala has quit IRC20:49
*** spzala has joined #openstack-infra20:50
anteayaif you don't want me to take this time to review this patch the way I'm comfortable go on ahead, you don't need me20:50
anteayaapprove and move on20:50
*** dboik_ has joined #openstack-infra20:51
*** mrmartin has quit IRC20:51
anteayaI haven't -1'd a thing20:51
fungithat wasn't my intent at all. simply suggesting that the infra team should attempt to remain neutral when it comes to projects deciding what sorts of jobs make sense to test their software20:51
anteayaand I am trying to be neutral20:51
russellbi seem to be pretty consistently getting "Too many concurrent connections" from gerrit ... is that a global thing, or a per-source thing?20:52
anteayabut having worked with third party folks and having heard them talk about being in the gate for some time, I am concerned about what action they will take as a result20:52
fungirussellb: what's emitting that error? ssh api calls?20:52
*** viglesias has quit IRC20:52
russellbfungi: well, was trying to get the commit hook this instant20:53
russellbso ssh, yeah20:53
fungias in git review -s?20:53
fungithat's no good. checking logs now20:53
*** SumitNaiksatam has quit IRC20:53
russellbi wonder if my last change to reviewstats has made it go crazy20:54
*** otter768 has quit IRC20:54
russellband i'm getting blocked :)20:54
fungiperhaps. i'm checking show-connections output. it's taking a _while_ to return details20:54
*** sarob has quit IRC20:54
fungii hope this isn't courtesy of our new patch to provide more details in the show-connections list output20:55
*** dboik has quit IRC20:55
fungiit works, just takes a few minutes20:56
fungioh, i wonder if it's just now coming down off something20:57
openstackgerritRussell Bryant proposed openstack-infra/reviewstats: Catch exception on exec_command()
*** spzala has quit IRC20:57
russellblast iteration of that didn't sleep before retrying the connection, oops20:58
russellbcould have got into a bad loop20:58
fungiif it just stopped a couple minutes ago, that might explain the behavior i was seeing too20:58
russellbyes, i just killed it20:58
russellbcouple minutes ago20:58
russellbsorry :(20:58
fungii think it may have been starving gerrit's ssh interface20:59
*** spzala has joined #openstack-infra20:59
fungibecause show-connections is snappy again nopw20:59
russellbin better news, I did recently change it so it doesn't download the entire review history of every project every day :-X20:59
anteaya16:30:09 <mriedem> DuncanT: i think that's the goal for all jobs that infra hosts which are non-voting today20:59
anteayado we have an infra goal of having open source jobs we run in the gate?21:00
russellbfungi: sorry for the disturbance :)21:00
*** gyee has joined #openstack-infra21:00
*** SumitNaiksatam has joined #openstack-infra21:01
mriedemanteaya: i think cinder wants to get the glusterfs and sheepdog jobs voting at some point for cinder, which are already hosted on infra21:01
fungianteaya: that sounds realistic to me. i don't know whether it's an explicit goal, but we do want to see openstack projects tested and we are able to provide the resources needed for them to test themselves with free software21:01
anteayamriedem: that is a cinder goal, I didn't know infra had that as a goal21:01
* anteaya didn't have that as a goal21:02
fungiso if that's cinder's goal, that seems fine to me and is something we're able to provide the systems to make possible as long as developers step up to maintain those jobs21:02
*** mattfarina has quit IRC21:02
nibalizerjeblair: I saw some chatter that sounded like in your view "secure secret storage(hiera)" and zuulv3 are related somehow21:02
nibalizeris that your view? can you expand on that?21:03
anteayaI'm fine supporting cinder's goals whatever they are, but I'm not going to say they are my goals21:03
anteayawell thingee put this in the meeting: 16:31:59 <thingee> #agreed infra-hosted open-source cinder jobs will be eligable to vote in future once they've proved stable21:04
anteayawhich is exactly what I was looking for21:04
fungii should clarify when using the word "we" that these are things as a group we've come forward in the past saying we were willing to provide help to accomplish if they're things the projects want21:04
*** sputnik13 has joined #openstack-infra21:04
* anteaya paints that on a sign to show the third party ci hoards21:04
*** duran has quit IRC21:04
sdaguemriedem: I think it's also important that any job has a sponsor on it21:05
*** aysyd has quit IRC21:05
fungiyes, that ^21:05
anteayaand sdague had moved ahead and approved so you didn't need me anyway21:05
fungihence my comment "as long as developers step up to maintain those jobs"21:05
dhellmannjeblair, mordred, clarkb: can one of you take a look at for us? We need the ACL changes there to finish setting up the stable branches (we can't approve anything into the branches now)21:05
* anteaya moves off to do something else21:06
sdagueanteaya: which patch?21:06
sdaguethe one I just approved removed ceph from stable21:06
fungisdague: the ceph job change21:06
sdagueok, but that was a delete21:06
mriedemanteaya: fungi: sdague: yes, fine print - devs must own the jobs, if they aren't we nack them21:06
sdaguemriedem: not just own jobs, I want a name of a person21:07
sdagueand that person needs to be in -infra and -qa21:07
fungialso street address at which to target icbms21:07
mtreinishsdague: sounds like mriedem is volunteering :)21:07
sdagueand if their job starts failing and they don't step up to debug fast, we turn it off21:07
*** ZZelle has quit IRC21:07
mriedemjbernard is the ceph guy21:07
mriedemthat came up in the cinder meeting last week21:07
sdaguemriedem: ok, and he'll have vacation coverage?21:08
*** mwagner_lap has quit IRC21:08
mriedemcells is basically alaski since !comstud :)21:08
mriedemsdague: then it's dansmith :)21:08
fungihe does seem to be lurking in the infra and qa channels21:08
sdaguefungi: yes, he is21:08
sdaguebecause I made that a requirement of my +2 :)21:08
sdaguein the ceph case, I'm kind of ok, I do think we have enough people that have now said they are fine being on the hook21:09
sdaguefor some of these others, I think we need to make it really clear you need to have folks around and engaged that are debugging these21:09
mriedemfor other things, if they want to gate on a project, they need to get that project's permission21:09
sdagueotherwise we nix them21:09
mriedemlike if sheepdog wanted to vote on nova patches, we'd probably nack that right now21:09
sdagueyeh, like how the docker job keeps getting demoted21:10
fungigetting that project's permissions is step the first. if they don't have that there's little point21:10
sdaguebecause there are no owners21:10
mriedemstep the first, i like it21:10
sdaguemriedem: but I do think it's more than a project level concern, because we can be in blocking situations where we need a cinder fix landed21:10
alaskimriedem: me and bauzas and melwitt now :)21:11
*** vlaza has quit IRC21:11
sdagueright, again, I feel like we've got a couple people legit stepping up on cells that are around regularlly, that's goodness :)21:11
mriedemsdague: yeah true21:11
*** esker has quit IRC21:12
*** dprince has quit IRC21:12
*** Sukhdev has joined #openstack-infra21:13
openstackgerritMerged openstack-infra/project-config: Don't run gate-tempest-dsvm-full-ceph job on stable icehouse/juno
openstackgerritgordon chung proposed openstack-infra/project-config: enable ceilometerclient functional tests on master only
jeblairfungi, dhellmann: has ttx signed off on  in principle?21:14
dhellmannjeblair: hmm, we didn't discuss it with him21:14
*** ldnunes has quit IRC21:14
dhellmannwe can wait, if it makes you more comfortable21:15
openstackgerritDavid Lenwell proposed openstack-infra/project-config: new-project
jeblairdhellmann: if it's not urgent / blocking the other stuff, let's do that.21:15
fungijeblair: yeah, i didn't ask. in general he's seemed fine with whatever group a project's leadership wanted taking care of stable maintenance in the new order (since we removed the iron fist of stable-maint-core a while back)21:15
*** esker has joined #openstack-infra21:15
*** yolanda has quit IRC21:15
dhellmannjeblair: yeah, that one isn't blocking us21:15
*** rfolco has quit IRC21:15
*** jkraj has quit IRC21:16
dhellmannjeblair: in practice the stable team is going to be the core team, which is really only 3-4 people anyway, but I'm OK with waiting for his sign-off21:16
*** yolanda has joined #openstack-infra21:16
jeblairyeah, i feel like he wouldn't object, but would just like to keep him in the loop if it's no rush.21:16
fungimakes total sense21:16
jeblairdhellmann, fungi: aprvd
dhellmannjeblair: thanks!21:17
fungithanks. that at least gets us out of the danger zone of stable core groups mucking with pre-release branches21:17
*** rlandy has quit IRC21:17
jeblairnibalizer: i believe that for the purposes of providing credentials to _jobs_, we need a solution in zuul so that we can eliminate the remaining static build slaves and can use nodepool nodes for those jobs21:18
fungialso once that merges we can consider removing the refs/heads/proposed/* section from the all-projects acl, simplifying it and our documentation some more21:18
*** rlandy has joined #openstack-infra21:19
*** hashar has quit IRC21:19
nibalizerjeblair: makes sense, doesn't seem tightly coupled to encrypted hiera tho21:19
openstackgerritDavid Lenwell proposed openstack-infra/project-config: new-project
nibalizerif I am understanding you correctly21:19
gordchi folks, if free, can somoene validate patch to unblock ceilometerclient stable/kilo branch:
jeblairnibalizer: i agree, i don't think it is.  if we plumb the credentials through zuul, i think there's very little, possibly no, interaction with puppet and ehiera.21:21
jeblairnibalizer: basically, if the problem statement is "we need to be able to provide credentials to jobs" then i think this is the way we should look.21:21
jeblairnibalizer: if there is another (non-job-running) reason to encrypt hiera, then i don't think this idea addresses that.21:22
jbernardsdague: o/21:23
jeblairnibalizer: but the chatter i have heard is "we need people to be able to provide passwords for the jobs that we run for them and we don't want them to send them to us in the clear", which seems to point in the direction of zuul21:23
jbernardsdague: ill be here and in -qa for any ceph related issues21:23
mordredjeblair: I agree with you21:24
nibalizerjeblair: ah, i think we crossed streams on you21:24
nibalizerwe have a couple problems downstream21:24
jeblairdon't cross the streams :)21:24
nibalizerwell, we are the ghostbusters21:24
mordredwho you gonna call?21:24
nibalizerso one problem is we need to centralize and code-review our changes to hiera, which is used exclusively for secret-storage21:25
*** andreykurilin has joined #openstack-infra21:25
nibalizerour code-review system doesn't have good acl support, so the plan is to just encrypt all the data in it and clunk along with manual reviews on the command line21:25
*** dboik_ has quit IRC21:25
openstackgerritDavid Lenwell proposed openstack-infra/project-config: New repos for Akanda
*** dizquierdo has quit IRC21:26
nibalizeranother problem is a cambrian explosion of jobs that require some form of secret, for which a zuul component that could inject the right secrets at the right time would be a good fit21:26
*** dboik has joined #openstack-infra21:26
*** viglesias has joined #openstack-infra21:27
jeblairi think i'm not opposed to adopting ehiera for general puppet secrets for openstack-infra if the other roots are okay with it.  i don't think it's a big deal for us, but i don't think it would hurt.  makes the DR story a little better.  :)21:27
jeblairi'd like mordred to review that to make sure it works with ansible-puppet21:28
jeblairand where all that is going21:28
clarkbhow does master unlock it?21:28
nibalizer key on the filesystem21:28
jeblairnibalizer: is in asymmetric or symmetric?21:28
clarkbso its not more secure21:28
jeblairis it21:28
mordredI think it's no more secure than the current file21:28
jeblairclarkb: yeah, frankly i don't believe we can improve on the security here, just make it "safe" to actually publish the yaml file21:29
nibalizerthis page has the details
mordredthe qusetion is does it provide the currently level of security with the ability for us to not have to shell in to server to add new keys21:29
nibalizerdownstream we have multiple masters, so one of the problems we are trying to solve is keeping the hiera dir the same on all masters21:29
nibalizerpushing it into gerrit and then all masters pull and decrypt is a good way to solve that, we think21:30
mordrednibalizer: that will also get better with ansible+puppet-apply for you21:30
nibalizermordred: no it wont21:30
mordredyeah it will21:30
mordredyou won't need two masters21:30
clarkbdat ssh21:30
mordredbecause you can ssh OUT to anything21:30
sdaguejbernard: yep, like I said, I think we're covered for now in the ceph case. But just wanted to make sure people knew we really wanted sponsors on things21:30
nibalizerand we use only one master then... maybe yea21:30
mordrednibalizer: the puppetmaster model involves the agent connecting back to the master21:30
lifelessI'm trying to diagnose the failure on
mordredso you need a master in each network zone21:31
lifelessbut is a 40421:31
jeblairmordred, clarkb: if it asymmetric (which i _think_ it is from a quick scan of the page) it would _theoretically_ allow non-roots to add credentials to the file, however, i'm not sure we would want to do that as a matter of policy since it would mean a non-root would _know_ a credential.21:31
jeblairnibalizer: ^21:31
mordredjeblair: ++21:31
jeblair(that may be a _desired_ behavior downstream, but possibly not because i think most needs for that would be obviated by the zuul-credential path)21:31
fungilifeless: this?
mordredyah. in fact, I think downstream should have less problems with needing to empower non-roots21:32
mordredbut more problems with zuul-credentials21:32
lifelessfungi: thanks, how did you get to that?21:32
fungilifeless: i was also looking at it while it was showing failed in the gate21:32
mordrednibalizer: in any case, I don't think ehiera breaks what we did towards puppet-apply21:32
fungilifeless: "curl: (22) The requested URL returned error: 502 Proxy Error" means jenkins06 may be struggling21:33
mordrednibalizer: assuming that the ehiera stuff still works with apply21:33
clarkbmordred not quite21:33
mordrednibalizer: it means our "copy the appropriate hiera files" code will need to also copy the key21:33
nibalizermordred: yep21:33
mordredbut, as before, that's no less secure than what we're doing today21:33
clarkbmordred and it has to be key per node21:33
mordredclarkb: nope21:33
mordredclarkb: one key - lives everywhere21:33
lifelessfungi: so, whats the current incantation to retry a +A ?21:34
clarkbthat would allow every node access to all secrets21:34
mordredbut that is true now21:34
clarkbthats a bug21:34
mordredit's not21:34
nibalizermordred: as for 'do we still need multiple masters' thats a more internal question than -infra really needs to hear but my initial response is that each major network zone should have its own control node21:34
clarkbno its not21:34
clarkbpuppet master doesnt allow arbitrary lookups21:34
mordredclarkb: neither does this21:34
*** achanda has quit IRC21:34
mordredclarkb: have you seen what the hiera solution is for this?21:34
fungilifeless: yeah, remove and readd your workflow +1 (or add a second workflow +1 if yours was not the original)21:34
mordredclarkb: it's already covered21:34
clarkbmordred it does because the data is in gerrit21:34
clarkbmordred its not ...21:35
mordredclarkb: AHHH21:35
mordredthank you21:35
mordredthat was the missing piece21:35
jeblairnibalizer: also, if the zuul-credential path is needed before zuulv3, i think that's an option.  it may involve some work on gearman-jenkins-plugin too, but other than that, could be a separate workitem from the rest of zuulv3.  iow, it is required for zuulv3, but zuulv3 is not required for it.  just a matter of prioritization.21:35
mordredjeblair, clarkb: in that case, I may want to revert my thinking - I think it's too complex for us21:35
nibalizeryea hrm21:36
mordredI do not want key-per-host21:36
clarkbmordred I agree21:36
jeblairmordred: your solution involves copying host-specific hiera data to the host, right?21:36
mordredjeblair: yes - but the overall file would be publically fetchable over the internet from git.o.o in the ehiera model21:36
jeblairmordred: so what about decrypting on master and then copying only that host's subset to the host?21:36
fungimordred: worth noting, we do already have unique keys for those hosts to which we could encrypt, though we may still not want to because complicated21:37
thingeeanteaya, fungi, sdague: so mriedem and dansmith assured me that this is a community supported CI. I think it's important to define what community means there. I'm assuming jbernard is part of this.21:37
mordredjeblair: ooh21:37
jeblairmordred: so hosts never see the keys21:37
mordrednibalizer: ^^21:37
mordredjeblair: yes.. I think that's sane and acceptable21:37
anteayathingee: definitions of words are good21:37
fungithingee: yes, we're all part of this. it's the openstack ci21:37
*** achanda has joined #openstack-infra21:37
*** achanda has quit IRC21:37
nibalizermordred: jeblair interesting21:38
nibalizerim not sure you'd want to use heira-gpg there then21:38
*** achanda has joined #openstack-infra21:38
nibalizeryou'd probably want to do it more traditionally21:38
anteayathingee: if the ceph job breaks I expect a ceph dev to offer a patch to fix it21:38
openstackgerritgordon chung proposed openstack-infra/project-config: enable ceilometerclient functional tests on master only
mordrednibalizer: what do you mean?21:38
*** doug-fish has left #openstack-infra21:38
fungithe openstack project infrastructure is maintained by the openstack community at large, and facilitated by the project infrastructure team. hopefully the devs working on ceph integration in cinder are part of our community21:38
mordrednibalizer: so - let's start from a set of steps21:38
clarkbmordred: basically we aren't using ehiera at that point we are just decrypting a tar of files21:38
mordrednibalizer: step 1 - add encrypted values into a hiera file so that puppetmaster decrypts individual values before providing them to clients21:39
mordrednibalizer: step 2 - split hiera files into mutliple files, but puppetmaster still ehiera's them21:39
gordcfungi: thanks for review, i hope this is what you meant:
mordrednibalizer: step 3 - puppetmaster goes away, _something_ decrypts individual values (not whole files) before copying file content to hosts21:40
mordrednibalizer, clarkb: ^^ right?21:40
*** armax has joined #openstack-infra21:40
nibalizermordred: yea that could work21:40
* mordred doesn't care if ehiera does the decrypting in step 3 - but the format should be such that if it was run through a puppetmaster, ehiera would do the right thing21:40
nibalizerunfortunately the ehiera utility is ruby21:40
nibalizermordred: so thats the question i think21:40
jeblairbut it's just using gpg, right?21:41
nibalizerdo you use the ehiera encrypted format21:41
clarkbmordred: I wouldn't encrypt individual values21:41
fungigordc: i think your editor may have messed up your indentation there21:41
clarkbI would encrypt a tarball21:41
mordredclarkb: that's how ehiera works21:41
clarkband splat that onto the master21:41
mordredclarkb: and that's the point of this21:41
nibalizeror do you gpg /armor the entire file21:41
clarkbmordred: right but this model is different21:41
mordredclarkb: the point is so that only the secret value goes into code review21:41
clarkbmordred: because we are using ansible to copy files21:41
gordcfungi: ah i see it now... apologies21:41
mordredso that you can see what the key names are21:41
mordredclarkb: no - that's not a problem we need solved21:41
clarkbmordred: you can ou just have to pull them locally21:41
mordredthe idea is that keynames are still readable21:41
nibalizerjeblair: yes there is nothing magical, just that it has interspersed encrypetd and non encrypted data21:41
mordredby people without the key21:41
nibalizerso you have to make a software to do that21:42
nibalizerwhich probably isn't super hard21:42
openstackgerritgordon chung proposed openstack-infra/project-config: enable ceilometerclient functional tests on master only
nibalizerbut ehiera is the utility kindof21:42
mordredclarkb: if we're goign to have a completely opaque blob, there is zero point to putting it in gerrit21:42
nibalizeri have to run off and do a thing21:42
jeblairnibalizer: yeah, i like that it's just some syntax around stardard gpg21:42
clarkbmordred: I think there is zero point regardless21:42
nibalizerill come back and we can bang things out21:42
nibalizertimrc: i think there is value to you proposing a spec for this upstream21:42
clarkbmordred: this will only serve to make our config management more complicated without any benefit to security21:42
mordredclarkb: I believe I'm mostly inclined to agree with you. otoh - I don't think it will hurt security21:43
jbernardthingee: yep, ill do my best to keep it healthy21:43
jeblairnibalizer, timrc: ++  it sounds like splitting into two parts -- stuff for puppet and stuff for jobs would help21:43
mordredso I think it's tenable21:43
*** dizquierdo has joined #openstack-infra21:43
* timrc blinks21:43
mordredwhether it's desirable for us is a different question21:43
gordcwhoops wrong window.21:43
jeblairclarkb: yeah, it's not a security benefit.  but it does mean we can put our secret yaml file in public git21:43
fungijeblair: nibalizer: mordred: clarkb: uunfortunately the values when encrypted will need a fair amount of padding to avoid revealing their relative lengths, which means the lines will be seriously long i expect21:44
jeblairclarkb: which is not something we've been clamoring for, but, honestly, it is good for disaster recovery, system visibility, and lets us do things like "Depends-On: some change that adds a credential"21:44
timrchiera-eyaml-gpg looks pretty nice...21:44
clarkbjeblair: I am not sure I find that particularly beneficial when balanced against the extra complications it introduces21:44
jeblairfungi: yeah, that could be a big drawback21:44
mordredI thinkk Depends-On: change that adds a credential is the biggest benefit for us21:45
timrcthe default pkcs7 was rather limited...21:45
mordredthe other things are largely meh-ish21:45
timrcit made decrypting a review a lot more awkard21:45
timrcUnless copying around private keys is how you roll.21:45
*** tiswanso has quit IRC21:45
timrcYou'd need to invent some weird thing to log into some central place where the private key exists to decrypt stuff.21:45
jeblairyeah, honestly, i'd probably never decrypt a change in review.  i'd just look at it and say "yep, it adds a key".21:46
*** dims_ has joined #openstack-infra21:46
*** jamielennox|away is now known as jamielennox21:46
*** Rockyg has quit IRC21:46
timrcjeblair, Yeah to me the most critical thing is making sure the key is correct... value? Maybe not as important, though jesusaurus brings up a good point about wanting to make sure the certificate you encrypted is actually valid.21:46
notmynamewhat's the policy for updating a dependency version during an RC? (asking for a friend)21:46
*** andreykurilin has quit IRC21:46
clarkbnotmyname ffe to requirements21:47
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1442272
openstackbug 1442272 in neutron "Ovs monitor tests fail with eventlet timeout" [High,In progress] - Assigned to Maru Newby (maru)21:47
fungiwe could in theory use delegate keys or encrypt to multiple keys (and boy would that be bloated in the file) but i agree that i would be unlikely to bother21:47
jeblairnotmyname: are you going to propose the change for your friend too? ;)21:47
*** annegentle has quit IRC21:47
notmynamejeblair: ;-)21:47
notmynameclarkb: thanks21:47
timrcSo anyway... I should probably read everything before someone though to mention 'timrc'21:48
*** dims has quit IRC21:48
*** mwagner_lap has joined #openstack-infra21:48
*** whoops has quit IRC21:49
jeblairfungi, clarkb: so i'm seeing some very slight benefits for this (visibility).  there may be some drawbacks (like file size), and a lot of stuff that doesn't really change at all (security).21:49
jeblairfungi, clarkb: is it reasonable to write up a spec and evaluate those further?21:49
jeblairi'd like to see what it actually looks like and what it would entail before making a final decision21:49
lifelessjogo: ping21:50
lifelessjogo: you say we can't use pip install -r .... successfully. Why not?21:50
mordredyeah. at the very least, if it's a thing that downstreams want to use, and we need to add $something to allow them to do so even if we don't want to use the feature, and that thing is not particularly onerous, I think that's worth consideration too21:50
fungijeblair: i'm not opposed to the idea as stated. on the fence about possible benefits but a more detailed description might help me decide21:50
*** dkranz has quit IRC21:51
clarkbmordred: I don't think we need to add anything, they can use ehiera transparent to us21:51
clarkbso my concern is in the 5 minutes of discussion here it has already been confusing to figure out where we can leak credentials21:51
jogolifeless: because of transitive dependency resolution21:51
clarkbthe current system does not leak them (at least by design and our understanding of said design) and has worked well21:52
lifelessjogo: I don't understand21:52
clarkbthe current system also does not prevent anyone else from using ehiera21:52
jogolifeless: if reqs.txt is pulled directly from pip freeze21:52
lifelessjogo: In my head it will work. Is there a bug or example case where it fails.21:52
jeblairclarkb: yeah, that's why i want to divert this to a conversation about having a spec to think this all out rather than a conversation about whether we use it.21:52
clarkbif someone wants to design how it may be used I don't have a problem with that21:52
*** annegentle has joined #openstack-infra21:52
mordredclarkb: I agree - except that I think there _will_ be a difference when we move to puppet-apply21:52
jeblairclarkb: i mean, it's a security topic, we ought to really get a lot of people thinking about it.21:52
*** tqtran has joined #openstack-infra21:52
*** dboik_ has joined #openstack-infra21:52
clarkbit just seems odd to spin cycles on a non issue21:53
clarkbbut maybe I am not understanding what the issue is21:53
mordredclarkb: that's what I was thinking about in terms of feature additions - basically to ansible-puppet21:53
*** ianw has joined #openstack-infra21:53
jeblair(also, i haven't called this out explicitly as a benefit, but we have a bad track record of git committing our changes to common.yaml)21:53
*** spzala has quit IRC21:53
*** markvoelker has joined #openstack-infra21:53
mordredI could also see it as more of a benefit once we have more than one file - in taht it will allow us to doublecheck that a key goes into the _correct_ file21:54
*** bknudson has quit IRC21:54
mordredso - worth a conversation- not convinced21:54
*** spzala has joined #openstack-infra21:54
*** pc_m has quit IRC21:54
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1439371
openstackbug 1439371 in Cinder "Volume creation from image fails for UEC and Glance API version 2" [High,Fix committed] - Assigned to Jon Bernard (jbernard)21:54
jrollis there a tl;dr on backporting patches to RC versions?21:55
jogolifeless: trying to remember the case21:55
jogoAFAIK sdague may remember21:55
clarkbjroll: its basically the same as backporting to stable21:55
clarkbjroll: should fix bugs etc etc and be land(ed|ing) on master21:55
jrollclarkb: ok, I guess I'm just not seeing a stable/kilo branch21:55
*** dboik has quit IRC21:56
*** tqtran has quit IRC21:57
clarkbjroll: there may be a proposed/kilo branch21:57
*** dboik_ has quit IRC21:57
jogolifeless: A ?21:57
jogolifeless: ?21:57
clarkbjroll: but if neither of those two branches exist then I don't think the project in question has done an RC yet21:57
jrollclarkb: yeah, just learned that, ty much :)21:57
*** markvoelker has quit IRC21:58
jogolifeless: so I may be wrong about the need for --no-deps if reqs.txt includes all transitive22:00
lifelessjogo: so that case doesn't describe a freeze-equivalent22:00
jogolifeless:  correct22:01
lifelessjogo: the requirements file there was broad and allowed a too-low version to be selected22:01
*** esker has quit IRC22:02
*** annegentle has quit IRC22:03
*** sarob has joined #openstack-infra22:03
*** patrickeast has quit IRC22:04
*** Swami has quit IRC22:10
*** hdd has quit IRC22:10
timrcI guess I should probably listen to my laptop when it tells me it's run out of power and is going to shut off.22:13
mordredtimrc: :)22:13
timrcSo not sure anything I said got sent, but...22:13
clarkbtimrc: doesn't apepar to have made it to the freenode ether22:14
timrcThe fundamental problem we're solving downstream with eyaml is being able to gate changes to Hiera yaml since a change to Hiera yaml now is essentially an unreviewed change to production.  Also, if we were to lose the system today, the only place those secrets are backed up to is tape, which is a 24 hour turn around time, since the repo is local to the puppet master.  Having the yaml in a Gerrit project would give us all the benefits of having a22:15
timrcdvcs in that regard.22:15
*** yamahata has joined #openstack-infra22:15
nibalizertimrc: another problem is we have 4 masters and need the hiera data on them to be consistent22:15
nibalizerspoiler alert: its not right now22:15
timrcWe had two incidents recently.  The first was a bad cert was deployed to a production system and then a good cert was deployed, but to the wrong system.22:15
timrcAnd, yeah, what nibalizer said.22:15
nibalizeri think those we both this guy <---22:15
clarkbjogo: iirc you had run into that before right? (the dhcp discover from cirros VM failing to get a lease), was that a problem on the single node job(s) too?22:15
timrcSo I'd like to eliminate localized repos and  make sure that changes are reviewed by someone other than the person making them.22:16
jogolifeless: yeah, I may be mistaken about needing --no-deps22:16
notmynamefungi: clarkb: what's the process for doing a requirements FFE?22:17
clarkbnotmyname: I think send mail to the list with details /me finds an example22:17
clarkbdhellmann: ^ iirc you and ttx were dealing with those?22:17
jogoclarkb: yes i have seen that22:17
jogostill don't know what the issue is though22:17
clarkbjogo: ok, let me try setting up my iptables capture for dhcp and see how ugly that makes our logs22:18
jogothat is the main blocker to getting the job stable22:18
jogoclarkb: awesome22:18
clarkbjogo: iirc that was my idea for debugging this as it is infrequent22:18
jeblairtimrc, nibalizer: so when you write the spec, having a good problem statement would help a lot, keeping both downstream and upstream in mind.22:18
*** belmoreira has quit IRC22:18
*** rlandy has quit IRC22:18
clarkbnotmyname: ya I see at least one requirements FFE to the dev list22:18
*** amitgandhinz has quit IRC22:19
clarkbnotmyname: mestery sent the message thread subject is Kilo stable branches for "other" libraries22:19
jogoclarkb: I think it happens 10 or 20% of the time or so22:19
timrcThere are days I want to just give up on managing secrets and he help rms turn emacs into a word processor.22:19
funginibalizer: also quaint stories about you blowing up servers because you did not have this feature is right on point ;)22:19
fungitimrc: the emacs operating system already contains word processor technology baked right in between the newsreader and the dogflosser22:20
*** annegentle has joined #openstack-infra22:20
notmynameclarkb: ok, just looks like a very brief note with a link to the patch22:21
timrcfungi, :)22:21
*** patrickeast has joined #openstack-infra22:21
jeblairtimrc: i edit gpg encrypted files directly with emacs :)22:21
clarkbnotmyname: yup, so I think push the change to requirements then point to it with a blurb about why the hcange is desired (I would put the same blurb in the commit message so that the reason follows the edit in git)22:21
notmynamegood news is this is a swift-only dependency (ie not something that affects other projects)22:22
*** julim has quit IRC22:22
*** annegentle has quit IRC22:24
jeblairyolanda: i have reviewed your change!  :)  i think it's 90% there and is generally a good approach, but i have suggested one api change.22:24
fungii think our clouds are mostly broken today22:25
jeblairwe are not going to space today22:25
fungiwrong end pointing toward ground, yes22:25
*** ivar-laz_ has joined #openstack-infra22:25
jeblairfungi, clarkb: should we just open an ec2 account and expense it to the foundation?  :)22:26
*** SumitNaiksatam has quit IRC22:26
jeblairanyone have a cloud we can use?22:26
fungii believe rax-dfw may still be in trouble. we have 106 instances there in a building state right now22:26
*** dboik has joined #openstack-infra22:26
fungijeblair: itym "borrow"22:26
jeblairwe'll give it back22:26
fungiwe'll give it back when we're done, honest ;)22:26
timrcLet's get on Azure.22:27
timrcAnd use snappy ubuntu core.22:27
fungiwe've also got rather a lot of nodes building in hpcloud and almost none running jobs there22:27
*** dboik_ has joined #openstack-infra22:27
fungilooks to me like hpcloud is mostly broken, and a third of rax is also broken22:28
jeblairstatus notice If anyone has an OpenStack cloud they aren't using, would you please give credentials to the infra team in #openstack-infra?  We will return it in the condition we found it aside from normal wear and tear.22:28
clarkbjeblair: ha22:28
clarkbcalling all clouds22:28
fungisend in the clouds22:28
jeblair10 points for fungi22:29
fungiat $oldjob we used to get our yuks by replacing the word "cloud" with "clown" in marketing materials22:29
clarkbfungi: you should install the cloud2butt firefox/chromium plugin22:29
*** ivar-lazzaro has quit IRC22:29
fungigot to the point where whenever i heard the sales guys talking about our cloud services i would have even more trouble than usual holding back my laughter22:30
*** SumitNaiksatam has joined #openstack-infra22:30
*** SumitNaiksatam has quit IRC22:30
*** dboik has quit IRC22:31
jeblairum.  so if, at the summit, you find that i am spending all of my time rolling around on the floor laughing, you will know why.22:31
fungithe words sounded similar enough i'd start hearing it as "clown" all the time, and even accidentally mispronouncing it clown at times22:31
*** SumitNaiksatam has joined #openstack-infra22:32
greghaynesI had to uninstall that when I started openstacking22:32
greghaynesit was too much22:32
greghaynesalso, the notice wikipedia had for editors to uninstall that plugin because they were unintentionally changing entire wikipedia pages was amazing22:33
jeblairfungi: i expect some of us will now try to pronounce it just exactly in the middle so you're never quite sure which work was said.22:33
fungii certainly hope so22:33
jeblairgreghaynes: haha!22:33
fungiyeah, rax-dfw has 106 nodes building, 74 deleting22:34
fungiit looks like all the ones in a building state are from ~7 hours ago when i deleted all the nodes which had been stuck in a delete state there since yesterday22:35
jeblairTypeError: None object cannot be quoted22:35
fungii'm going to see about purging any alien nodes. seems like nodepool might be fighting back an invasion22:35
jeblairthat's showing up a lot in the logs.22:35
*** annegentle has joined #openstack-infra22:36
fungirussellb: you have 63 connections open to gerrit22:37
fungiin good news, show-connections is correctly showing account names now!22:37
*** mriedem has quit IRC22:38
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Don't try to delete nonexistent jenkins nodes
jeblairfungi: i thought it always did that?22:38
*** dizquierdo has quit IRC22:39
openstackgerritKhai Do proposed openstack-infra/system-config: Allow puppet to manage replication keys on review-dev.o.o
jeblairfungi, clarkb: ^ we may want to push that nodepool change in.  i believe we may be underutilizing quota because of it.22:39
clarkbjeblair: will take a look in just a moment22:39
zarofungi: ^ may need your help setting up that key in heira?22:40
fungizaro: yep, i can do that part no problem22:40
jeblairfungi, clarkb: (of course, that's only happening because of extreme breakange from earlier, but it would still theoretically make it harder to recover from.  and of course if clouds can't satisfy demand anyway, it probably won't help.  but hey, it might a little.)22:40
*** annegentle has quit IRC22:41
jeblairfungi: i think my change to gerrit should show you the username or connection or something in the "show-queue" command if one of the tasks it is running is a stream-event task (which almost never shows up there unless there's a problem)22:41
fungijeblair: oh, good catch. so that's why we're getting that logged22:41
fungijeblair: ahh, yep, now i remember22:42
jeblairOverLimit: Maximum number of ports exceeded22:42
jeblairwe're also getting that22:42
*** bswartz has joined #openstack-infra22:42
fungii'm currently looking for disassociated floating ips to delete too22:42
clarkbmay need to rerun the port cleaner script22:42
fungiproof this is janitorial duty22:43
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net runs
clarkbjogo: ^22:43
jeblairfungi, clarkb: once we're done mopping up, we could consider giving yolanda's change a test run in production22:44
jeblairfungi, clarkb: it's not ready to land for API reasons, but I think should _work_ and it might bump hpcloud throughput a little22:44
jeblairanyway, something to keep in the back pocket22:45
clarkbjeblair: by not ready to land you mean not ready to merge but you are ok deploying it as is temporarily?22:45
jeblairclarkb: yeah22:45
*** gordc has quit IRC22:45
clarkbok, I will review that one now22:45
fungiyeah, a few hundred disassociated fips. deleting them now22:46
*** ivar-laz_ has quit IRC22:46
*** ivar-lazzaro has joined #openstack-infra22:46
jeblair(also, i hear the process to actually fix this in hpcloud is proceeding, but not implemented yet)22:46
fungioh, awesome!22:47
openstackgerritMerged openstack-infra/elastic-recheck: remove query for bug 1285323
openstackbug 1285323 in grenade "Services fail to shut down on the old side of Grenade" [Undecided,In progress] - Assigned to Sean Dague (sdague)22:48
*** achanda has quit IRC22:48
*** patrickeast has quit IRC22:48
fungiremoving >400 alien nodes now too22:50
*** otter768 has joined #openstack-infra22:50
*** markvoelker_ has joined #openstack-infra22:53
lifelessfungi: I hope you removed the one I had yesterday :)22:53
fungilifeless: no idea, but i'll check in a bit. thanks for the reminder22:54
fungii take it you worked out the pbr oddity there?22:54
*** otter768 has quit IRC22:54
lifelessfungi: kindof. I have NFI why it worked for me locally. It was broken ass code.22:55
fungiand yes, i've _now_ deleted that helpd node. thanks again22:55
lifelessso I figured out why it was breaking there, and fixed it.22:55
*** soren has quit IRC22:57
*** baoli has quit IRC22:57
*** achanda has joined #openstack-infra22:58
*** baoli has joined #openstack-infra22:58
*** dimtruck is now known as zz_dimtruck22:59
*** erlon has quit IRC23:01
*** Rockyg has joined #openstack-infra23:01
*** baoli has quit IRC23:03
clarkbmordred: jeblair re the nodepool change with queues. Does shade implement the task manager? No right? instead you tell shade here is a task manager you can use if the code using shade needs it23:04
clarkbmordred: jeblair: then shade submits tasks into that manager23:04
clarkb(trying to make sure I grok jeblairs comments)23:04
fungichecked all the jenkins masters and i think only 06 is struggling. i'm putting it into shutdown to get ready for a restart23:04
clarkbfungi: thanks23:05
fungithat's also the one that ate lifeless's pbr docs job in the gate earlier, so seems to be related23:05
clarkbfungi: yes I think that the wget/curl for the console.html has a high chance of failing when jenkins has thread leaked23:06
fungii wouldn't be surprised if the number of 5xx proxy errors from jenkins increases when it gets into this state23:06
lifelesswow, lots of building nodes23:06
lifelessflaky cloud?23:06
clarkbI think that happens because jenkins wants to make a thread for your http session but it fails to do so in a timely manner23:07
jrolllifeless: rhetorical question? :)23:07
lifelessjroll: no, it could be other reasons.23:07
lifelessjroll: so genuine q23:07
jrolllifeless: was a joke23:07
lifelessjroll: ah :)23:07
*** jaypipes has quit IRC23:08
clarkblifeless: flaky clouds :)23:08
clarkbhpcloud had leaked a bunch of resources (fungi just cleaned them up) and rax dfw is having trouble booting/deleting nodes23:08
fungiyeah, for whatever reason nodepool can't seem to build nodes in rax-dfw at all. i haven't debugged it yet but seems that all its attempts to build new instances there since i cleared all the hung deleting ones from their incident in dfw yesterday are now spinning indefinitely with no response23:09
clarkbfungi: have you tried spinning a node up there by hand?23:09
funginot yet23:09
clarkbfungi: I can do that if it will help23:09
fungiplease do23:09
clarkbkk starting that now23:09
fungii sort of wonder if something happened to the api endpoint there and nodepool is just unhappy with it, which would also explain why i was able to manually delete nodes it should have retried deletion for on its own23:10
*** chlong has joined #openstack-infra23:11
*** Sukhdev has quit IRC23:12
clarkbI am booting a clarkbtest VM based on our latest devstack-trusty image in dfw23:13
jrollfungi: I'm happy to look into things if you have a tenant id or node uuids23:15
clarkbjroll: I can hand you clarkbtest's uuid if it doesn't boot up properly23:15
clarkbnope it booted fine. fungi
*** otter768 has joined #openstack-infra23:16
*** mhu has quit IRC23:16
*** patrickeast has joined #openstack-infra23:16
*** mhu has joined #openstack-infra23:16
*** asettle has quit IRC23:19
clarkbI am going to delete it and try again23:19
jrollclarkb: what's the tenant ID, I can look for api weirdness23:19
clarkbrandom note: it would be nice if our clients handled insecure platform warnings sanely23:19
reedclarkb, like a 'no panic'-'ok panic' sign?23:21
clarkbjroll: you want the username or the tenant number?23:21
jrollclarkb: number?23:21
clarkbreed: one print per invocation would be nice23:21
*** jaypipes has joined #openstack-infra23:22
fungiclarkb: other than a handful in an error state and our long-running (proposal, pypi) slaves, we have nothing in that region for that tenant. i think nodepool is trying to build/delete and never getting a response from the api endpoint23:23
clarkbsecond boot came up fine too23:24
fungiclarkb: forcing that socket to close from the nodepoold end (possibly by restarting nodepoold itself) will probably clear this up23:24
clarkbfungi: we can put jeblairs patch for the jenkins thing in when we do it23:24
fungithat's what i'm thinking, yes23:24
clarkbjeblair: ^ want ot try yolanda's queues change too since a restart looks handy anyways?23:24
jeblairclarkb: just got back, catching up23:25
fungialso, nodepool could benefit from a periodic no-op api "ping" just to test that the provider's endpoint is still responding, and then close and reopen the socket if not23:25
fungiwe've seen this before, though it was usually with the tripleo cloud's api endpoint going away in an outage/hard reboot23:26
jeblairclarkb: yeah, i'll stage both of those changes locally on nodepool23:26
clarkbfungi: iirc we applied a timeout to hpcloud but not nodepool connections23:26
clarkbfungi: let me find that and I can update the rax config to use it too23:26
fungiclarkb: oh! that's right, forgot about that23:27
fungithat's even better23:27
*** hichihara has joined #openstack-infra23:27
jrollfungi: clarkb: so to start, I don't see any POST requests to dfw from your tenant between 19:16:35 and 23:11:3823:27
fungiespecially since it already exists ;)23:27
fungijroll: yep, i think your dfw api endpoint went away ungracefully at some point yesterday and nodepoold never noticed because the socket was still in an established state at our end23:28
jrolland only two since 23:11 which I presume is clark's stuff23:28
jrollI guess it could have, LB failover or something23:29
openstackgerritClark Boylan proposed openstack-infra/system-config: Set nodepool api-timeout on rax providers
fungijroll: 19:16 likely corresponds to when i was doing out-of-band deletes23:29
jeblairclarkb, fungi: those two changes staged.  i'll wait until clarkb is ready with the rax timeout config change so we don't immediately yank the provider out after the restart23:29
clarkbfungi: ^ that should do it23:29
jrollfungi: that wouldn't be a POST though23:29
*** amitgandhinz has joined #openstack-infra23:29
fungioh, the api uses post for creates? i get you23:30
* fungi is woefully unfamiliar with openstack's underlying rest apis23:30
jeblairclarkb, fungi: want to force-merge that patch, then stop nodepool, then manually apply that patch, then start nodepool?23:30
*** teran_ has joined #openstack-infra23:30
jrollyeah, POST is create, DELETE is delete23:30
fungijeblair: sounds fine23:30
jeblair(i think that should keep the config from flapping with puppet)23:31
clarkbjeblair: I think we have to disable puppet too until the next pass23:31
jrollfungi: so this would have been like 4 hours ago that it stopped, I didn't hear anything about things happening to APIs but I'm only halfway in the loop on stuff like that :)23:31
clarkbjeblair: but I am on board with the rest of your plan23:31
jeblairclarkb: well, i was thinking if we force-merged it, it should work out23:31
*** armax_ has joined #openstack-infra23:32
*** miqui_ has joined #openstack-infra23:32
jeblairclarkb: i guess there's a race in there23:32
clarkbjeblair: ya only updates system-config at the start of its run23:32
*** a13x2121 has joined #openstack-infra23:32
jeblairokay, we'll disable puppet then23:32
*** crc32 has quit IRC23:32
jeblairpuppet is disabled on nodepool and not running, so we should have the mutex.23:32
jeblairfungi: you want to force-merge when you're ready?23:33
*** mestery_ has joined #openstack-infra23:33
fungijroll: we must have been doing something else around that timeframe. this looks like nodepoold hasn't actually been communicating successfully with your api since sometime yesterday (since it never successfully retried deletes on the stuff which it should have from yesterday and had more than 100 nodes it thought were building from 8 hours ago when i deleted those)23:33
fungijeblair: can do23:33
*** pblaho_ has joined #openstack-infra23:33
*** tteggel_ has joined #openstack-infra23:34
*** amitgandhinz has quit IRC23:34
*** kmartin_ has joined #openstack-infra23:34
jrollfungi: hrm, ok, if there's anything else you'd like me to dig into let me know (yesterday there was network shenanigans in DFW so that may make more sense)23:34
*** Ryan_Lan1 has joined #openstack-infra23:34
*** Shrews_ has joined #openstack-infra23:35
*** frickler_ has joined #openstack-infra23:35
*** dhp_ has joined #openstack-infra23:35
*** HenryG_ has joined #openstack-infra23:35
*** sarob_ has joined #openstack-infra23:35
*** salv-orl_ has joined #openstack-infra23:36
*** Shrews has quit IRC23:36
*** Shrews_ is now known as Shrews23:36
openstackgerritMerged openstack-infra/system-config: Set nodepool api-timeout on rax providers
fungithat's ^ merged. stopping nodepool and applying manually now23:37
jeblairfungi: oh i already staged the config change23:37
fungijust saw that--thanks23:37
fungii'll start nodepoold back up again in that case23:37
jeblairBadHTTPException: Error communicating with server[]23:38
*** aarefiev_ has joined #openstack-infra23:38
*** skolekonov_ has joined #openstack-infra23:38
*** a13x212 has quit IRC23:38
*** dmsimard has quit IRC23:38
*** teran has quit IRC23:38
*** Ryan_Lane has quit IRC23:38
*** gnuoy has quit IRC23:38
*** Mithrandir has quit IRC23:38
*** grantbow has quit IRC23:38
*** kmartin has quit IRC23:38
*** miqui has quit IRC23:38
*** HenryG has quit IRC23:38
*** skolekonov has quit IRC23:38
*** jamespag` has joined #openstack-infra23:38
*** gnuoy` has joined #openstack-infra23:38
*** jamespag` has quit IRC23:38
*** jamespag` has joined #openstack-infra23:38
*** reed_ has joined #openstack-infra23:38
clarkbjeblair: ya fungi put it in shutdown mode because its thread leaking23:38
*** cburgess_ has joined #openstack-infra23:38
*** dougshelley66_ has joined #openstack-infra23:38
fungiyeah, it's in prepare for shutdown. i'm getting ready to restart 06 as soon as a couple lingering jobs wrap up23:38
jeblairok cool23:38
*** Mithrandir has joined #openstack-infra23:38
*** markvoelker has joined #openstack-infra23:39
*** vhoward- has joined #openstack-infra23:39
*** cdent has quit IRC23:39
*** ctlaugh_ has joined #openstack-infra23:39
*** Krinkle has quit IRC23:39
*** jesusaurus has quit IRC23:39
*** vhoward has quit IRC23:39
*** sarob has quit IRC23:39
*** armax has quit IRC23:39
*** pblaho has quit IRC23:39
*** salv-orlando has quit IRC23:39
*** heyongli has quit IRC23:39
*** frickler has quit IRC23:39
*** mestery has quit IRC23:39
*** aarefiev has quit IRC23:39
*** reed has quit IRC23:39
*** jamespage has quit IRC23:39
*** dougshelley66 has quit IRC23:39
*** tteggel has quit IRC23:39
*** dhp has quit IRC23:39
*** ctlaugh has quit IRC23:39
*** cburgess has quit IRC23:39
*** heyongli has joined #openstack-infra23:39
*** dmsimard_ has joined #openstack-infra23:39
*** dmsimard_ is now known as dmsimard23:39
*** dougshelley66_ is now known as dougshelley6623:39
*** heyongli has quit IRC23:39
*** heyongli has joined #openstack-infra23:39
*** armax_ is now known as armax23:39
*** sarob_ is now known as sarob23:39
fungiit 5xx'd a console log curl on a gate job earlier, which prompted me reviewing all the masters for signs of the thread leak. that seemed to be the only one23:39
jeblairclarkb: re earlier question -- shade will implement a simple synchronous task manager that it will use by default.  nodepool will supply it's threading taskmanager when it uses shade23:39
*** ajmiller has quit IRC23:39
clarkbjeblair: roger23:39
*** jesusaurus has joined #openstack-infra23:39
*** markvoelker_ has quit IRC23:40
*** reed_ is now known as reed23:40
*** markvoelker has quit IRC23:40
*** Krinkle has joined #openstack-infra23:40
*** markvoelker has joined #openstack-infra23:40
jeblairclarkb: (and so all the task classes will end up being defined in shade and removed from nodepool)23:40
clarkbjeblair: right, and just the taskmanager would be passed from nodepool to zuul23:41
*** markvoelker_ has joined #openstack-infra23:41
mordredjeblair: we said the same thing but spelled it with different regional accents23:41
jeblairmordred: uh-huh23:41
*** grantbow has joined #openstack-infra23:41
*** grantbow has quit IRC23:41
*** grantbow has joined #openstack-infra23:41
jeblairso far, nodepool is not spewing exceptions to the log.23:42
fungiyou bet yer sweet bippy!23:42
*** emagana has quit IRC23:43
jeblairoh, it's doing that thing where the flavor cache isn't populated so 50 listflavortasks get enqueued.  i'm assuming that'll be fixed by shade, yeah?23:43
jeblair(i mean, should be easy to fix in nodepool, but i think we're about to just drop that code, and it only happens at startup)23:43
jeblairalso extensions23:43
mordredjeblair: yes - I it should be23:44
*** esker has joined #openstack-infra23:44
*** Ryan_Lan1 is now known as Ryan_Lane23:45
mordredSpamapS: ^^ as we're working on the caching stuff, we should think about the thundering herd case above ^^ and make sure we're doing a good job with it23:45
*** Ryan_Lane has quit IRC23:45
*** Ryan_Lane has joined #openstack-infra23:45
*** markvoelker has quit IRC23:45
fungilooks like jenkins07 just threw a shoe on a change in the gate too...
mordredShrews: ^^ you too23:45
SpamapSmordred: we're not. ;)23:45
SpamapSmordred: we need to write a dogpile.cache backend that uses gearman ;-)23:46
mordredSpamapS: I know we're not _now_23:46
clarkbfungi: hrm we are supposed to retry23:46
SpamapSor teach dogpile.cache to do soft expires23:46
jeblairSpamapS: clearly yes, but i don't know why23:46
clarkbfungi: there may be a bug in the retry logic, I will take a look23:46
* mordred throws a nicely seasoned salmon at SpamapS23:46
jeblairSpamapS: and probably AFS too? :)23:46
fungiclarkb: thanks23:46
SpamapSjeblair: this is where coalesce comes in23:46
mordredSpamapS: can we reimplement gearman using AFS23:46
SpamapSIIRC geard doesn't actually implement coalesce?23:46
clarkbSpamapS: it does not23:47
SpamapSmordred: geArFS?23:47
mordredSpamapS: ++23:47
SpamapSOk, so without coalesce, gearman is not good for thundering herd protection23:47
jeblairSpamapS: there was nothing called coalesce in the docs when i wrote gear23:47
SpamapSWith it, it does all the magic for you.23:47
clarkbjeblair: its still not there23:47
clarkbjeblair: gearman appears to have grown quite a few features without the protocol being updated23:47
SpamapSjeblair: unique jobs should only be queued once, and if a unique arrives with the same unique as an in-progress job and it is not backgrounded, its client should be sent the response for the in-progress job.23:48
SpamapSThis was basically the first feature of gearman.23:48
SpamapSLike, the perl gearman.23:48
fungisince it looks like jenkins07 may also be starting to show signs of unhappy, i'll start quiescing it as soon as i get 06 restarted23:48
SpamapSSo it's possible nobody wrote it down because it was the reason the wrote gearman. ;)23:49
SpamapSthey wrote23:49
jeblairi think there's a lesson here23:49
jeblairSpamapS: i thought unique ids were added later23:49
clarkbfungi: I wonder, could nodepool restarts somehow trigger the jenkins unhappyness? thought 06 was unhappy prior23:49
SpamapSAlso the word coalescence does appear _once_ in libgearman's documentation. :)23:50
fungimaybe we have a thundering herd condition with nodepool hitting the jenkins api at startup too? seems unlikely but possible23:50
jeblairSpamapS: not here:
SpamapSjeblair: uniques being fed to the worker was added later23:51
SpamapSjeblair: but it was always in the client request23:51
jeblairSpamapS: yeah23:51
jeblairSpamapS: it never says what the server will do with it.  other than (possibly) provide it to the worker23:51
SpamapSso the protocol isn't what implements it, and I totally agree geard implements the protocol. :)23:51
openstackgerritClark Boylan proposed openstack-infra/project-config: Fix retries in
clarkbfungi: ^23:53
jeblairSpamapS: i would just expect the documentation for SUBMIT_JOB.* to say "if the unique thingy matches an existing job, the client will get the results from that job"... more or less in the way that it says "If on of the BG versions is used, the client is not updated with status or notified when the job has completed (it is detached)."23:53
SpamapSjeblair: also my merge proposals against gearmand have sat ignored by upstream for a while, so I am a little worried that protocol document is unmaintained. I'd love to add a note "This unique ID may be used by the server to coalesce requests to a single worker."23:53
fungiclarkb: thanks!23:53
jeblairSpamapS: yeah i think that's what's needed23:53
SpamapSjeblair: yeah, the thing is, the protocol doesn't need to say that.. because coalescing is invisible to worker and client. :-P23:54
jeblairSpamapS: anyway, we can implement coalesce; i don't think it will break gear23:54
jeblairSpamapS: i think it should because it's an important behavior for a client to know about23:54
SpamapSbut if the protocol _did_ say that it would help implementors and might explain "why would I ever want a unique ID?" to the protocol users.23:54
jeblairSpamapS: i think that if a client submits a job, it should know whether it's going to run once or twice23:54
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net tempest runs
SpamapSjeblair: the problem is that you can't know that if you have multiple servers. :-P23:54
jeblairSpamapS: so i think it's important for it to be defined, since jobs might have side effects23:54
jeblairSpamapS: the protocol only applies to one server.  a client that cared would know how many servers it submitted the job to23:55
SpamapSand it could even run twice-ish .. if it just finished, you submit again, that looks like twice. :-P23:55
SpamapSanyway, I 100% agree, protocol doc should point out the server can use this for coalesce. Submitting merge proposal now.23:56
SpamapSHopefully it merges before 2016.23:56
* mordred wishes SpamapS good luck23:56
SpamapSNot looking too active. :(23:56
*** tqtran_afk is now known as tqtran23:57
jeblairnodepool used is about 2x what we had before23:57
fungimuch of which is probably just having dfw back in the pool again23:58
jeblair281 currently23:58
jeblair/usr/local/lib/python2.7/dist-packages/novaclient/v1_1/ UserWarning: Module novaclient.v1_1 is deprecated (taken as a basis for novaclient.v2). The preferable way to get client class or object you can find in novaclient.client module. warnings.warn("Module novaclient.v1_1 is deprecated (taken as a basis for "23:59
clarkbhttp:// hopefully that graph will look better23:59
*** dboik_ has quit IRC23:59
*** wolsen_ is now known as wolsen23:59
jeblairlike i get the general sense that something about client deprecation is being communicated to me23:59

Generated by 2.14.0 by Marius Gedminas - find it at!