Friday, 2015-11-06

*** xcv <xcv!~xcv@2a02:2450:102b:5b:ed60:9ffb:6452:ede3> has joined #immutant00:10
*** xcv <xcv!~xcv@2a02:2450:102b:5b:ed60:9ffb:6452:ede3> has quit IRC (Ping timeout: 252 seconds)00:14
*** conan <conan!~conan@mdproctor.plus.com> has quit IRC (Quit: ["Textual IRC Client: www.textualapp.com"])00:26
*** conan <conan!~conan@mdproctor.plus.com> has joined #immutant00:27
*** xcv <xcv!~xcv@37.120.30.253> has joined #immutant00:50
*** xcv <xcv!~xcv@37.120.30.253> has quit IRC (Ping timeout: 250 seconds)00:55
*** jcrossley3 is now known as jcrossley3-away00:59
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has joined #immutant01:46
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)02:23
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant03:17
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 244 seconds)03:31
*** lnostdal <lnostdal!~lnostdal@2001:8a0:491c:2601:2c0:caff:fe58:5c47> has quit IRC (Ping timeout: 246 seconds)05:36
*** lnostdal <lnostdal!~lnostdal@2001:8a0:4922:e501:2c0:caff:fe58:5c47> has joined #immutant05:50
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant06:02
*** deadghost <deadghost!~deadghost@122.167.219.181> has joined #immutant07:28
*** mgoldmann|away is now known as mgoldmann07:59
*** deadghost <deadghost!~deadghost@122.167.219.181> has quit IRC (Ping timeout: 250 seconds)08:15
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)09:46
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant09:47
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 240 seconds)09:51
*** dm3 <dm3!~dm3@pub082136076233.dh-hfc.datazug.ch> has joined #immutant10:15
*** qwerty_nor <qwerty_nor!~Thunderbi@37.229.49.175> has joined #immutant10:42
*** bobmcw <bobmcw!~bobmcw@redhat/jboss/bobmcw> has joined #immutant13:00
*** bbrowning_away is now known as bbrowning13:12
*** tcrawley-away is now known as tcrawley13:26
*** jcrossley3-away is now known as jcrossley313:31
jcrossley3tcrawley: so i think i can explain the 2 1 3 ordering13:32
*** lance|afk is now known as lanceball13:33
jcrossley3maybe13:33
jcrossley3the order of the TC requests reflect the invocation of offset13:34
jcrossley3first the host controller, then the sys-props13:34
jcrossley3because both offset and resolve-port calls sys-props13:34
jcrossley3but that means we don't see the TC request for the subsequent socket-binding call13:35
tcrawleyTC?13:35
jcrossley3your log message13:36
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has joined #immutant13:36
jcrossley3tcrawley: i think we should move your first TC log of the request *above* the client/post call, which may be somehow failing13:37
tcrawleyah. why don't we see the request? I'm not following.13:37
tcrawleyokay, I'll move it up, and maybe add a smattering of logging elsewhere13:37
jcrossley3well, i would've thought client/post would return null if it fails, but maybe it tosses and we're eating it somewhere?13:38
tcrawleywe are setting :throw-exceptions false on the client/post call13:38
tcrawleybut who nows13:39
jcrossley3yeah, hence the expected null return13:39
jcrossley3but i can't come up with any other explanation for the request log ordering13:39
tcrawleygotcha13:39
tcrawleyI'll add logging to see what fns are triggering the api calls, and log their args13:40
jcrossley3thx!13:43
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant13:50
*** agarman <agarman!~agarman@169.130.108.20> has joined #immutant14:05
projectodd-ciProject immutant2-incremental-integs build #188: SUCCESS in 38 min: https://projectodd.ci.cloudbees.com/job/immutant2-incremental-integs/188/14:28
jcrossley3dammit!14:33
tcrawleyI know, right?14:33
jcrossley3starting another?14:33
tcrawleyyeah, I want to make a couple of changes to the debug output first, then I'll restart14:34
jcrossley3kk!14:34
jcrossley3any particular reason you're not committing these?14:35
tcrawleyI wanted to make sure I didn't break anything before pushing. I'll push these changes14:35
jcrossley3i don't care too much other than matching up the line #'s14:36
tcrawleyright, my plan was to push when we had something worth analyzing :)14:36
jcrossley3kk!14:37
jcrossley3i'm fine if you want to branch it to avoid a bunch of commit history noise14:37
tcrawleygood call, wdm14:37
*** tcrawley is now known as tcrawley-away14:43
*** tcrawley-away is now known as tcrawley14:54
projectodd-ciProject immutant2-incremental-integs build #189: SUCCESS in 25 min: https://projectodd.ci.cloudbees.com/job/immutant2-incremental-integs/189/15:06
*** tcrawley is now known as tcrawley-away15:06
*** tcrawley-away is now known as tcrawley15:08
*** tcrawley is now known as tcrawley-away15:13
*** tcrawley-away is now known as tcrawley15:14
*** jcrossle_ <jcrossle_!~user@71-90-202-1.dhcp.stls.mo.charter.com> has joined #immutant15:15
*** jcrossley3 <jcrossley3!~user@71-90-202-1.dhcp.stls.mo.charter.com> has quit IRC (Ping timeout: 252 seconds)15:16
*** jcrossle_ is now known as jcrossley315:21
projectodd-ciProject immutant2-incremental-integs build #190: FAILURE in 12 min: https://projectodd.ci.cloudbees.com/job/immutant2-incremental-integs/190/15:24
tcrawleyjcrossley3: when things fail, we're asking for the port for a null server name. when they succeed, we never do that15:39
tcrawleyhttps://github.com/immutant/jboss-as-management/tree/port-debugging, btw15:39
jcrossley3huh15:39
tcrawleyah! https://github.com/immutant/immutant/tree/master/integration-tests/test-clustering/integs/cluster_test.clj#L7115:40
tcrawleyif that deref times out, we end up with a nil server name15:40
tcrawleywe should probably throw if host1 is nil15:41
jcrossley3hah!15:41
jcrossley3good catch!15:41
tcrawleyso the real question is: why is that msg/request call failing?15:41
jcrossley360s?!15:42
tcrawleyI did make a change around request lately: https://github.com/projectodd/wunderboss/commit/b82cdc43dceeefa5c1c0d3caedfbc6c90f6a786115:42
jbossbotgit [wunderboss] b82cdc4.. Toby Crawley Only reuse the context for the request router if it is remote [IMMUTANT-589]15:42
jbossbotjira [IMMUTANT-589] Only reuse the context given to request for the response listener if it is remote [Resolved (Done) Bug, Major, Unassigned] https://issues.jboss.org/browse/IMMUTANT-58915:42
tcrawleybut that's a remote context, so it should behave the same way as before15:43
jcrossley3i'm guessing a 1s sleep prior to that let form will solve all our problems, i.e. MSP15:43
*** hjrnunes <hjrnunes!~hjrnunes@80.193.116.101> has joined #immutant15:46
*** hjrnunes_ <hjrnunes_!~hjrnunes@80.193.116.101> has joined #immutant15:47
tcrawleymaybe? we already wait a minute15:47
tcrawleylet me see if there are any messaging errors in the log15:47
jcrossley3tcrawley: is it possible that the responder hasn't been created by the time the requestor is invoked?15:47
tcrawleyif so, it should just sit on the queue until a listener comes along15:48
tcrawleyhmm, we have: 10:21:35.953 INFO  [org.projectodd.wunderboss.messaging] (main) Starting listener for '/queue/cache' concurrency=1615:50
tcrawleyTC: (mgt/port jboss_as.management.Domain@78457da :http null) ; pre-memoized15:50
tcrawleyTC: (mgt/port jboss_as.management.Domain@78457da :http null) ; memoized15:50
tcrawley10:22:36,141 TC: (resolve-port http://localhost:10057/management :http null)15:50
tcrawleythat "Starting listener" message is likely the response router for the request15:50
tcrawleyand the time there is < 200ms from the resolve-port call15:51
tcrawleyso we maybe burn most of the 60s waiting for messaging bits to come up?15:51
tcrawleyerr, nevermind15:52
* tcrawley didn't notice the 21->22 between those messages :)15:52
tcrawleyhmm, /me also wonders if response routers should have the default concurrency, but that's a different issue15:52
*** qwerty_nor <qwerty_nor!~Thunderbi@37.229.49.175> has quit IRC (Ping timeout: 240 seconds)15:53
jcrossley3:)15:55
jcrossley3tcrawley: so those TC messages are occurring at 10:21 or 10:22?15:56
jcrossley3the mgt/port calls, i mean15:57
tcrawleythey would have to be at 10:22:36, since mgt/port immediately calls resolve-port, and there wouldn't be anything in between that sleeps15:58
jcrossley3so we need to explain that 60s16:01
jcrossley3tcrawley: what if the queue doesn't exist?16:04
jcrossley3what if our order is...16:04
jcrossley31 client context16:04
jcrossley32 client request16:04
jcrossley33 server queue created16:04
jcrossley34 server listener created16:05
tcrawleyit seems like the client would complain if you tried to publish to a non-existent remote queue though16:06
jcrossley3i feel like that request is DOA if the queue isn't there16:06
jcrossley3i guess we can ask Mr Empiracle16:07
tcrawleyfuck. I think it's a bug in wildfly's singleton supprt16:19
jcrossley3really?16:20
tcrawleyfrom server-one: 2015-11-06 10:21:33,524 INFO  [org.wildfly.clustering.server] (ServerService Thread Pool -- 71) WFLYCLSV0003: master:server-two elected as the singleton provider of the wunderboss.singleton."ROOT.war".daemon-cache-status service16:20
jcrossley3how'd you get there?16:20
tcrawleyfrom server-two: 2015-11-06 10:21:33,505 INFO  [org.wildfly.clustering.server] (remote-thread--p9-t1) WFLYCLSV0003: master:server-one elected as the singleton provider of the wunderboss.singleton."ROOT.war".daemon-cache-status service16:20
tcrawleythe responder for /queue/cache is started/stopped by singleton-daemon16:20
tcrawleyeach node thinks the other is master, so doesn't start the daemon16:21
jcrossley3huh16:21
tcrawleyso there is no responder16:21
jcrossley3is there an open issue?16:21
tcrawleyhaven't looked yet16:21
jcrossley3good catch, regardless :)16:21
tcrawleyit's odd that they are on different threadpools16:22
jcrossley3they're different servers?16:22
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)16:22
tcrawleyyes, but you would think the threadpools would have similar names16:22
jcrossley3agree16:23
tcrawleyI'll see if there is an issue16:23
jcrossley3maybe we're doing something unexpected? all bets are off if we're starting threads.16:23
tcrawleyidu16:23
jcrossley3which i guess the request must be16:23
tcrawleywe start the daemon from -main16:24
jcrossley3you're never supposed to start your own threads in a JEE container16:24
tcrawleyhttps://github.com/immutant/immutant/tree/master/integration-tests/src/integs/cluster.clj#L4216:24
jcrossley3oh, right16:24
jcrossley3the request wouldn't be in the container, duh16:24
tcrawleybrb16:25
*** mgoldmann is now known as mgoldmann|away16:42
tcrawleyjcrossley3: I think it is this: https://issues.jboss.org/browse/WFLY-510816:44
jbossbotjira [WFLY-5108] SingletonService can start multiple master nodes if election policies are not identical [Resolved (Done) Bug, Major, Paul Ferraro] https://issues.jboss.org/browse/WFLY-510816:44
tcrawleywe're using a random election policy, and, according to that issue, each node determines the master individually16:45
tcrawleyso it's possible for each node to elect the other16:46
tcrawleyit should be fixed in 10, so maybe we use the simple election policy if < 1016:46
jcrossley3where do we set that policy?16:46
tcrawleywhich selects the first node, and the order of that list should be stable from jgroups16:47
tcrawleylet me find it16:47
tcrawleyhttps://github.com/projectodd/wunderboss/tree/master/wildfly/singletons/src/main/java/org/projectodd/wunderboss/as/singletons/SingletonHelper.java#L11516:48
tcrawleywe use our own policy that I pulled from WF9: https://github.com/projectodd/wunderboss/tree/master/wildfly/singletons/src/main/java/org/projectodd/wunderboss/as/singletons/RandomSingletonElectionPolicy.java#L2816:49
tcrawleysince it doesn't exist in 816:49
tcrawleyand I assumed it was safe to use16:49
jcrossley3great detective work, tcrawley16:51
tcrawleythanks!16:51
tcrawleyI'll switch us to the simple policy, and we'll see how that fairs16:51
*** jcrossle_ <jcrossle_!~user@71-90-202-1.dhcp.stls.mo.charter.com> has joined #immutant17:00
*** jcrossley3 <jcrossley3!~user@71-90-202-1.dhcp.stls.mo.charter.com> has quit IRC (Ping timeout: 252 seconds)17:04
*** GitHub184 <GitHub184!~GitHub184@192.30.252.42> has joined #immutant17:10
GitHub184[wunderboss] tobias pushed 1 new commit to master: http://git.io/v83zv17:10
GitHub184wunderboss/master d172d0a Toby Crawley: Use the simple election policy for singletons when in WF 8/9...17:10
*** GitHub184 <GitHub184!~GitHub184@192.30.252.42> has left #immutant17:10
jbossbotTitle: Use the simple election policy for singletons when in WF 8/9 · projectodd/wunderboss@d172d0a · GitHub17:10
projectodd-ciProject wunderboss-incremental build #298: SUCCESS in 4 min 33 sec: https://projectodd.ci.cloudbees.com/job/wunderboss-incremental/298/17:15
projectodd-ciToby Crawley: Use the simple election policy for singletons when in WF 8/917:15
*** jcrossle_ is now known as jcrossley317:21
jbossbotgit [immutant] push master d14a8bd.. Toby Crawley Throw if the cluster host is nil...17:22
jbossbotgit [immutant] push master febffc1.. Toby Crawley Update to a newer wunderboss that should solve singleton inconsistencies17:22
jbossbotgit [immutant] push master URL: http://github.com/immutant/immutant/compare/be8e100...febffc117:22
*** zimablue <zimablue!~user@100.218.208.46.dyn.plus.net> has joined #immutant17:26
*** dm3_ <dm3_!~dm3@pub082136076233.dh-hfc.datazug.ch> has joined #immutant17:30
projectodd-ciProject immutant2-incremental build #683: SUCCESS in 8 min 0 sec: https://projectodd.ci.cloudbees.com/job/immutant2-incremental/683/17:30
*** dm3 <dm3!~dm3@pub082136076233.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 250 seconds)17:34
projectodd-ciProject immutant2-incremental-integs-eap build #230: STILL FAILING in 1 min 51 sec: https://projectodd.ci.cloudbees.com/job/immutant2-incremental-integs-eap/230/17:35
projectodd-ci* Toby Crawley: Throw if the cluster host is nil17:35
projectodd-ci* Toby Crawley: Update to a newer wunderboss that should solve singleton inconsistencies17:35
*** dm3_ <dm3_!~dm3@pub082136076233.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 252 seconds)17:35
projectodd-ciProject immutant2-incremental-integs build #191: STILL FAILING in 24 min: https://projectodd.ci.cloudbees.com/job/immutant2-incremental-integs/191/17:55
projectodd-ci* Toby Crawley: Throw if the cluster host is nil17:55
projectodd-ci* Toby Crawley: Update to a newer wunderboss that should solve singleton inconsistencies17:55
*** mgaare_ is now known as mgaare18:25
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant18:40
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)18:52
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant18:53
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)18:55
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant18:55
*** lanceball is now known as lance|afk18:56
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 240 seconds)18:59
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 250 seconds)19:02
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant19:13
projectodd-ciProject immutant2-incremental-integs build #192: STILL FAILING in 39 min: https://projectodd.ci.cloudbees.com/job/immutant2-incremental-integs/192/19:15
*** conan <conan!~conan@mdproctor.plus.com> has quit IRC (Quit: Computer has gone to sleep.)19:35
jcrossley3what a dumpster fire19:36
tcrawleyha19:37
tcrawleyfighting with cardboard19:38
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant19:39
*** deadghost <deadghost!~deadghost@122.166.95.51> has joined #immutant20:19
*** lance|afk is now known as lanceball20:20
*** conan <conan!~conan@mdproctor.plus.com> has joined #immutant20:21
*** conan <conan!~conan@mdproctor.plus.com> has quit IRC (Ping timeout: 246 seconds)20:26
jcrossley3indeed20:27
*** jcrossley3 is now known as jcrossley3-away20:28
*** conan <conan!~conan@mdproctor.plus.com> has joined #immutant20:28
*** zimablue <zimablue!~user@100.218.208.46.dyn.plus.net> has quit IRC (Ping timeout: 240 seconds)20:38
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 272 seconds)20:47
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)21:17
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant21:18
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 244 seconds)21:22
*** deadghost <deadghost!~deadghost@122.166.95.51> has quit IRC (Ping timeout: 272 seconds)21:23
*** jcrossley3-away is now known as jcrossley321:28
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has joined #immutant21:33
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has joined #immutant21:41
*** bbrowning is now known as bbrowning_away21:42
*** lanceball is now known as lance|afk21:47
*** agarman <agarman!~agarman@169.130.108.20> has quit IRC (Ping timeout: 264 seconds)21:59
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)22:01
*** tcrawley is now known as tcrawley-away22:02
*** zimablue <zimablue!~user@100.218.208.46.dyn.plus.net> has joined #immutant22:05
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant22:21
*** dm3 <dm3!~dm3@pub082136109248.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)22:27
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 265 seconds)22:29
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has joined #immutant22:45
*** bobmcw <bobmcw!~bobmcw@redhat/jboss/bobmcw> has quit IRC (Remote host closed the connection)22:46
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant23:29
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 255 seconds)23:41
*** bostonaholic <bostonaholic!~textual@97-122-166-12.hlrn.qwest.net> has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)23:48
*** jcrossley3 is now known as jcrossley3-away23:51

Generated by irclog2html.py 2.9.2 by Marius Gedminas - find it at mg.pov.lt!