Wednesday, 2015-04-29

*** sundbp <sundbp!uid18520@gateway/web/irccloud.com/x-kwnaclewqycpwiot> has quit IRC (Quit: Connection closed for inactivity)00:27
*** bbrowning_away is now known as bbrowning00:28
*** tcrawley-away is now known as tcrawley00:30
*** tcrawley is now known as tcrawley-away00:31
*** JulioBarros <JulioBarros!~juliobarr@c-50-186-32-133.hsd1.or.comcast.net> has left #immutant01:01
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant01:04
*** cap10morgan <cap10morgan!~cap10morg@2601:1:b200:1c6:c590:ff6b:3572:13d1> has joined #immutant01:50
*** bbrowning is now known as bbrowning_away01:52
*** cap10morgan <cap10morgan!~cap10morg@2601:1:b200:1c6:c590:ff6b:3572:13d1> has quit IRC (Ping timeout: 272 seconds)01:55
*** [1]cnb <[1]cnb!~cnb@pc-74-8-74-200.cm.vtr.net> has joined #immutant02:38
*** [2]cnb <[2]cnb!~cnb@pc-74-8-74-200.cm.vtr.net> has quit IRC (Ping timeout: 264 seconds)02:41
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 252 seconds)03:10
*** galderz <galderz!~galder@redhat/jboss/galderz> has joined #immutant06:13
*** mgoldmann|away is now known as mgoldmann06:27
*** galderz <galderz!~galder@redhat/jboss/galderz> has quit IRC (Quit: Leaving)06:52
*** dm3 <dm3!~dm3@pub151248158012.dh-hfc.datazug.ch> has joined #immutant07:04
*** je <je!~je@mail.natur-energi.dk> has joined #immutant07:11
*** dm3 <dm3!~dm3@pub151248158012.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)07:24
*** dm3 <dm3!~dm3@pub151248158012.dh-hfc.datazug.ch> has joined #immutant07:25
*** dm3 <dm3!~dm3@pub151248158012.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 264 seconds)07:29
*** dm3 <dm3!~dm3@pub158181119172.dh-hfc.datazug.ch> has joined #immutant08:03
*** marianoguerra <marianoguerra!~marianogu@emesene/grandpa/marianoguerra> has joined #immutant08:14
*** qwerty_nor <qwerty_nor!~Thunderbi@5.248.107.224> has joined #immutant08:51
*** jcrossle_ <jcrossle_!~user@71-90-202-246.dhcp.stls.mo.charter.com> has joined #immutant09:03
*** jcrossley3-away <jcrossley3-away!~user@71-90-202-246.dhcp.stls.mo.charter.com> has quit IRC (Remote host closed the connection)09:04
*** deadghost <deadghost!~deadghost@49.230.115.228> has quit IRC (Ping timeout: 256 seconds)09:05
*** deadghost <deadghost!~deadghost@49.230.225.236> has joined #immutant10:06
*** galderz <galderz!~galder@redhat/jboss/galderz> has joined #immutant10:12
*** tcrawley-away is now known as tcrawley10:44
*** galderz <galderz!~galder@redhat/jboss/galderz> has quit IRC (Quit: Leaving)10:45
*** jcrossle_ is now known as jcrossley3-away11:04
*** egli <egli!~user@alouette.sbs.ch> has quit IRC (Remote host closed the connection)11:49
*** galderz <galderz!~galder@redhat/jboss/galderz> has joined #immutant11:50
*** bbrowning_away is now known as bbrowning12:02
*** tcrawley is now known as tcrawley-away12:17
*** sundbp <sundbp!uid18520@gateway/web/irccloud.com/x-vbsmkztviqvhpuow> has joined #immutant12:18
*** dm3_ <dm3_!~dm3@pub158181119172.dh-hfc.datazug.ch> has joined #immutant12:18
*** dm3 <dm3!~dm3@pub158181119172.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 276 seconds)12:22
*** dm3_ <dm3_!~dm3@pub158181119172.dh-hfc.datazug.ch> has quit IRC (Ping timeout: 240 seconds)12:22
*** tcrawley-away is now known as tcrawley12:24
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant12:38
*** jcrossley3-away is now known as jcrossley312:56
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 265 seconds)12:58
*** lance|afk is now known as lanceball13:01
*** egli <egli!~user@alouette.sbs.ch> has joined #immutant13:04
*** marianoguerra <marianoguerra!~marianogu@emesene/grandpa/marianoguerra> has quit IRC (Ping timeout: 256 seconds)13:04
*** tmbrooke <tmbrooke!~tmbrooke@rrcs-24-172-72-154.midsouth.biz.rr.com> has joined #immutant13:32
*** je <je!~je@mail.natur-energi.dk> has quit IRC (Ping timeout: 256 seconds)13:37
*** galderz <galderz!~galder@redhat/jboss/galderz> has quit IRC (Quit: This computer has gone to sleep)14:21
*** cap10morgan <cap10morgan!~cap10morg@75-148-43-81-Colorado.hfc.comcastbusiness.net> has joined #immutant14:21
*** conan_ <conan_!~conan@mdproctor.plus.com> has joined #immutant14:57
*** conan <conan!~conan@mdproctor.plus.com> has quit IRC (Read error: Connection reset by peer)14:57
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant15:05
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 272 seconds)15:26
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant15:37
*** cap10morgan <cap10morgan!~cap10morg@75-148-43-81-Colorado.hfc.comcastbusiness.net> has quit IRC (Ping timeout: 244 seconds)15:45
*** deadghost <deadghost!~deadghost@49.230.225.236> has quit IRC (Ping timeout: 252 seconds)15:47
*** tmbrooke <tmbrooke!~tmbrooke@rrcs-24-172-72-154.midsouth.biz.rr.com> has quit IRC (Quit: tmbrooke)15:52
*** tcrawley is now known as tcrawley-away16:03
*** deadghost <deadghost!~deadghost@49.230.225.236> has joined #immutant16:15
*** lanceball is now known as lance|afk16:47
*** bbrowning is now known as bbrowning_away16:53
*** bbrowning_away is now known as bbrowning17:08
*** galderz <galderz!~galder@redhat/jboss/galderz> has joined #immutant17:29
*** galderz <galderz!~galder@redhat/jboss/galderz> has quit IRC (Remote host closed the connection)17:30
*** lance|afk is now known as lanceball18:02
*** tmbrooke <tmbrooke!~tmbrooke@cpe-69-132-201-26.carolina.res.rr.com> has joined #immutant18:09
*** bbrowning is now known as bbrowning_away18:12
*** dm3 <dm3!~dm3@pub151248158012.dh-hfc.datazug.ch> has joined #immutant18:12
*** tmbrooke <tmbrooke!~tmbrooke@cpe-69-132-201-26.carolina.res.rr.com> has quit IRC (Quit: tmbrooke)18:36
*** bbrowning_away is now known as bbrowning18:54
*** je <je!~je@x1-6-c0-3f-0e-f8-01-dc.cpe.webspeed.dk> has joined #immutant18:58
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 252 seconds)19:16
*** sundbp <sundbp!uid18520@gateway/web/irccloud.com/x-vbsmkztviqvhpuow> has quit IRC (Quit: Connection closed for inactivity)19:17
*** lanceball is now known as lance|afk19:20
*** moquist <moquist!~moquist@c-75-67-246-255.hsd1.nh.comcast.net> has quit IRC (Quit: WeeChat 0.4.2)19:29
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant19:32
*** tcrawley-away is now known as tcrawley19:36
*** moquist <moquist!~moquist@c-75-67-246-255.hsd1.nh.comcast.net> has joined #immutant19:36
*** lance|afk is now known as lanceball19:52
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 256 seconds)19:55
*** tmbrooke <tmbrooke!~tmbrooke@cpe-69-132-201-26.carolina.res.rr.com> has joined #immutant20:10
*** cap10morgan <cap10morgan!~cap10morg@75-148-43-81-Colorado.hfc.comcastbusiness.net> has joined #immutant20:13
cap10morganwhen deploying to wildfly, does Immutant / WildFly need the -main fn to spawn threads and return (as opposed to just blocking)?20:14
*** eric_normand <eric_normand!~eric@ip72-211-99-52.no.no.cox.net> has joined #immutant20:14
tcrawleycap10morgan: -main has to return, yes. you don't want it to block20:15
tcrawleyelse it will block deployment20:15
cap10morgantcrawley: yep, OK. that's what we just ran into. good to know! :)20:15
*** qwerty_nor <qwerty_nor!~Thunderbi@5.248.107.224> has quit IRC (Ping timeout: 244 seconds)20:17
*** bbrowning <bbrowning!~bbrowning@redhat/jboss/bbrowning> has quit IRC (Quit: Leaving)20:19
*** bbrowning <bbrowning!~bbrowning@redhat/jboss/bbrowning> has joined #immutant20:22
*** cap10morgan <cap10morgan!~cap10morg@75-148-43-81-Colorado.hfc.comcastbusiness.net> has quit IRC (Ping timeout: 256 seconds)20:33
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant20:35
eric_normandtcrawley: is there any way to undeploy an app that is stuck because the main never returns?20:43
tcrawleyeric_normand: how was it deployed?20:44
tcrawleyby copying the war to the deployments/ dir?20:44
tcrawleyor via the cli?20:44
eric_normandcurl to the /management20:44
tcrawleyyou should be able to curl the undeploy command20:44
eric_normandok, will try that20:45
eric_normandthanks20:45
tcrawleythat should undeploy it, but if it doesn't do so right away, you should be able to restart WildFly and it wouldn't deploy it in reboot I'd think20:45
tcrawleywe need to clarify that in the docs20:45
tcrawleyI'll add a @jire20:46
tcrawley@jira20:46
proddbottcrawley would be happy as a bear in a koi pond, my friend, if you would file an issue at https://issues.jboss.org/browse/IMMUTANT20:46
* tcrawley is going to write a bug tracking system called JIRE20:46
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 264 seconds)20:46
tcrawleyand brew a beer called Millar Lite20:46
tcrawleyIMMUTANT-55820:48
jbossbotjira [IMMUTANT-558] Clarify that the :main fn must return for deployment to succeed in WildFly [Open (Unresolved) Documentation, Major, Unassigned] https://issues.jboss.org/browse/IMMUTANT-55820:48
eric_normandtcrawley: does not appear to be undeploying20:50
tcrawleyare you able to restart the container?20:51
eric_normandcan't do that20:51
eric_normandit's production20:51
tcrawleyok, let me see what I can find out20:51
eric_normandthanks20:52
eric_normandis there a way for the deploy to timeout?20:52
eric_normandso we don't get in this state again?20:52
tcrawleydo you have a replacement war ready that doesn't block? you could try deploying over it20:52
eric_normandtried that20:52
eric_normanddoesn't deploy anything now20:52
bbrowninglikely some of the MSC threads are tied up20:53
tcrawleyI thought deploy would timeout - I believe it used to in AS7, but I may be remembering incorrectly20:53
bbrowningit did eventually20:53
tcrawleybbrowning: good point20:53
jcrossley3might be possible to stop the deployer via jmx console20:53
tcrawleylet me poke at that20:54
tcrawleyeric_normand: what version of WildFly is this?20:54
eric_normand8.2.020:54
jcrossley3tcrawley: should be jboss.as.deployment-scanner20:58
tcrawleyI'm working on locking it first20:58
jcrossley3er... maybe not if via cli20:59
tcrawleyjcrossley3: eric_normand: can either of you share the curl to deploy via the mgt console?21:01
* tcrawley is ashamed to say he's never done it21:01
eric_normandtcrawley: can't access to management console21:01
eric_normandtcrawley: it just loads forever21:01
tcrawleynormally, or now that deployment is blocked?21:02
eric_normandtcrawley: now21:02
tcrawleywhat command do you use to deploy normally? that's what I'm looking for21:02
eric_normandtcrawley: it has worked before21:02
eric_normandok21:02
eric_normandone sec21:02
jcrossley3tcrawley: curl -d '{"operation":"deploy","address":["deployment","simple.war"],"json.pretty":1}' "http://localhost:9990/management"21:02
jcrossley3tcrawley: you can also use jboss-as-management21:03
jcrossley3just fire it up in a repl21:03
tcrawleyeric_normand: thanks21:03
eric_normandtcrawley: first `curl -s -S -F "file=@target/${APP}.war" --digest ${WILDFLY_ADMIN_URL}/add-content`21:03
tcrawleyjcrossley3: pb21:03
eric_normandto add the war21:03
jcrossley3eric_normand: you could use it, too :)21:04
eric_normandthen: curl -s -S -H "Content-Type: application/json" -d "{\"content\":[{\"hash\": ${WILDFLY_CONTENT}}], \"address\": [{\"deployment\": \"${APP}.war\"}], \"operation\": \"add\", \"enabled\": \"true\"}" --digest ${WILDFLY_ADMIN_URL}21:04
jcrossley3https://github.com/immutant/jboss-as-management21:05
eric_normandjcrossley3: thanks21:05
eric_normandwill check out21:05
eric_normandwe're doing most of our ops in bash :)21:05
eric_normandit still would be nice to not have it get into this state21:05
eric_normandwhere no management requests respond21:06
tcrawleymost definitely21:06
*** dm3 <dm3!~dm3@pub151248158012.dh-hfc.datazug.ch> has quit IRC (Remote host closed the connection)21:10
jcrossley3tcrawley: prolly worth debugging body here just to get the curl commands: https://github.com/immutant/jboss-as-management/blob/master/src/jboss_as/api.clj#L1321:11
jcrossley3maybe21:12
*** marianoguerra <marianoguerra!~marianogu@p4FDBEDED.dip0.t-ipconnect.de> has joined #immutant21:12
*** marianoguerra <marianoguerra!~marianogu@emesene/grandpa/marianoguerra> has joined #immutant21:12
tcrawleyI just went to the mgt console directly21:12
jcrossley3i can't deal with that21:12
tcrawleycurl and jboss-as-management failed because I didn't have creds21:12
jcrossley3ah21:13
jcrossley3i knew there was a reason i used fntest instead of j-a-m directly :)21:16
*** bbrowning is now known as bbrowning_away21:16
tcrawleywell, I have a hung deployment now, but the mgt console is still responsive for me21:16
tcrawleylet me poke around21:17
jcrossley3could be you have more msc threads at your disposal, esp if they're in a vm21:18
tcrawleyyeah21:19
tcrawleyeric_normand: how many cores are available the machine/vm WF is running on?21:19
eric_normandtcrawley: let me see21:19
eric_normandtcrawley: the box has 8 cores21:22
tcrawleyit should have enough MSC threads then, so that's not the issue21:22
tcrawleywhat is your -main blocking on?21:22
tcrawleyeric_normand: is there any app deployed to that WF instance that might have a repl open?21:26
eric_normandno, I don't think so21:27
eric_normandit's an infinite loop, waiting on a core.async channel21:27
tcrawleywhat has access to the channel? anything that has already been started?21:28
tcrawleyis it tied to messaging in some way?21:28
tcrawleyI'm trying to come up with a way to send a nil to it21:29
tcrawleyfrom looking through the WF code, I'm not seeing a way we can interrupt that deploy21:30
tcrawleyunless we can kill a thread somehow. let me look at that21:30
*** tmbrooke <tmbrooke!~tmbrooke@cpe-69-132-201-26.carolina.res.rr.com> has quit IRC (Quit: tmbrooke)21:37
eric_normandlet me see21:40
eric_normandI know there's a try/catch around it21:40
*** mgoldmann is now known as mgoldmann|away21:48
tcrawleyeric_normand: I've found a way to kill that thread from gdb, but attaching from gdb suspends the process, and killing the thread from there has a high chance of segfaulting the jvm. so that's not going to work21:52
tcrawleyhow painful is restarting process?21:52
tcrawleys/process/the process/21:52
eric_normandI'm not sure21:58
eric_normandthe core.async channel is fed from a rabbitmq21:58
eric_normandso I'm going to try to send it nil21:58
*** cap10morgan <cap10morgan!~cap10morg@75-148-43-81-Colorado.hfc.comcastbusiness.net> has joined #immutant21:59
jcrossley3eric_normand: is the wildfly process currently serving user requests in production?22:01
eric_normandyes22:01
jcrossley3but not to the app in question?22:01
jcrossley3it's a little odd that the management app is hung but others aren't22:02
tcrawleyit's probably tied up all the MSC threads, which aren't used by apps themselves22:03
jcrossley3but that would only be to deploy, right? do simple queries of the model use msc threads, too?22:04
tcrawleyI don't know. but if mgt has it's own thread pool, and those threads are blocked waiting for msc threads to complete actions...22:05
eric_normand(loop []22:05
eric_normand      (try22:05
eric_normand        (message-handler (async/<!! channels/incoming))22:05
eric_normand        (catch Throwable t22:05
eric_normand          (log/error t)))22:05
eric_normand      (recur))22:05
eric_normandI can send it a message that causes an error22:06
eric_normandbut with the catch, it's not killing the thread22:06
tcrawleycrap22:06
jcrossley3worth asking in #wildfly?22:06
tcrawleyno one will answer, they all use hipchat or somesuch now22:06
tcrawleybut don't let me stop you, jcrossley322:06
jcrossley3srsly?22:07
* jcrossley3 just noticed he isn't in there :)22:07
eric_normandI think a timeout on deployments would be a good solution to this problem and would generally enhance robustness22:08
jcrossley3no argument here at all, but that doesn't solve your immediate problem22:09
eric_normandno22:09
*** je <je!~je@x1-6-c0-3f-0e-f8-01-dc.cpe.webspeed.dk> has quit IRC (Ping timeout: 272 seconds)22:09
eric_normandjust talked to cto, though22:09
eric_normandwe'll restart wildfly tomorrow morning22:09
tcrawleyunfortunately, all we could do is call kill() on the thread doing deployment, which isn't great22:09
eric_normandand hope the load balancer takes care of it22:09
eric_normandluckily there are three wildflies22:09
tcrawleyhmm, calling stop() on a thread can be bad, but it's better than the current situation22:13
tcrawleyI'll add a @jira to look in to that further22:13
proddbotWe'd be happy as a poor man with a bag of gold if you would file an issue at https://issues.jboss.org/browse/IMMUTANT22:13
tcrawleyhe's not a poor man any longer22:14
*** marianoguerra <marianoguerra!~marianogu@emesene/grandpa/marianoguerra> has quit IRC (Ping timeout: 264 seconds)22:14
*** eric_normand <eric_normand!~eric@ip72-211-99-52.no.no.cox.net> has quit IRC (Quit: eric_normand)22:15
jcrossley3tcrawley: the scanner has a deployment-timeout property. we can probably replicate whatever WF is doing22:15
tcrawleyyeah, maybe so22:15
*** tmbrooke <tmbrooke!~tmbrooke@cpe-174-110-147-78.carolina.res.rr.com> has joined #immutant22:16
tcrawleyIMMUTANT-55922:19
jbossbotjira [IMMUTANT-559] Consider timing out the :main call [Open (Unresolved) Feature Request, Major, Unassigned] https://issues.jboss.org/browse/IMMUTANT-55922:19
*** lanceball is now known as lance|afk22:25
*** cap10morgan <cap10morgan!~cap10morg@75-148-43-81-Colorado.hfc.comcastbusiness.net> has quit IRC (Quit: (null))22:26
*** tmbrooke <tmbrooke!~tmbrooke@cpe-174-110-147-78.carolina.res.rr.com> has quit IRC (Quit: tmbrooke)22:37
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has joined #immutant23:29
*** cemerick <cemerick!~cemerick@c-24-34-140-98.hsd1.ma.comcast.net> has quit IRC (Ping timeout: 244 seconds)23:43
*** tcrawley is now known as tcrawley-away23:50

Generated by irclog2html.py 2.9.2 by Marius Gedminas - find it at mg.pov.lt!