Sporadic 500 errors....error 232?

[cwb124 Profile]

I have a sinatra site being hosted by Helicon Zoo. This site accepts rest calls (post, put, delete currently) and then calls out to powershell to do stuff. Randomly we are getting 500 errors and I don't know how to resolve and have a stable system. Here is the verbose 500 error:



Job Failed: {:message=>"VmiBuild Error: target => vmi-po-1p.cable.mycompany.com/vmi-service/deploy/a48c3496-169d-11e5-ae8a-0050568c6218, error => 500 Internal Server Error:
500
Error Helicon Zoo module has caught up an error. Please see the details below.
Worker Status
Ready to shutdown
Windows error
The pipe is being closed.\r (ERROR CODE: 232)\r
Internal module error
message: Application backend write error.\rtype: ZooException\rfile: App\\Jobs\\JobBase.cpp\rline: 443\rversion: 3.1.98.538\r
STDERR
Empty stderr
Environment variables

ALLUSERSPROFILE C:\\ProgramData
APPDATA C:\\WINDOWS\\system32\\config\\systemprofile\\AppData\\Roaming
APPL_ID vmi-service
APPL_PHYSICAL_PATH c:\\inetpub\\wwwroot\\vmi-service
APPL_PHYSICAL_SHORT_PATH c:\\inetpub\\wwwroot\\VMI-SE~1
APPL_VIRTUAL_PATH /vmi-service
APP_POOL_CONFIG C:\\inetpub\\temp\\apppools\\VMI\\VMI.config
APP_POOL_ID VMI
CHOCOLATEYINSTALL C:\\ProgramData\\chocolatey
COMMONPROGRAMFILES C:\\Program Files\\Common Files
COMMONPROGRAMFILES(X86) C:\\Program Files (x86)\\Common Files
COMMONPROGRAMW6432 C:\\Program Files\\Common Files
COMPUTERNAME VMI-PO-1P
COMSPEC C:\\WINDOWS\\system32\\cmd.exe
CONSOLE_URL console
CURRENT_DIRECTORY c:\\inetpub\\wwwroot\\VMI-SE~1
DEPLOY_FILE deploy.rb
ERROR_LOG_DIR logs
FP_NO_HOST_CHECK NO
GEM_HOME c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
GEM_PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
HOME c:\\inetpub\\wwwroot\\VMI-SE~1
HOST 127.0.0.1
IIS_BINDNGS http=*:80:
INSTANCE_ID 1
LOCALAPPDATA C:\\WINDOWS\\system32\\config\\systemprofile\\AppData\\Local
NUMBER_OF_PROCESSORS 2
OS Windows_NT
PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME\\bin;C:\\Ruby19\\bin;C:\\Ruby19\\devkit\\bin;C:\\Ruby19\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\Ruby;C:\\Windows\\Ruby\\bin;C:\\Program Files (x86)\\Symantec\\Critical System Protection\\Agent\\IPS\\bin;C:\\ProgramData\\chocolatey\\bin;C:\\mycompanytools\\Puppet\\bin
PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.RB;.RBW
PROCESSOR_ARCHITECTURE AMD64
PROCESSOR_IDENTIFIER Intel64 Family 6 Model 47 Stepping 2, GenuineIntel
PROCESSOR_LEVEL 6
PROCESSOR_REVISION 2f02
PROGRAMDATA C:\\ProgramData
PROGRAMFILES C:\\Program Files
PROGRAMFILES(X86) C:\\Program Files (x86)
PROGRAMW6432 C:\\Program Files
PSMODULEPATH C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules\\;C:\\Program Files\\Common Files\\mycompany\\PowerShellModules
PUBLIC C:\\Users\\Public
RACK_ENV production
RAILS_RELATIVE_URL_ROOT /vmi-service
SITE_NAME DEFAULT WEB SITE
SITE_PHYSICAL_PATH C:\\inetpub\\wwwroot
SYSTEMDRIVE C:
SYSTEMROOT C:\\WINDOWS
TEMP C:\\WINDOWS\\TEMP
TMP C:\\WINDOWS\\TEMP
UATDATA C:\\Windows\\CCM\\UATData\\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77
USERDOMAIN CABLE
USERNAME VMI-PO-1P$
USERPROFILE C:\\WINDOWS\\system32\\config\\systemprofile
WINDIR C:\\WINDOWS
WORKER_REQUEST_TIMEOUT 7200



Helicon Zoo engine settings
\r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r \r
appName ruby.project
appFullPath C:\\Ruby19\\bin\\ruby.exe
appDirectory c:\\inetpub\\wwwroot\\VMI-SE~1
appArguments C:\\Zoo\\Workers\\ruby\\zoorack.rb
deployArguments
transportName NamedPipe
protocol fastcgi
scriptProcessor ruby.1.9.rack
errorLogDir c:\\inetpub\\wwwroot\\vmi-service\\logs
postBufferDir
watchFileChangesMask
host 127.0.0.1
port %PORT%
proxyPass
minInstances 1
maxInstances 2
minRequestsPerWorker 3
maxRequestsPerWorker 15
portLower 49152
portUpper 65535
workerRequestTimeout 7200
timeLimit 0
postBuffer 1024
memoryLimit 0
gracefulShutdownTimeout 30



Web.config variables

APPL_ID vmi-service
APPL_PHYSICAL_PATH c:\\inetpub\\wwwroot\\vmi-service
APPL_PHYSICAL_SHORT_PATH c:\\inetpub\\wwwroot\\VMI-SE~1
APPL_VIRTUAL_PATH /vmi-service
CONSOLE_URL console
CURRENT_DIRECTORY c:\\inetpub\\wwwroot\\VMI-SE~1
DEPLOY_FILE deploy.rb
ERROR_LOG_DIR logs
GEM_HOME c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
GEM_PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
HOME c:\\inetpub\\wwwroot\\VMI-SE~1
HOST 127.0.0.1
IIS_BINDNGS http=*:80:
INSTANCE_ID 1
PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME\\bin;C:\\Ruby19\\bin;C:\\Ruby19\\devkit\\bin;C:\\Ruby19\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\Ruby;C:\\Windows\\Ruby\\bin;C:\\Program Files (x86)\\Symantec\\Critical System Protection\\Agent\\IPS\\bin;C:\\ProgramData\\chocolatey\\bin;C:\\mycompanytools\\Puppet\\bin
RACK_ENV production
RAILS_RELATIVE_URL_ROOT /vmi-service
SITE_NAME DEFAULT WEB SITE
SITE_PHYSICAL_PATH C:\\inetpub\\wwwroot
WORKER_REQUEST_TIMEOUT 7200

Request variables
There are no variables
Troubleshooting

Support

", :exception_class=>"ProvWorkersError", :backtrace=>"/app/support/copa-provisioning-workers/lib/workflow/vmi_build.rb:69:in `block in submit_request'/app/support/copa-provisioning-workers/lib/rest.rb:136:in `call'/app/support/copa-provisioning-workers/lib/rest.rb:136:in `rescue in make_call'/app/support/copa-provisioning-workers/lib/rest.rb:94:in `make_call'/app/support/copa-provisioning-workers/lib/rest.rb:53:in `post'/app/support/copa-provisioning-workers/lib/workflow/vmi_build.rb:65:in `submit_request'/app/support/copa-provisioning-workers/lib/workflow/vmi_build.rb:43:in `deploy'/app/support/copa-provisioning-workers/lib/workflow/steel_cloud_build.rb:62:in `build_vm'/app/support/copa-provisioning-workers/lib/workflow/steel_cloud_build.rb:36:in `perform'/app/support/copa-provisioning-workers/lib/workers/cloud_build.rb:29:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-status-0.5.0/lib/resque/plugins/status.rb:161:in `safe_perform!'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-status-0.5.0/lib/resque/plugins/status.rb:137:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in '/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'/app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/bin/rake:33:in `'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/bin/rake:23:in `load'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/bin/rake:23:in `
'"}

[Anton]

Hello,

Can you please look at Windows Event Viewer and look for any error messages for the time when this crash occurred.
This would be really helpful.

[cwb124 Profile]

There are zero relevant entries in the event log when this error occurs.

[Anton]

Can you note any specific actions that lead to this error?
Anything that is done before the error occurs?

[cwb124 Profile]

Nothing abnormal that leads to this error. Event logs are clear. No services have restarted, nothing happening on this machine. This service gets dozens of calls per day. Sometimes it just throws this 500. I absolutely can't figure it out.

[cwb124 Profile]

Is error code 232 an internal Helicon Zoo error, and if so what does it mean? We are seeing several 232 500 errors today.

[Anton]

Hello,

"The pipe is being closed.\r (ERROR CODE: 232)"

This is a sporadic Ruby-related error.
We are trying to figure out the reason, but cannot get enough information to be able to reproduce it.
Anyway, our developers are working on figuring it out.

[cwb124 Profile]

Had another error today, very similar but different error. Error code 995 this time. Any clues?


Job Failed: {:message=>"VmiBuild Error: target => vmi-po-1p.company/vmi-service/deploy/2bf97d26-2594-11e5-add8-0050568c6218, error => 500 Internal Server Error:

Error
Helicon Zoo module has caught up an error. Please see the details below.

Worker Status

GracefulShutdownTimeout. Begin force shutdown

Windows error

The I/O operation has been aborted because of either a thread exit or an application request.(ERROR CODE: 995)

Internal module error

message: Application backend read Error.type: ZooExceptionfile: App\\Jobs\\JobBase.cpp line: 531 version: 3.1.98.538
STDERR

Empty stderr

Environment variables

ALLUSERSPROFILE C:\\ProgramData
APPDATA C:\\WINDOWS\\system32\\config\\systemprofile\\AppData\oaming
APPL_ID vmi-service
APPL_PHYSICAL_PATH c:\\inetpub\\wwwroot\\vmi-service
APPL_PHYSICAL_SHORT_PATH c:\\inetpub\\wwwroot\\VMI-SE~1
APPL_VIRTUAL_PATH /vmi-service
APP_POOL_CONFIG C:\\inetpub\\temp\\apppools\\VMI\\VMI.config
APP_POOL_ID VMI
CHOCOLATEYINSTALL C:\\ProgramData\\chocolatey
COMMONPROGRAMFILES C:\\Program Files\\Common Files
COMMONPROGRAMFILES(X86) C:\\Program Files (x86)\\Common Files
COMMONPROGRAMW6432 C:\\Program Files\\Common Files
COMPUTERNAME VMI-PO-1P
COMSPEC C:\\WINDOWS\\system32\\cmd.exe
CONSOLE_URL console
CURRENT_DIRECTORY c:\\inetpub\\wwwroot\\VMI-SE~1
DEPLOY_FILE deploy.rb
ERROR_LOG_DIR logs
FP_NO_HOST_CHECK NO
GEM_HOME c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
GEM_PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
HOME c:\\inetpub\\wwwroot\\VMI-SE~1
HOST 127.0.0.1
IIS_BINDNGS http=*:80:
INSTANCE_ID 1
LOCALAPPDATA C:\\WINDOWS\\system32\\config\\systemprofile\\AppData\\Local
NUMBER_OF_PROCESSORS 2
OS Windows_NT
PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME\\bin;C:\uby19\\bin;C:\uby19\\devkit\\bin;C:\uby19\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\uby;C:\\Windows\uby\\bin;C:\\Program Files (x86)\\Symantec\\Critical System Protection\\Agent\\IPS\\bin;C:\\ProgramData\\chocolatey\\bin;C:\\companytools\\Puppet\\bin
PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.RB;.RBW
PROCESSOR_ARCHITECTURE AMD64
PROCESSOR_IDENTIFIER Intel64 Family 6 Model 47 Stepping 2, GenuineIntel
PROCESSOR_LEVEL 6
PROCESSOR_REVISION 2f02
PROGRAMDATA C:\\ProgramData
PROGRAMFILES C:\\Program Files
PROGRAMFILES(X86) C:\\Program Files (x86)
PROGRAMW6432 C:\\Program Files
PSMODULEPATH C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules\\;C:\\Program Files\\Common Files\\company\\PowerShellModules
PUBLIC C:\\Users\\Public
RACK_ENV production
RAILS_RELATIVE_URL_ROOT /vmi-service
SITE_NAME DEFAULT WEB SITE
SITE_PHYSICAL_PATH C:\\inetpub\\wwwroot
SYSTEMDRIVE C:
SYSTEMROOT C:\\WINDOWS
TEMP C:\\WINDOWS\\TEMP
TMP C:\\WINDOWS\\TEMP
UATDATA C:\\Windows\\CCM\\UATData\\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77
USERDOMAIN CABLE
USERNAME VMI-PO-1P$
USERPROFILE C:\\WINDOWS\\system32\\config\\systemprofile
WINDIR C:\\WINDOWS
WORKER_REQUEST_TIMEOUT 7200



Helicon Zoo engine settings

appName ruby.project
appFullPath C:\uby19\\bin\uby.exe
appDirectory c:\\inetpub\\wwwroot\\VMI-SE~1
appArguments C:\\Zoo\\Workers\uby\\zoorack.rb
deployArguments
transportName NamedPipe
protocol fastcgi
scriptProcessor ruby.1.9.rack
errorLogDir c:\\inetpub\\wwwroot\\vmi-service\\logs
postBufferDir
watchFileChangesMask
host 127.0.0.1
port %PORT%
proxyPass
minInstances 1
maxInstances 2
minRequestsPerWorker 3
maxRequestsPerWorker 15
portLower 49152
portUpper 65535
workerRequestTimeout 7200
timeLimit 0
postBuffer 1024
memoryLimit 0
gracefulShutdownTimeout 30



Web.config variables

APPL_ID vmi-service
APPL_PHYSICAL_PATH c:\\inetpub\\wwwroot\\vmi-service
APPL_PHYSICAL_SHORT_PATH c:\\inetpub\\wwwroot\\VMI-SE~1
APPL_VIRTUAL_PATH /vmi-service
CONSOLE_URL console
CURRENT_DIRECTORY c:\\inetpub\\wwwroot\\VMI-SE~1
DEPLOY_FILE deploy.rb
ERROR_LOG_DIR logs
GEM_HOME c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
GEM_PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME
HOME c:\\inetpub\\wwwroot\\VMI-SE~1
HOST 127.0.0.1
IIS_BINDNGS http=*:80:
INSTANCE_ID 1
PATH c:\\inetpub\\wwwroot\\VMI-SE~1\\GEM_HOME\\bin;C:\uby19\\bin;C:\uby19\\devkit\\bin;C:\uby19\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\uby;C:\\Windows\uby\\bin;C:\\Program Files (x86)\\Symantec\\Critical System Protection\\Agent\\IPS\\bin;C:\\ProgramData\\chocolatey\\bin;C:\\companytools\\Puppet\\bin
RACK_ENV production
RAILS_RELATIVE_URL_ROOT /vmi-service
SITE_NAME DEFAULT WEB SITE
SITE_PHYSICAL_PATH C:\\inetpub\\wwwroot
WORKER_REQUEST_TIMEOUT 7200



Request variables
%RequestVariables%









Troubleshooting

Support








", :exception_class=>"ProvWorkersError", :backtrace=>"/app/support/copa-provisioning-workers/lib/workflow/vmi_build.rb:69:in `block in submit_request'/app/support/copa-provisioning-workers/lib/rest.rb:136:in `call'/app/support/copa-provisioning-workers/lib/rest.rb:136:in `rescue in make_call'/app/support/copa-provisioning-workers/lib/rest.rb:94:in `make_call'/app/support/copa-provisioning-workers/lib/rest.rb:53:in `post'/app/support/copa-provisioning-workers/lib/workflow/vmi_build.rb:65:in `submit_request'/app/support/copa-provisioning-workers/lib/workflow/vmi_build.rb:43:in `deploy'/app/support/copa-provisioning-workers/lib/workflow/steel_cloud_build.rb:62:in `build_vm'/app/support/copa-provisioning-workers/lib/workflow/steel_cloud_build.rb:36:in `perform'/app/support/copa-provisioning-workers/lib/workers/cloud_build.rb:29:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-status-0.5.0/lib/resque/plugins/status.rb:161:in `safe_perform!'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-status-0.5.0/lib/resque/plugins/status.rb:137:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in '/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'/app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/gems/rake-10.4.2/bin/rake:33:in `'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/bin/rake:23:in `load'/app/support/copa-provisioning-workers/vendor/bundle/ruby/1.9.1/bin/rake:23:in `
'"}

[Anton]

Hello,

Well, it seems that the origin of the problem is quite the same:
"The I/O operation has been aborted because of either a thread exit or an application request"
which means that some application crashed and Zoo tries to read its "Empty stderr"
You should try monitoring which apps are run by Zoo and how they crash.
BTW, in this last case, it might not actually be the error - it's GracefulShutdownTimeout, which means that worker didn't wait until normal shutdown and forced it.