If the Target Service is returning an error with a JSON payload, then APIX-G can send it as it is to the client if the property below is set in asg.properties file:
tibco.clientVar.ASG/Response/DefaultContentType=application/json
If the error is generated in APIX-G itself, then it would send a response in JSON format if the accept header,in the incoming request, is set as:
-------------
accept: application/json, text/javascript, */*; q=0.01
-------------