How to perform HTML entity encoding in Java

http://www.texttacaouca.com

Status

Released 14/1/2008

Overview

Injection attacks rely on the fact that interpreters take data and execute it as commands. If an attacker can modify the data that's sent to an interpreter, they may be able to make it misbehave. One way to help prevent this from happening is to encode the attacker's data in such a way that the interpreter will not get confused. HTML entity encoding is just such an encoding mechanism for many interpreters.

This is not a guarantee by the way. It's almost certain that someone, probably from the XML/Web Services world, will create an engine that performs HTML entity decoding automatically, thus reintroducing the injection threat. However, for the time being, HTML entity encoding seems to work pretty well to prevent many types of injection.

Approach

We're going to implement a simple little method that encodes special characters. The nice .NET folks over at Microsoft had the foresight to build this into their platform, but the Java community seems to resist adding validation to the Java EE environment despite all the security issues that it could solve. View layers such as Java Server Faces, Spring-MVC, WebWork and others automatically perform HTML encoding through custom tags that is often incomplete.

For example, Spring provides both HTML and JavaScript encoding functionality (spring:message htmlEscape and htmlEscape) that can be set at the form element level. [1] HTML escape functionality in Spring can also be set at the page or servlet container. [2] Note that's Spring's default entity encoder only encodes the "big 5" and does not handle double-encoding. This code that handles this functionality was last updated in 2003.[3]

  21          {"#39", new Integer(39)}, // ' - apostrophe
  22          {"quot", new Integer(34)}, // " - double-quote
  23          {"amp", new Integer(38)}, // & - ampersand
  24          {"lt", new Integer(60)}, // < - less-than
  25          {"gt", new Integer(62)}, // > - greater-than

Encoding the "big 5" serves exactly the purpose it was designed for: prevents injecting HTML markup with ilegal characters inside tags and attribute values. However it does not prevent more elaborate injections, does not help with "out of range characters = question marks" when outputting Strings to Writers with single byte encodings, nor prevents character reinterpretation when user switches browser encoding over displayed page.--A.in.the.k 07:18, 18 March 2009 (UTC)

The best place for a more complete method of HTML entity encoding is in some kind of ValidationEngine, but since it's a good candidate for being static, it doesn't matter what class it ends up in that much.

Note that this implementation doesn't produce the special characters like & lt; or & gt; - but it's not difficult to implement with a simple lookup table.

   /* return StringBuilder and/or make Writer param and write to stream directly*/
   public static String htmlEntityEncode( String s )
   {
       StringBuilder buf = new StringBuilder(s.length());
       for ( int i = 0; i < len; i++ )
       {
           char c = s.charAt( i );
           if ( c>='a' && c<='z' || c>='A' && c<='Z' || c>='0' && c<='9' )
           {
               buf.append( c );
           }
           else
           {
               buf.append("&#").append((int)c).append(";");
           }
       }
       return buf.toString();
   }

--A.in.the.k 07:25, 19 March 2009 (UTC) When testing this simple approach on several browsers and comparing with non-escaped version we can observe several problems, specially on ISOControlCharacter ranges (0000-001F and 0080-009F).

MSIE 6.0 does display � in escaped form. All other browsers (tested on Win platform) display escaped range 0080-009F as incorrect displayable characters, mapped to local OS charset (windows-1250 in the test case).

This leads to confusion, and self question: "how and why control characters should be outputted in html." I recommend removing all nonWhitespace ISOControlCharacters from the outputted stream.

 public static StringBuilder escapeHtmlFull(String s)
 {
     StringBuilder b = new StringBuilder(s.length());
     for (int i = 0; i < s.length(); i++)
     {
       char ch = s.charAt(i);
       if (ch >= 'a' && ch <= 'z' || ch >= 'A' && ch <= 'Z' || ch >= '0' && ch <= '9')
       {
         // safe
         b.append(ch);
       }
       else if (Character.isWhitespace(ch))
       {
         // paranoid version: whitespaces are unsafe - escape
         // conversion of (int)ch is naive
         b.append("&#").append((int) ch).append(";");
       }
       else if (Character.isISOControl(ch))
       {
         // paranoid version:isISOControl which are not isWhitespace removed !
         // do nothing do not include in output !
       }
       else
       {
         // paranoid version
         // the rest is unsafe, including <127 control chars
         b.append("&#" + (int) ch + ";");
       }
     }
     return b;
  }

Another issue brought with 1.5 is support of Unicode supplementary characters. In short this means that unicode characters are not chars but ints. The code needs to be fixed again:

 public static StringBuilder escapeHtmlFull(String s)
 {
     StringBuilder b = new StringBuilder(s.length());
     for (int i = 0; i < s.length(); i++)
     {
       char ch = s.charAt(i);
       if (ch >= 'a' && ch <= 'z' || ch >= 'A' && ch <= 'Z' || ch >= '0' && ch <= '9')
       {
         // safe
         b.append(ch);
       }
       else if (Character.isWhitespace(ch))
       {
         // paranoid version: whitespaces are unsafe - escape
         // conversion of (int)ch is naive
         b.append("&#").append((int) ch).append(";");
       }
       else if (Character.isISOControl(ch))
       {
         // paranoid version:isISOControl which are not isWhitespace removed !
         // do nothing do not include in output !
       }
       else if (Character.isHighSurrogate(ch))
       {
         int codePoint;
         if (i + 1 < s.length() && Character.isSurrogatePair(ch, s.charAt(i + 1))
           && Character.isDefined(codePoint = (Character.toCodePoint(ch, s.charAt(i + 1)))))
         {
            b.append("&#").append(codePoint).append(";");
         }
         else
         {
           log("bug:isHighSurrogate");
         }
         i++; //in both ways move forward
       }
       else if(Character.isLowSurrogate(ch))
       {
         // wrong char[] sequence, //TODO: LOG !!!
         log("bug:isLowSurrogate");
         i++; // move forward,do nothing do not include in output !
       }
       else
       {
         if (Character.isDefined(ch))
         {
           // paranoid version
           // the rest is unsafe, including <127 control chars
           b.append("&#").append((int) ch).append(";");
         }
         //do nothing do not include undefined in output!
       }
    }
    return b;
 }

Now after all, if-elses and other constructs in this method should be optimized. In this article it is left "as is" to be readable.

Libraries

• The Jakara Commons Lang package has a generic class for performing a wide range of String escaping functions.
• jTidy includes an HTMLEncode class for performing HTML encoding.