Of char, int and unicode

So my overflowing smartness and largely my ignorance made me believe that I could convert a HTML encoded string into UTF8 by doing a normal split and join. Something like this:

String testStr = "呀数达";

String[] splitStr = testStr.split("&#");

StringBuilder bldr = new StringBuilder();

for (String str : splitStr)

{

str = str.replace(";","");

bldr.append("\\u"+Integer.toHexString(Integer.valueOf(str)));

}

So I was thinking that a string like this "\u1234\u2345\u3456" as created is what I was looking for.

Well, WRONG! The thing is the string as written above is of length 3 (each set of \uxxxx is treated as one character) and what I was creating was a string of length 18. So how do I fix this? I ended up doing something I never thought I would, casting an int as char!

The way characters are stored is they are stored as their int representations and hence, that is what would make it work. So my code now looks like:

String testStr = "呀数达";

String[] splitStr = testStr.split("&#");

StringBuilder bldr = new StringBuilder();

char chr;

int i;

for (String str : splitStr)

{

str = str.replace(";","");

i = Integer.valueOf(str);

chr = (char) i;

bldr.append(chr);

}

Of JNDI, DBCP and Oracle

In continuation with the earlier post, I was pretty close to hitting my head against a big, indifferent wall as I struggled with JNDI connections and Oracle Prepared Statement to dump in Chinese characters into a database column.

So here's the step by step account:

• Make sure your database column type is NVARCHAR2. This is just like VARCHAR2 but allows you to insert pretty much everything into it.
• Query up your national character set: select value from v$nls_parameters where PARAMETER = 'NLS_NCHAR_CHARACTERSET'. This should return AL16UTF16
• Now the first option is to use the unistr function. So my statement would look something like insert into table_name (col1, col2, col3...,coln) values (?,?,?,...unistr(?)).Now use a standard setString API call, but make sure that the input string is of the form \[four digit hex code] for each character.
• Second option is to call the setFormOfUse API on the OraclePreparedStatement class as ps.setFormOfUse([index], OraclePreparedStatement.FORM_NCHAR) and set the string in a standard fashion. However, the string here needs to be of the form \u[four digit hex code] for each character.

However, on the last step I kept getting a class cast exception whenever I tried getting the OraclePreparedStatement object from the standard PreparedStatement. This is what you will expect to work normally:

DelegatingPreparedStatement dps = (DelegatingPreparedStatement) ps;

OraclepreparedStatement ops = (OraclePreparedStatement) dps.getInnermostDelegate();

ops.setFormOfUse(...);

But inspite of the runtime class of the object returned by the getInnerMostDelegate() method is OraclePreparedStatement, it kept giving me a class cast exception (I still don't know why!). However, a reflective call like the one below worked!

DelegatingPreparedStatement dps = (DelegatingPreparedStatement) ps;

Statement stmt = ps.getInnermostDelegate();

Method setUsageMethod = stmt.getMethod("setFormOfUse", int.class, short.class);

setUsageMethod.invoke(stmt, [args]);

Now I need to revisit the search implementation on the nvarchar column and then post this to other databases. Hmmph!

Unicode and HTML

I have been struggling with developing a sample application in Chinese for the past few days. Its a pretty simple app, add, search and edit employees. Well, not quite!

After building the sample pages, a search and list page and an add / edit page, I tried converting them into Chinese. Here's what I learnt till now:

1. Don't forget to set pageEncoding tags in jsps. Had to set this to UTF-8 in conjunction with the meta charset tag.
2. JavaScript understands Unicode strings in hex as \u, HTML needs it as &#;. So if I use hex strings in tags, it works, but same in an AJAX response makes the browser ignore it, they will show up as \u. Need to convert them into decimals.
3. understands HTML encoded Unicode too (the decimal strings).
4. IE has troubles understanding charset of a a HTTP post request. In spite of setting charset and encodings, it continues to treat it as windows specific or ISO-8859-1. The hack? Add a hidden variable in your form thats denotes a character not part of the standard encoding (any Cyrillic value will do!) and IE starts responding. Bizarre!
5. Data posted by the form will most always be HTML encoded. Even if it is stored in DB as such the browser is able to pick it up correctly.
6. For resource bundles, use the java native2ascii command. Store your files with native characters first, then run this tool as "native2ascii -encoding UTF-16 [input file] [output file]" and you're done!
7. One form refused to post data in HTML encoded form. The hack? Forcefully converted it such (code below).
8. Yet to fix the NCHAR / NVARCHAR storage :(

So how do you convert hex to decimal and back?

• JavaScript: Remember that decimals get prefixed with &# and suffixed with semicolon. The hex prefixed with \u

1. Try the charCodeAt() built in function. Take a string, iterate over it and call this function for each char.
2. Hex to decimal using [str].parseInt(16)
3. Decimal to hex using parseInt([str],16)

• Java: Similar conversions, Integer.parseInt(str, 16) and Integer.toHexString(int)

Now need to figure out the nchar, nvarchar storage and Im done!