Geek asks and answers

Is there a way to fix invalid encoded strings?

I get this line through a message broker (Stomp):

João

and how he supposed to be:

João

Is there any way to get this back in Java ?! Thank!

+3
source share
2 answers
U+00C3  Ã   c3 83   LATIN CAPITAL LETTER A WITH TILDE
U+00C2  Â   c3 82   LATIN CAPITAL LETTER A WITH CIRCUMFLEX
U+00A3  £   c2 a3   POUND SIGN
U+00E3  ã   c3 a3   LATIN SMALL LETTER A WITH TILDE

I am having trouble figuring out how this could be a data conversion (coding) problem. Is it possible that the data is just bad?

If the data is not bad, then we must assume that you are interpreting the encoding incorrectly. We do not know the original encoding, and if you do not do something else, the default encoding for Java is UTF-16. I do not see how João, encoded in any general encoding, can be interpreted as Joãoin UTF-16

, python script . , , , FWIW.

#!/usr/bin/env python                                                                                                                   
# -- coding: utf-8 --                                                                                                                   
import pkgutil
import encodings

good = u'João'
bad = u'João'

false_positives = set(["aliases"])

found = set(name for imp, name, ispkg in pkgutil.iter_modules(encodings.__path__) if not ispkg)
found.difference_update(false_positives)
print found


for x in found:
    for y in found:
        res = None
        try:
            res =  good.encode(x).decode(y)
            print res,x,y
        except:
            pass
        if not res is None:
            if res == bad:
                print "FOUND"
                exit(1)
+4

. - .

, , , HTTP , IE , 1, . ( , , IE), , . , , .

, , ( Ã " " ). , , "latin1" , (, , ).

package peter.test;

import java.io.UnsupportedEncodingException;

/**
* User: peter
* Date: 2012-04-12
* Time: 11:02 AM
*/
public class TestEncoding {
    public static void main(String args[]) throws UnsupportedEncodingException {
        //In some cases a hack works. But best is to prevent it from ever happening.
        String good = "João";
        String bad = "João";

        //this line demonstrates what the "broken" string should look like if it is reversible.
        String broken = breakString(good, bad);

        //here we show that it is fixable if broken like breakString() does it.
        fixString(good, broken);

        //this line attempts to fix the string, but it is not fixable unless broken in the same way as breakString()
        fixString(good, bad);
    }

    private static String fixString(String good, String bad) throws UnsupportedEncodingException {
        byte[] bytes = bad.getBytes("latin1"); //read the Java bytes as if they were latin1 (if this works, it should result in the same number of bytes as java characters; if using UTF8, it would be more bytes)
        String fixed = new String(bytes, "UTF8"); //take the raw bytes, and try to convert them to a string as if they were UTF8

        System.out.println("Good: " + good);
        System.out.println("Bad: " + bad);
        System.out.println("bytes1.length: " + bytes.length);
        System.out.println("fixed: " + fixed);
        System.out.println();

        return fixed;
    }

    private static String breakString(String good, String bad) throws UnsupportedEncodingException {
        byte[] bytes = good.getBytes("UTF8");
        String broken = new String(bytes, "latin1");

        System.out.println("Good: " + good);
        System.out.println("Bad: " + bad);
        System.out.println("bytes1.length: " + bytes.length);
        System.out.println("broken: " + broken);
        System.out.println();

        return broken;
    }
}

( Sun jdk 1.7.0_03):

Good: João
Bad: João
bytes1.length: 5
broken: João

Good: João
Bad: João
bytes1.length: 5
fixed: João

Good: João
Bad: João
bytes1.length: 6
fixed: Jo £o
+2

More articles:


All Articles