Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string.

[0]: https://simonsapin.github.io/wtf-8/



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: