Hands on: Foreign characters

A particular problem affecting websites is letters that have accents and diacritical marks in assorted languages.

One site I work with has a member directory for a professional organisation. The version on the site shouldn’t really go by a name as grand as a database; it’s essentially a text file, in CSV format, that has been exported from a database in someone’s office and uploaded to the website.

A form on the site allows people to specify what they’re searching for - name, city and so on - and then a Perl script reads through the text file, spitting out an HTML page header, followed by nicely formatted information for the matching entries.

It does the job, but as there are more and more international members, there’s a need to make sure that their names are not only presented correctly on the web page but can be found by searching too.

The first issue is solved by a search and replace each month, swapping single characters in Word for the matching HTML equivalents. For example, ö is changed to ö, and where someone searches for a result that includes that character in, say, the address, then it’s displayed OK.

But how do they search for a name with a foreign character? That needs the script changing to look for the appropriate entity. In PHP that’s a trivial thing to do, with specialised functions that work in both directions, and in Perl a little magic with regular expressions should do the trick.

There’s another problem, though. A lot of words become Anglicised, and with a city such as Paris it makes no difference to the spelling. With others it does - Zurich and Zürich.

You probably need to be consistent when you enter information; either always Anglicise foreign town names, or never. In this case, that means persuading the keepers of the database to change lots of information, which isn’t always possible.