Reading ASCII file with some codes above 127 (exten ascii)

[Paul]

On Wednesday, 23 May 2012 at 19:01:53 UTC, Graham Fawcett wrote:
> On Wednesday, 23 May 2012 at 18:43:04 UTC, Paul wrote:
>> On Wednesday, 23 May 2012 at 18:04:56 UTC, Graham Fawcett 
>> wrote:
>>> On Wednesday, 23 May 2012 at 15:48:20 UTC, Paul wrote:
>>>> On Monday, 14 May 2012 at 12:58:20 UTC, Graham Fawcett wrote:
>>>>> On Sunday, 13 May 2012 at 21:03:45 UTC, Paul wrote:
>>>>>> I am reading a file that has a few extended ASCII codes 
>>>>>> (e.g. degree symdol). Depending on how I read the file in 
>>>>>> and what I do with it the error shows up at different 
>>>>>> points.  I'm pretty sure it all boils down to the these 
>>>>>> extended ascii codes.
>>>>>>
>>>>>> Can I just tell dmd that I'm reading a Latin1 or ISO 
>>>>>> 8859-1 file?
>>>>>> I've messed with the std.encoding module but really can't 
>>>>>> figure out what I need to do.
>>>>>>
>>>>>> There must be a simple solution to this.
>>>>>
>>>>> This seems to work:
>>>>>
>>>>>
>>>>> import std.stdio, std.file, std.encoding;
>>>>>
>>>>> void main()
>>>>> {
>>>>> auto latin = cast(Latin1String) read("/tmp/hi.8859");
>>>>> string s;
>>>>> transcode(latin, s);
>>>>> writeln(s);
>>>>> }
>>>>>
>>>>>
>>>>> Graham
>>>>
>>>> I thought I was in good shape with your above suggestion.  I 
>>>> does help me read and process text.  But when I go to print 
>>>> it out I have problems.
>>>>
>>>> Here is my input file:
>>>> °F
>>>>
>>>> Here is my code:
>>>> import std.stdio;
>>>> import std.string;
>>>> import std.file;
>>>> import std.encoding;
>>>>
>>>> // Main function
>>>> void main(){
>>>>  auto fout = File("out.txt","w");
>>>>  auto latinS = cast(Latin1String) read("in.txt");
>>>>  string uniS;
>>>>  transcode(latinS, uniS);
>>>>  foreach(line; uniS.splitLines()){
>>>>     transcode(line, latinS);
>>>>     fout.writeln(line);
>>>>     fout.writeln(latinS);
>>>>  }
>>>> }
>>>>
>>>> Here is the output:
>>>> °F
>>>> [cast(immutable(Latin1Char))176, 
>>>> cast(immutable(Latin1Char))70]
>>>>
>>>> If I print the Unicode string I get an extra weird character.
>>>> If I print the Unicode string retranslated to Latin1, it get 
>>>> weird pseudo-code.
>>>> Can you help?
>>>
>>> I tried the program and it seemed to work for me.
>>>
>>> What program are you using to read "out.txt"? Are you sure it 
>>> supports UTF-8, and knows to open the file as UTF-8? (This 
>>> looks suspiciously like a tool's attempt to misinterpret a 
>>> UTF-8 string as Latin-1.)
>>>
>>> If you're on a Unix system, what does "file in.txt out.txt" 
>>> report?
>>>
>>> Graham
>>
>> Hmmm.  I'm not communicating well.
>> I want to read and write ASCII.  The only reason I'm 
>> converting to Unicode is because D needs it (as I understand).
>>
>> Yes if I open °F in notepad++ and tell notepad++ that it is 
>> UTF-8, it shows °F.
>>
>> I want to:
>> 1) Read an ascii file that may have codes above 127.
>> 2) Convert to unicode so D funcs like .splitLines() can work 
>> with it.
>> 3) Convert back to ascii so that stuff like °F writes out as 
>> it was read in.
>>
>> If I open in.txt and out.txt in an ascii editor, °F should 
>> look the same in both files with the editor encoding the files 
>> as ANSI/ASCII.  I thought my program was doing just that.
>> Thanks for your assistance.
>
> To make sure we're on the same page -- ASCII is a 7-bit 
> encoding, and any character above 127 is by definition not an 
> ASCII character. At that point we're talking about an encoding 
> other than ASCII, such as UTF-8 or Latin-1.
>
> If you're reading a file that has bytes > 127, you really have 
> no choice but to specify (assume?) an encoding, Latin-1 for 
> example. There's no guarantee your input file is Latin-1, 
> though, and garbage-in will result in garbage-out.
>
> So I think what you're trying to do is
>
> 1. read a Latin-1 file, into unicode (internally in D)
> 2. do splitLines(), etc., generating some result
> 3. Convert the result back to latin-1, and output it.
>
> Is that right?
> Graham

Exactly.