JSON and UTF8 problem

[Heinz57]

Hello,

I would like to use JSON via http to exchange data with my main application. That data is, of course, in utf8.

To test this, I used your second (larger) JSON example from the manual and simply added some Umlaut charaters to one of the names in the sample data.

The contents of MyJSON label were correctly shown in the WatchList with the utf8 chars but when data is then transferred into some variable the utf8 chars are repesented only by ���� chars.

I spend about 2 hrs reading all posts about JSON and utf8 and thoroughly searched your docs but there was no solution.

I also added WLN_ENCODING=UTF8 to the top of my test script but no improvement.

What will solve my problem?

Best regards

[hagchr]

Hi, These encoding problems can be quite messy. Not sure what the best solution is, let's see if there are other comments.

I tried two versions (in both cases using: WLN_ENCODING=UTF8)

Use WriteLn> and
- write text to new file
- write text to existing file (empty txt file created in Notepad and saved as UTF8)
then read from the file and process the data.

I did not get the first one to work but the second one seems to work fine. So what I did was to create an empty template in Notepad (saved as UTF8) which can then be copied each time the program is run. Simply write the text to the file and read it back to a variable and process...
(You need to update links for fileTemplate and fileNew.)

Hope it helps (until better solution)

Code: Select all

/*
MyJSON:
{ "uid" : "1234", 
  "clients" : ["client1","client2","client3"],
  "people" : [{"Name":"König","Age":"21"},{"Name":"Dörthe","Age":"18"}],
  "color" : "red",
  "size" : 14 }
*/


// Outside MS, Create an empty txt file in Notepad and save as UTF8. Link to variable
Let>fileTemplate=C:\Users\Christer\Desktop\tmpTemplate.txt

// Copy template file to new file (delete any prior copies).
Let>fileNew=C:\Users\Christer\Desktop\CHtmpForumQ.txt
IfFileExists>fileNew
  DeleteFile>fileNew
EndIf
CopyFile>fileTemplate,fileNew


LabelToVar>MyJSON,sJSON

// Write text to file using UTF8 and read back to variable sJSON
Let>WLN_ENCODING=UTF8
WriteLn>fileNew,nWLNRes,sJSON
ReadFile>fileNew,sJSON


//get the UID
JSONParse>sJSON,$.uid,result
MessageModal>UID is: %result_1%
 
//gets the first client
JSONParse>sJSON,$.clients[0],result
MessageModal>client 1 is %result_1%
 
//gets the second name (base zero)
JSONParse>sJSON,$.people[1].Name,result
MessageModal>second name is %result_1%
 
//this will get an array of all the names
JSONParse>sJSON,$.people[*].Name,result
 
//let's loop through the output 
Let>k=0
Repeat>k
  Let>k=k+1
  Let>this_name=result_%k%
  MessageModal>this_name
Until>k=result_count
 
//use the [' ... '] format instead of . - this is useful if names have dots or special chars in them
JSONParse>sJSON,$['people'][1]['Name'],result

[Heinz57]

Dear hagchr,

thank you very much for trying to help me and for putting that much efforts and time into this problem. Unfortunately, this does not really solve my problem as I must use JSON data that is received via http (as I wrote above).

I had read a post by Marcus Tettmar saying that MS is supporting Unicode, of which ASFAIK utf8 is insofar a subset as it uses only a part of the Unicode charater range. That all Uniode statement seems to be incorrect.

I currently see these possible solutions (and this is mainly to perhaps help other readers):

1) I will first try to use standard http as I have a webserver in my main application anyway. But I must find out if MS supports utf8 in http at all.

2) I could finally exchange data via the clipboard using the UI of my main application but that limits all to those first and few cases where both are on the same computer.

3) For the first use cases I could use data files as an "interface", because this data exchange happens on one
computer. But, well, that is really Flinstone style!

4) Another solution could be the use the Python interface but I have no clue if that would work and I also want to avoid the hassle with another language and the primitive Python IDE as I am on far better standards with my main IDE and its language.

Very frustrating!

I will definetly try to avoid writing some decoder in MS Basic similar to the ancient days of ASCII limited html.

Best regards
Heinz57

[Heinz57]

Hello again,

well, I tried your example and it seems to work. Well, it DOES work in the IDE. Thank you once more.

"Seems" only because I have not yet been able to test this with JSON data loaded via http.

It is beyond my comprehension why this hack via a file is needed but I am not a compiler-based "bit level" developer. I have been working for over 35 years in a much higher and far more sophisticated pure o-o language and IDE where we never declare variable types or care about data types, typcasts and such low-level stuff (stoneage BS in my view).

I will come back to report when I have the result from http tranfer.

Thank you again and a hint in the manual would be very helpful for others. The MacroScheduler manuals are often extremely sparse.

Regards
Heinz57

[Grovkillen]

viewtopic.php?f=8&t=10911&p=46602&hilit=escape#p46602