Suche nach String mit 'decomposed' Character

[jaenicke]

Zitat:

Zitat von himitsu (Beitrag 1541852)

Ich wüsste auch nicht, warum

Delphi-Quellcode:

Götterdämmerung

eine Dekompositation ergeben sollte, vor allem nicht, da ö und ä das Selbe wären.

Das stimmt nicht. Das ist quasi a und o gefolgt von dem Trema (Zweipunktakzent), in der Datei gespeichert als Big-Endian (0x0803 in UTF-16). Nur dieser Akzent ist identisch.

Ich sehe allerdings für eine solche Zweiteilung keinen Grund bei der Zeichenkodierung.

[Gausi]

Zitat:

Zitat von jaenicke (Beitrag 1541853)

Ich sehe allerdings für eine solche Zweiteilung keinen Grund bei der Zeichenkodierung.

Ist auch nicht mein Spezialgebiet, aber deswegen scheint es die Normalisierung (Unicode) zu geben. Der letzte Beitrag von himitsu dürfte auch in diese Richtung gehen, wenn ich das richtig sehe.

[himitsu]

Zitat:

Zitat von jaenicke (Beitrag 1541853)

Das ist quasi a und o gefolgt

Ich bin mir sicher, das sah gestern noch aus, wie zwei O. :shock:

Irgendwer hat da heimlich 3 Bit geändert. :shock:

Zitat:

Zitat von jaenicke (Beitrag 1541853)

Ich sehe allerdings für eine solche Zweiteilung keinen Grund bei der Zeichenkodierung.

De Wiki sagt es so schön.

Zitat:

Precomposed characters are the legacy solution for representing many special letters in various character sets. In Unicode, they are included primarily to aid computer systems with incomplete Unicode support, where equivalent decomposed characters may render incorrectly.

So einen ähnlichen Mist gibt es inzwischen auch für Smileys ... Geschlechtsteil und Rasse (Hautfarbe)
Wobei man es hier unvollständig lassen und nicht alles x-fach nochmal aufnehmen wollte.
http://www.unicode.org/L2/L2016/1618...-sequences.pdf

[Kas Ob.]

Hi,

Zitat:

Zitat von Maekkelrajter (Beitrag 1541835)

In diesem Thread (Artikel #20) hatte ich bereits das Problem mit unterschiedlich codierten Umlauten thematisiert. Zur Zeit brüte ich über einer Suchfunktion, die in einem TMemo bei einer Suche nach 'Götterdämmerung' sowohl 'Götterdämmerung' (precomposed = Götterdämmerung) als auch 'Götterdämmerung' (decomposed = Götterdämmerung) findet.
Das ist wohl alles andere als trivial,

To be honest it is trivial ! but with the right tools.

Which is completely different APIs like FindNLSString and FindNLSStringEx
https://learn.microsoft.com/en-us/wi...-findnlsstring
https://learn.microsoft.com/en-us/wi...indnlsstringex
in fact all the functions the following link capable to perform what you want, with more or less tweaking parameters
https://learn.microsoft.com/en-us/wi...port-functions

Windows has National Language Support (NLS) and International Components for Unicode (ICU), but there is differences, ICU introduced to Windows 10 Creator, while NLS was there since Windows Vista (at least i think)

Anyway here is a simple test for you search problem and its solution, i will paste the code and attach the project because i can't trust the browsers to keep encoding right.

Delphi-Quellcode:

unit Unit11;

interface

uses
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants,
  System.Classes, Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs,
  Vcl.StdCtrls;

type
  TForm11 = class(TForm)
    Button1: TButton;
    Memo1: TMemo;
    Memo2: TMemo;
    procedure FormCreate(Sender: TObject);
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    procedure SearchStringList(AList: TStringList);
    procedure SearchStringFindNLSString(AList: TStringList);
    procedure SearchStringFindNLSStringEx(const LOCALE_NAME: string; AList: TStringList);
  end;

var
  Form11: TForm11;

implementation

{$R *.dfm}

const
  SUB_STR_1 = 'Götterdämmerung';
  SUB_STR_2 = 'Götterdämmerung';

procedure TForm11.FormCreate(Sender: TObject);
begin
  Button1.Click;
end;

procedure TForm11.SearchStringFindNLSString(AList: TStringList);

  procedure DoFindWithFindNLSString(const SubStr: string);
  var
    i, Res, Found: Integer;
  begin
    for i := 0 to AList.Count - 1 do
    begin                 // LOCALE_USER_DEFAULT = $400
      Res := FindNLSString(LOCALE_USER_DEFAULT, FIND_FROMSTART, PChar(AList.Strings[i]), -1, PChar(SubStr), -1, @Found);
      if (Res <> -1) and (Found > 0) then
        Memo1.Lines.Add(IntToStr(i));
    end;
  end;

begin
  Memo1.Lines.Add('finding lines with SUB_STR_1 = ' + SUB_STR_1);
  DoFindWithFindNLSString(SUB_STR_1);

  Memo1.Lines.Add('finding lines with SUB_STR_2 = ' + SUB_STR_2);
  DoFindWithFindNLSString(SUB_STR_2);
end;

procedure TForm11.SearchStringFindNLSStringEx(const LOCALE_NAME: string; AList: TStringList);

  procedure DoFindWithFindNLSStringEx(const SubStr: string);
  var
    i, Res, Found: Integer;
  begin
    for i := 0 to AList.Count - 1 do
    begin
      Res := FindNLSStringEx(PChar(LOCALE_NAME), FIND_FROMSTART, PChar(AList.Strings[i]), -1, PChar(SubStr), -1, @Found, nil, nil, 0);
      if (Res <> -1) and (Found > 0) then
        Memo1.Lines.Add(IntToStr(i));
    end;
  end;

var
  i, Res, Found: Integer;
begin
  Memo1.Lines.Add('finding lines with SUB_STR_1 = ' + SUB_STR_1);
  DoFindWithFindNLSStringEx(SUB_STR_1);

  Memo1.Lines.Add('finding lines with SUB_STR_2 = ' + SUB_STR_2);
  DoFindWithFindNLSStringEx(SUB_STR_2);
end;

procedure TForm11.SearchStringList(AList: TStringList);

  procedure DoFindWithPos(const SubStr: string);
  var
    i: Integer;
  begin
    for i := 0 to AList.Count - 1 do
      if Pos(SubStr, AList.Strings[i]) > 0 then
        Memo1.Lines.Add(IntToStr(i));
  end;

begin
  Memo1.Lines.Add('finding lines with SUB_STR_1 = ' + SUB_STR_1);
  DoFindWithPos(SUB_STR_1);

  Memo1.Lines.Add('finding lines with SUB_STR_2 = ' + SUB_STR_2);
  DoFindWithPos(SUB_STR_2);
end;

procedure TForm11.Button1Click(Sender: TObject);
var
  sList: TStringList;
  i: Integer;
begin
  sList := TStringList.Create;
  try
    sList.LoadFromFile('MacOS_ItunesContent_Small.txt');
    if sList.Count = 0 then
      Exit;
    for i := 0 to sList.Count - 1 do
      Memo2.Lines.Add(IntToStr(i) + #9 + sList.Strings[i]);

    Memo1.Lines.Add('Searching using Pos');
    SearchStringList(sList);
    Memo1.Lines.Add(#13#10'Searching with FindNLSString');
    SearchStringFindNLSString(sList);
    Memo1.Lines.Add(#13#10'Searching with FindNLSStringEx and LOCALNAME=''''');
    SearchStringFindNLSStringEx('', sList);

  finally
    sList.Free;
  end;
end;

end.

The data file in question, which is minimized and edited a little is MacOS_ItunesContent_Small.txt

Zitat:

dummy line

Götterdämmerung
Götterdämmerung
Track 4344
Track ID: 11572
Titel: Morgendämmerung und Siegfried's Rheinfahrt aus "Götterdämmerung"
Artist: Richard Wagner
Album: Rheingold
Track-Art: Abgeglichene AAC-Audiodatei
GUID:
Hinzugefügt: 14.12.2017 17:57:46
Persistent ID: 91453A3B4084EC74
Tracktype: File
Speicherort: Macintosh HD/Users/AlfonsYondraschek/Music/iTunes 1/iTunes Media/Music/Richard Wagner/Rheingold/05 Morgendämmerung und Siegfried's Rheinfahrt aus _Götterdämmerung_.m4a

Track 4345
Track ID: 11574
Titel: Siegfried's Trauermarsch und Finale aus "Götterdämmerung"
Artist: Richard Wagner
Album: Rheingold
Track-Art: Abgeglichene AAC-Audiodatei
GUID:
Hinzugefügt: 14.12.2017 17:57:46
Persistent ID: 6C45FF4271B8A57B
Tracktype: File
Speicherort: Macintosh HD/Users/AlfonsYondraschek/Music/iTunes 1/iTunes Media/Music/Richard Wagner/Rheingold/06 Siegfried's Trauermarsch und Finale aus _Götterdämmerung_.m4a

The result should be like this

Zitat:

Searching using Pos
finding lines with SUB_STR_1 = Götterdämmerung
2
6
18
finding lines with SUB_STR_2 = Götterdämmerung
3
14
26

Searching with FindNLSString
finding lines with SUB_STR_1 = Götterdämmerung
2
3
6
14
18
26
finding lines with SUB_STR_2 = Götterdämmerung
2
3
6
14
18
26

And here screenshot of the result for any cases
Anhang 57176

The project with the data file
Anhang 57178

Notes on FindNLSString and FindNLSStringEx:
1) Although the documentation of FindNLSString advice to move to FindNLSStringEx, yet Notpad.exe is using FindNLSString !
2) FindNLSStringEx use LOCALE_NAME (plain string) instead of the structured LOCALE_NAME value, yet it is go complicated very fast when you need to chain many languages, so the sticking to default (USER or SYSTEM) is easier, in that case it is better and easier to use FindNLSString.
3) implementing similar algorithm in pure Pascal is huge job unless you will depend on either NLS or the ICU library, well .. such dependency will render any implementation useless and waste of time.

Hope that help

[Kas Ob.]

After smoking a cigarette here few more notes
4) there is a nice yet a little complicate documentation for LOCALE_NAME on Windows OS
https://learn.microsoft.com/en-us/wi...l/locale-names
if you want to use multiple LOCALE then you only can do with sort so value can be

Zitat:

<language>
<language>-<Script>
<language>-<Script>-<REGION>
<language>-<REGION>\_<sort order>
when needed there it can be combined with
<language>-<REGION>-x-<custom> or <language>-<Script>-<REGION>-x-<custom>

The IETF BCP 47 mentioned there is complete standardized and future proof.
https://www.rfc-editor.org/info/bcp47

5) Notice that : I tried to stay away from the lengths of the parameters i passed to FindNLSStringX, because it can be tricky and easily can go wrong and cause overflow, generating wide range of problems from AV in the OS API to simple corrupted data, the reason is these functions require the length in chars, and here is the problem with this terminology, what is char ?
Is it render-able one or code unit which 2 bytes for Delphi by default and Windows also by default has 2 bytes WideChar, the troubles comes from using or mixing different API such as WideCharToMultiByte this one in particular can be very dangerous due its ability to handle almost everything, and by MultiByte you should never assume it is 2 bytes output or even input for that API.
The moral of this is to do as i do, prefer to stay away as far as you can from calculating the length or passing it, and try to stay on the safe side by leaning on using the null-terminating strings while passing (-1) as length, the API is safer that way.

Again hope that helps somebody !

[Maekkelrajter]

Hi,
Wenn man weiß wie's geht, ist es tatsächlich trivial!
FindNLSString ist in der Tat die wirklich einfache Lösung.
Vielen Dank an alle Tippgeber, nicht zuletzt an Himitsu, aber vor allem Kas Ob., der mir mit seinem ausführlichen Beitrag entscheidend auf die Sprünge geholfen hat!

Once you know how to do it, it's actually trivial!
FindNLSString is indeed the really simple solution.
Many thanks to everyone who contributed good tips, not least to Himitsu, but especially to Kas Ob., who helped me significantly with his detailed contribution!

Gruß LP