Hallo zusammen,
Ich hab ein kleines Tool, für die Anzeige von .CSV Inhalten. Das hat sich jetzt an ein paar tausend Datensätzen (nicht genügend Speicher) verschluckt.
Ich bräuchte also eine möglichst (speicher)sparsame Anzeigekomponente oder müßte mir selbst etwas basteln. Was aber u.U. die Neuerfindung des Rades wäre.
Könnt Ihr mir da etwas empfehlen?

gruß
K-H

Hallo,
also das ganz normale TStringGrid sollte da keine Probleme haben bei "ein paar 1000 Einträgen".
Sicher, dass die Anzeige des Problem ist, und nicht die Komponente, die die CSV-Datei lädt?

Falls es doch klemmt, wäre dann noch das TDrawGrid.

Wie hast Du es denn bisher gelöst?

Unter groß kann man sich viel vorstellen. Eine CSV mit ca. 36000 Zeilen wird in ca. 300ms eingelesen und dann fix präsentiert.

(Kann man insgesamt auch schöner machen - aber für eine Demo reicht es ja)

zusammenfalten · markieren

Delphi-Quellcode:

unit CsvViewer.Classes;

interface

uses
  System.Classes, System.SysUtils, System.Generics.Collections,
  Data.DB,
  Datasnap.DBClient,
  Spring; // ACHTUNG! Spring4D wird benötigt!

type
  TCsvContent = class( TList < TArray < string >> )
  private
    FHeader: TArray<string>;
    FHasHeader: Boolean;
  public
    property Header: TArray<string> read FHeader write FHeader;
    property HasHeader: Boolean read FHasHeader write FHasHeader;
  end;

  TCsvReader = class
  public const
    DefaultBufferSize = 4096;
    DefaultDetectBOM = false;
    DefaultQuoteChar = Char( '"' );
    DefaultSeparatorChar = Char( ',' );
  private
    FQuoteChar: Char;
    FSeparatorChar: Char;
    FEncoding: TEncoding;
    FDetectBOM: Boolean;
    FBufferSize: Integer;
    FFirstRowContainsHeader: Boolean;
  public
    constructor Create;

    property BufferSize: Integer read FBufferSize write FBufferSize;
    property DetectBOM: Boolean read FDetectBOM write FDetectBOM;
    property Encoding: TEncoding read FEncoding write FEncoding;
    property FirstRowContainsHeader: Boolean read FFirstRowContainsHeader write FFirstRowContainsHeader;
    property QuoteChar: Char read FQuoteChar write FQuoteChar;
    property SeparatorChar: Char read FSeparatorChar write FSeparatorChar;

    function Read( const Filename: TFilename ): IShared<TCsvContent>;
  end;

implementation

{ TCsvReader }

constructor TCsvReader.Create;
begin
  inherited;

  FBufferSize := DefaultBufferSize;
  FDetectBOM := DefaultDetectBOM;
  FEncoding := TEncoding.UTF8;
  FQuoteChar := DefaultQuoteChar;
  FSeparatorChar := DefaultSeparatorChar;
end;

function TCsvReader.Read( const Filename: TFilename ): IShared<TCsvContent>;
var
  streamReader: IShared<TStreamReader>;
  line: string;
  stringList: IShared<TStringList>;
  firstRow: Boolean;
  I: Integer;
  Header: TArray<string>;

  function ReadNextLine( ): Boolean;
  var
    separatorCharCount: Integer;
  begin
    line := string.Empty;
    while not streamReader.EndOfStream do
      begin
        line := line + streamReader.ReadLine( );
        separatorCharCount := line.CountChar( QuoteChar );
        if separatorCharCount mod 2 = 0
        then
          Exit( True );
      end;
    Result := false;
  end;

begin
  Result := Shared.New( TCsvContent.Create( ) );

  streamReader := Shared.New( TStreamReader.Create( Filename, Encoding, DetectBOM, BufferSize ) );

  stringList := Shared.New( TStringList.Create );
  stringList.QuoteChar := QuoteChar;
  stringList.Delimiter := SeparatorChar;
  stringList.StrictDelimiter := True;

  firstRow := True;

  while ReadNextLine( ) do
    begin
      stringList.DelimitedText := line;

      if firstRow
      then
        try

          if FirstRowContainsHeader
          then
            begin
              Result.Header := stringList.ToStringArray( );
              Result.HasHeader := True;

              Continue;
            end;
        finally
          firstRow := False;
        end;

      Result.Add( stringList.ToStringArray( ) );
    end;
end;

end.

und die Form

zusammenfalten · markieren

Delphi-Quellcode:

unit CsvCiewer.Forms.MainForm;

interface

uses
  Winapi.Windows, Winapi.Messages,
  System.SysUtils, System.Variants, System.Classes, System.Actions,
  System.ImageList, System.Diagnostics,
  Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.ComCtrls,
  Vcl.ActnList, Vcl.StdActns, Vcl.ImgList, Vcl.ToolWin,
  Spring, CsvViewer.Classes;

type
  TMainForm = class( TForm )
    ListView1: TListView;
    ToolBar1: TToolBar;
    ToolButton1: TToolButton;
    ActionList1: TActionList;
    ImageList1: TImageList;
    FileOpen1: TFileOpen;
    StatusBar1: TStatusBar;
    procedure FileOpen1_Accept( Sender: TObject );
    procedure ListView1_Data( Sender: TObject; Item: TListItem );
  private
    FCsvReader: IShared<TCsvReader>;
    FCsvContent: IShared<TCsvContent>;
  public
    procedure AfterConstruction; override;
  end;

var
  MainForm: TMainForm;

implementation

{$R *.dfm}
{ TMainForm }

procedure TMainForm.AfterConstruction;
begin
  inherited;
  FCsvReader := Shared.New( TCsvReader.Create );
  FCsvReader.FirstRowContainsHeader := True;
end;

procedure TMainForm.FileOpen1_Accept( Sender: TObject );
var
  col: TListColumn;
  loadingStopwatch, presentingStopwatch: TStopwatch;
begin
  loadingStopwatch := TStopwatch.StartNew( );

  FCsvContent := FCsvReader.Read( FileOpen1.Dialog.FileName );

  loadingStopwatch.Stop( );

  presentingStopwatch := TStopwatch.StartNew();

  ListView1.Items.BeginUpdate( );
  try

    ListView1.Items.Count := 0;
    ListView1.Columns.Clear;

    while ListView1.Columns.Count < Length( FCsvContent.Items[0] ) do
      begin
        col := ListView1.Columns.Add;

        if FCsvContent.HasHeader
        then
          col.Caption := FCsvContent.Header[col.Index]
        else
          col.Caption := 'Field ' + ( col.Index + 1 ).ToString( );
      end;

    ListView1.Items.Count := FCsvContent.Count;
  finally
    ListView1.Items.EndUpdate( );
  end;

  presentingStopwatch.Stop();

  StatusBar1.SimpleText := loadingStopwatch.ElapsedMilliseconds.ToString() + 'ms - '+presentingStopwatch.ElapsedMilliseconds.ToString()+'ms';
end;

procedure TMainForm.ListView1_Data( Sender: TObject; Item: TListItem );
var
  rowData: TArray<string>;
  idx: Integer;
begin
  rowData := FCsvContent.Items[Item.Index];

  Item.Caption := rowData[0];

  for idx := 1 to High( rowData ) do
    begin
      Item.SubItems.Add( rowData[idx] );
    end;
end;

end.

und die dfm zur Form

zusammenfalten · markieren

Delphi-Quellcode:

object MainForm: TMainForm
  Left = 0
  Top = 0
  Caption = 'MainForm'
  ClientHeight = 411
  ClientWidth = 852
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'Tahoma'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object ListView1: TListView
    Left = 0
    Top = 29
    Width = 852
    Height = 363
    Align = alClient
    Columns = <>
    OwnerData = True
    ReadOnly = True
    RowSelect = True
    TabOrder = 0
    ViewStyle = vsReport
    OnData = ListView1_Data
    ExplicitTop = 35
    ExplicitHeight = 382
  end
  object ToolBar1: TToolBar
    Left = 0
    Top = 0
    Width = 852
    Height = 29
    Caption = 'ToolBar1'
    Images = ImageList1
    TabOrder = 1
    ExplicitLeft = 360
    ExplicitTop = 208
    ExplicitWidth = 150
    object ToolButton1: TToolButton
      Left = 0
      Top = 0
      Action = FileOpen1
    end
  end
  object StatusBar1: TStatusBar
    Left = 0
    Top = 392
    Width = 852
    Height = 19
    Panels = <>
    ExplicitLeft = 432
    ExplicitTop = 216
    ExplicitWidth = 0
  end
  object ActionList1: TActionList
    Images = ImageList1
    Left = 496
    Top = 200
    object FileOpen1: TFileOpen
      Category = 'File'
      Caption = #214'&ffnen...'
      Dialog.DefaultExt = '.csv'
      Dialog.Filter = 'CSV-Dateien (*.csv)|*.csv'
      Hint = #214'ffnen|Vorhandene Datei '#246'ffnen'
      ImageIndex = 0
      ShortCut = 16463
      OnAccept = FileOpen1_Accept
    end
  end
  object ImageList1: TImageList
    Left = 568
    Top = 208
    Bitmap = {
      494C010101000500040010001000FFFFFFFFFF00FFFFFFFFFFFFFFFF424D3600
      0000000000003600000028000000400000001000000001002000000000000010
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000008080000080
      8000008080000080800000808000008080000080800000808000008080000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000FFFF00000000000080
      8000008080000080800000808000008080000080800000808000008080000080
      8000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      00000000000000000000000000000000000000000000FFFFFF0000FFFF000000
      0000008080000080800000808000008080000080800000808000008080000080
      8000008080000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000FFFF00FFFFFF0000FF
      FF00000000000080800000808000008080000080800000808000008080000080
      8000008080000080800000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      00000000000000000000000000000000000000000000FFFFFF0000FFFF00FFFF
      FF0000FFFF000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000FFFF00FFFFFF0000FF
      FF00FFFFFF0000FFFF00FFFFFF0000FFFF00FFFFFF0000FFFF00000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      00000000000000000000000000000000000000000000FFFFFF0000FFFF00FFFF
      FF0000FFFF00FFFFFF0000FFFF00FFFFFF0000FFFF00FFFFFF00000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000FFFF00FFFFFF0000FF
      FF00000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      000000000000000000000000000000000000424D3E000000000000003E000000
      2800000040000000100000000100010000000000800000000000000000000000
      000000000000000000000000FFFFFF00FFFF000000000000FFFF000000000000
      001F000000000000000F00000000000000070000000000000003000000000000
      00010000000000000000000000000000001F000000000000001F000000000000
      001F0000000000008FF1000000000000FFF9000000000000FF75000000000000
      FF8F000000000000FFFF000000000000}
  end
end

@Schokohase
Danke für die Mühe aber da ich noch antik (D7) unterwegs bin, für mich wohl nicht umsetzbar.

Zitat von hoika:

Hallo,
also das ganz normale TStringGrid sollte da keine Probleme haben bei "ein paar 1000 Einträgen".

Naja 80.000+x , da war ein paar etwas untertrieben.

Zitat von hoika:

Sicher, dass die Anzeige des Problem ist, und nicht die Komponente, die die CSV-Datei lädt?

Autsch, in der Vergangenheit hatte ich zunächst einen TListview genutzt, und nachdem diesem die Luft ausgegangen war, ein TStringgrid. Darum bin ich davon ausgegangen, daß auch in diesem Falle die Anzeige der Schwachpunkt ist.
Da muß ich mich nochmal aufmachen und suchen.

Zitat von hoika:

Falls es doch klemmt, wäre dann noch das TDrawGrid.

TDrawGrid kenn ich nicht. Hab ich da bisher etwas Übersehen, oder ist das eine externe Komponente?

vielen Dank
K-H

TDrawGrid ist eine Standardkomponente.

@Redeemer
Danke für den Hinweis, jetzt hab ich sie auch gefunden.

Ich muß die Mengenangabe etwas Korrigieren es sind 890.000 Datensätze mit netto ca 90 MB Dateigröße.

Gruß
K-H

Wenn wirklich die Anzeige und nicht das Laden, Parsen etc. der Datei den Flaschenhals darstellt, könnte man mal den VST ins Auge fassen. Die Lernkurve ist zwar etwas steil und man muss auch vergleichsweise viel Code schreiben, dafür kann das Ding so ziemlich alles und ist pfeilschnell.

Unter Delphi 7 sollte die TListView auch den virtuellen Modus können (siehe TListView.OwnerData).

Alles andere ist eigentlich überflüssig bzw. lässt sich auch ganz einfach nach Delphi 7 übertragen.

Ich habe die vorliegende Datei einfach mal etwas dupliziert um so eine Datei mit ca. 890000 Zeilen und ca. 90MB zu erhalten. Braucht etwas mehr als 6 Sekunden zum Einlesen und 40 Millisekunden zum Anzeigen. Die Anwendung selber benötigt dabei ca. 540MB Speicher

Zitat von Schokohase:

Unter Delphi 7 sollte die TListView auch den virtuellen Modus können (siehe TListView.OwnerData).

Uff, dann muß ich mir das mal anschauen.

Bis jetzt hab ich versucht mir einen Überblick über TDrawGrid zu verschaffen. Schon interessant, daß ich bei den Jedis was dazu gefunden habe und in der D7-Hilfe und den Samples nichts konkretes zu finden ist.
Es hat etwas gedauert, bis ich verstanden habe, daß VirtualStringTree Teil von VirtualTreeView ist. Aber jetzt hab ich auch die passende Version gefunden.

Geschwindigkeit ist schön, aber ob es jetzt 2 oder 10 sec dauert bis ich eine Anzeige habe ist nicht so wichtig. Hauptsache ich habe alle Daten im Zugriff.

Erst einmal allen die mir hier geholfen haben vielen Dank
K-H