Umfrage/Architekturfrage zur DEC

[Kas Ob.]

Well, first sorry for bad English and for that i will be (very) verbose trying to explain my opinion and my findings on this.

I looked at the code and browsed the concerning files, and now i see the problem(s) and shortcoming in utilizing streams.

So to confirm my doubts i wrote my own test to amplify and pin point the real cause, and my doubts was on the point:
the test project with cherry picked test vectors from DEC own test vectors

Delphi-Quellcode:

program AesGCMStream;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  Generics.Collections,
  classes,
  System.Math,
  DECBaseClass,
  DECCipherBase,
  DECCipherModes,
  DECCipherFormats,
  DECFormat,
  DECCiphers;

const
 { TV_Key = '11754cd72aec309bf52f7687212e8957';
  TV_IV = '3c819d9a9bed087615030b65';
  TV_PT = '';
  TV_AAD = '';
  TV_CT = '';
  TV_Tag = '250327c674aaf477aef2675748cf6971';   }

  {TV_Key = 'fe0121f42e599f88ff02a985403e19bb';
  TV_IV = '3bb9eb7724cbe1943d43de21';
  TV_PT = 'fd331ca8646091c29f21e5f0a1';
  TV_AAD = '2662d895035b6519f3510eae0faa3900ad23cfdf';
  TV_CT = '59fe29b07b0de8d869efbbd9b4';
  TV_Tag = 'd24c3e9c1c73c0af1097e26061c857de';  }

  TV_Key = 'a976d8ae4528bff2911094a421f4ed83';
  TV_IV = '6904cb03535dce4b080148df37e81a5c9a9bebfa0037658228456a8a1a3db8e5c0be70c082bfbf50f43ab59cdbf24312ad6dd041e'+
  '1d179df5d36f9fa7b33acb2983f8d66443b44c1478253ee2cdea9840a6698e9be130de46947121af9223d411cc4f7c715da83a30dcd4c54f5b52701ee315b52c44b5d58cb6caaae946ad940';
  TV_PT = 'f235d5c4f71b40d2afe4b85a6d62a5bb426e3606f086bee79f6788d3b7ce82a6';
  TV_AAD = '348ae1548058be59efd0f830ca3b9b0805320632';
  TV_CT = '837af64be0d0bcfb9688e7043f5594e5c483cfc06a5e809fbe091cd702bf419c';
  TV_Tag = '71e22c2b15d4d2caeba54036';

 { TV_Key = '88c81827cb514632c8b0c76b7ecbd1cc';
  TV_IV = 'b3632bb439c8811f1454e6a368c4c9d0bbd3d9507ed1050cac3f19ba085063af9d162eb1c02077a51bad143be939d32c68'+
  '5b6fb3f330b8b382cc6567c55f2f4ecfeff88ff281e1e5ee1cfc813a13c9e69096761f58d13b9cad6221b5aaee03e40ad56f1a61c250ef57f94985ab6a603ded02b513e035ac8b2e3c3b69d35d2918';
  TV_PT = 'e254bf464879b4c48200541d359ecce478c67a62f4f5aaaa047d8e4a4ad6adff19da9a535a0be0758d5e7e992ccbb936d3c496';
  TV_AAD = '';
  TV_CT = '453ca80e69d37a6c8338da0deabb5ed1d5f8c006e67aca5d0bfbcd3aa32290521e91f379b7db57764c2755bf8691451e72a295';
  TV_Tag = '01485fb4f9675740b354bf7557f0f23a';
   }



procedure DoTest(Step:Integer);
var
  CipherAES: TCipher_AES;
  Key, IV, PT, AAD, CT, Tag, OutPut: TBytes;
  ctbStream, ptbStream: TBytesStream;
  dataLeftToEncode: Integer;
begin
  Writeln('Feeding Step = ',Step);
  CipherAES := TCipher_AES.Create;
  try
    CipherAES.Mode := TCipherMode.cmGCM;

    Key := TFormat_HEX.Decode(BytesOf(TV_Key));
    IV := TFormat_HEX.Decode(BytesOf(TV_IV));
    PT := TFormat_HEX.Decode(BytesOf(TV_PT));
    AAD := TFormat_HEX.Decode(BytesOf(TV_AAD));
    CT := TFormat_HEX.Decode(BytesOf(TV_CT));
    Tag := TFormat_HEX.Decode(BytesOf(TV_Tag));

    CipherAES.Init(Key, IV);
    CipherAES.AuthenticationResultBitLength := Length(Tag) * 8;
    CipherAES.DataToAuthenticate := AAD;

    ptbStream := TBytesStream.Create(PT);
    ctbStream := TBytesStream.Create;

    dataLeftToEncode := ptbStream.Size;
    repeat
      CipherAES.EncodeStream(ptbStream, ctbStream, Min(STEP, ptbStream.Size - ptbStream.Position));
      ctbStream.SetSize(ctbStream.Position);
      Writeln('Step : ' + StringOf(TFormat_HEX.Encode(ctbStream.Bytes)));
      Dec(dataLeftToEncode,STEP);
    until dataLeftToEncode <= 0;

    CipherAES.Done;

    ctbStream.SetSize(ctbStream.Position);
    OutPut := ctbStream.Bytes;

    Writeln('Expected : ' + StringOf(TFormat_HEX.Encode(CT)));
    Write('Encrypted: ' + StringOf(TFormat_HEX.Encode(OutPut)));
    if CompareMem(CT,OutPut,Length(ct)) then
      Writeln(#9'OK') else
      Writeln(#9'FAIL');

    Writeln('Tag expected : ' + StringOf(TFormat_HEX.Encode(Tag)));
    Write('Tag encrypted: ' + StringOf(TFormat_HEX.Encode(CipherAES.CalculatedAuthenticationResult)));
    if CompareMem(Tag,CipherAES.CalculatedAuthenticationResult,Length(Tag)) then
      Writeln(#9'OK') else
      Writeln(#9'FAIL');

  finally
    CipherAES.Free;
  end;
  Writeln;
end;

begin
  try
    DoTest(100);
    DoTest(1);
    DoTest(5);
    DoTest(16);
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
  Writeln('Done.');
  Readln;
end.

The output for the above test vector is

Delphi-Quellcode:

Feeding Step = 100
Step : 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C
Expected : 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C
Encrypted: 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C     OK
Tag expected : 71E22C2B15D4D2CAEBA54036
Tag encrypted: 71E22C2B15D4D2CAEBA54036 OK

Feeding Step = 1
Step : 83
Step : 83B3
Step : 83B337
Step : 83B33726
Step : 83B3372677
Step : 83B3372677D9
Step : 83B3372677D9C7
Step : 83B3372677D9C7B1
Step : 83B3372677D9C7B17B
Step : 83B3372677D9C7B17B91
Step : 83B3372677D9C7B17B91D3
Step : 83B3372677D9C7B17B91D3FD
Step : 83B3372677D9C7B17B91D3FD0A
Step : 83B3372677D9C7B17B91D3FD0AE5
Step : 83B3372677D9C7B17B91D3FD0AE5CE
Step : 83B3372677D9C7B17B91D3FD0AE5CE7C
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE6013
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE601333
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BC
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F1
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B22
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B2236
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B2236AC
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B2236ACFA
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B2236ACFA90
Step : 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B2236ACFA9047
Expected : 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C
Encrypted: 83B3372677D9C7B17B91D3FD0AE5CE7CBE60133390BCC4D9F15B2236ACFA9047     FAIL
Tag expected : 71E22C2B15D4D2CAEBA54036
Tag encrypted: 92D5435B6AB1ECF5B3AC7B15 FAIL

Feeding Step = 5
Step : 837AF64BE0
Step : 837AF64BE09DAD2B697E
Step : 837AF64BE09DAD2B697E5AF67EBD71
Step : 837AF64BE09DAD2B697E5AF67EBD7159A08E74A9
Step : 837AF64BE09DAD2B697E5AF67EBD7159A08E74A970706309C1
Step : 837AF64BE09DAD2B697E5AF67EBD7159A08E74A970706309C1A5D6D2E10B
Step : 837AF64BE09DAD2B697E5AF67EBD7159A08E74A970706309C1A5D6D2E10B0571
Expected : 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C
Encrypted: 837AF64BE09DAD2B697E5AF67EBD7159A08E74A970706309C1A5D6D2E10B0571     FAIL
Tag expected : 71E22C2B15D4D2CAEBA54036
Tag encrypted: F4904F062055FD0124C1D3A3 FAIL

Feeding Step = 16
Step : 837AF64BE0D0BCFB9688E7043F5594E5
Step : 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C
Expected : 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C
Encrypted: 837AF64BE0D0BCFB9688E7043F5594E5C483CFC06A5E809FBE091CD702BF419C     OK
Tag expected : 71E22C2B15D4D2CAEBA54036
Tag encrypted: A7CF85F2D18FD9BC086E2663 FAIL

Done.

This show exactly what went wrong and where, like for 5 bytes the first 5 of the cipher is correct the rest is wrong, and for step=16 the encryption is correct but the tag is off.

First lets apply the AES with GCM correctly and for that i will refer to https://en.wikipedia.org/wiki/Galois...asic_operation
GCM-Galois_Counter_Mode_with_IV.svg.png
This visual representation explain AES in GCM mode, E_k is encrypted block and mult_H is the Galois field multiplication, now both must be 128 bit (16 bytes), this is important, notice
1) E_k is irrelevant to the input (text/Plaintext), and we get the cipher (Ciphertext) by XORing the E_k with Plaintext
2) mult_H is chained on the Ciphertext.

These are the most important details concern us here, now to explain what is going wrong i will take example same as my tests above, lets say we have text length of 20 bytes and want encrypted it in chunks, same as performing using a stream.
assuming we passed 5 bytes then E_k is correct and will be the same no matter how much bytes we feed, as it generate 16 bytes, then we XOR 5 bytes and get the Ciphertext for 5 bytes, this is shown in the output of my example where first 5 bytes is correct, on the second step, i mean another 5 bytes comes the problem and the whole thing collapse,
why ?
Because we should saved E_k that was 16 bytes and used only 5 from it, we should kept the rest and XOR the new input (second 5 bytes) with the rest, and only generate new E_k when we used all the 16 bytes of the last one.
As for the tag correctness, see in my test above the last case where the steps are at 16 bytes, meaning and showing evidently the correctness of the encryption, but the tag is wrong,
why ?
Because similar to above the multiplicaiton must be performed on 16 bytes, not any less, so full block on Ciphertext, if there is no data then a padding should present and only then the multiplication should be performed.

Thoughts here:
1) a refactor and re-structure is due, DoEncodeDecodeStream is wrong and depending on TGCM.EncodeGCM is also short and miss the accumulation.
2) Most important thing is the use of "Done;" this should be the trigger for finalization same or similar to hash functions, don't add any padding unless "Done" being called, otherwise save E_k and incomplete Ciphertext until they are multiple of 16 bytes then drop E_k and calculate the multiplication, for me accessing the tag if there is residue or accumulation should trigger an exception, this is best practice to prevent users from wrongly using encryption and enforcing the correct values, see here if the encryption is not ended (called done) this mean the encrption is correct abut the tag is wrong and only can be accessed after "Done"
3) I think if you managed to fix the code to make my test above pass then it will work for any thing.
4) 8k bytes can't be excessive nor small, should be available to the user to pick or adjust, can be useful for speed, or when want to encrypt big files, i see no problem with it, the problem is in (2)
5) in (2) either save E_k in full or just the rest, as shown in AESGCM representation next chunk should be xored at the right index, and this will fix the tests above.
6) Again voting to raise exception on accessing CalculatedAuthenticationResult if Done is not called.

Hope it was clear and helpful.