Extended File Transfer sample (Delphi)

Overview

This sample shows how the client can download files exposed by the server in chunks using multiple threads.
Notice that this sample is cross-platform compatible with .NET and Cocoa versions.

How it works

The approach is quite simple: Instead of downloading a complete file at once, it is split into small chunks to be downloaded separately one after another or simultaneously using multiple threads and connections. On the client side, those chunks are combined back into a single file.

This sample provides a RemObject server that exposes the content of a folder (see the GetFilesList method in the ExtendedFileTransferService implementation) and a client that can connect to the server to obtain a list of the files available for downloading. The server also provides another method for downloading a chunk of a file called DownloadFilePart. This method has three parameters:

• FileName (AnsiString) – Name of the file to be downloaded.
• PartNo (int32) – The chunk part that should be downloaded starting with 1.
• PartSize (int32) – Size of the chunk that should be downloaded.

DownloadFilePart reads the section of the file with the offset PartSize*PartNo and returns the data as a Binary stream to the client. When you want to download a file, you can specify the size of the chunk (below the Chunk Size combo box you will see the calculated total count of chunks). You can also specify the number of threads you want to use for the downloading operation. Since each partial chunk download is independent, using threads is quite attractive in this scenario.

Getting Started

• Compile and run the server and client applications.
• Prepare the server side:
  • Put one or more files into the shared folder (you can easily locate and open this folder when you click on the ellipsis button on the server main form). For the best experience, the file size should be several times larger than the chunk size.

• Prepare the client side:
  • Get the available files on the client side by clicking the GetFiles button.
  • Adjust the chunk size and thread count on the client.
• Try to download the selected file. At the end of the downloading process, you can open the file to check its integrity and ensure that downloading was successful.

Examine the Code

The server

• See the implementation of the GetFilesList method in the ExtendedFileTransferService_Impl class. Notice how the file information is packed into an array of TFileInfo structures, both the structure and the array type are defined in the RODL.

function TExtendedFileTransferService.GetFilesList: FileInfoArray;
var
  sr: TSearchRec;
  fi: TFileInfo;
begin
  Result := FileInfoArray.Create;
  if FindFirst(ServerForm.SharedFolder + '\*.*', faReadOnly + faArchive, sr) = 0 then begin
    repeat
      fi := Result.Add;
      fi.FileName := {$IFDEF UNICODE}WideStringToAnsiString{$ENDIF}(ExtractFileName(sr.Name));
      fi.TypeName := {$IFDEF UNICODE}WideStringToAnsiString{$ENDIF}(ExtractFileExt(sr.Name));
      fi.Size := sr.Size;
    until FindNext(sr) <> 0;
    FindClose(sr);
  end;
end;

• See how the file chunk is requested from the server via the DownloadFilePart method in the ExtendedFileTransferService_Impl class. See how you can read a chunk of the file, store it into the Binary instance and return to the client.

function TExtendedFileTransferService.DownloadFilePart(const FileName: AnsiString; const PartNo: Integer; const PartSize: Integer): Binary;
var
  fs: TFileStream;
  lRemains, lOffset, lBufferSize: integer;
begin
  Result := Binary.Create;
  fs := TFileStream.Create(ServerForm.SharedFolder + '\' +
    {$IFDEF UNICODE}AnsiStringToWideString{$ENDIF}(FileName),
    fmOpenRead + fmShareDenyWrite);
  lOffset := PartSize*(PartNo-1);
  lRemains := fs.Size - lOffset;
  if lRemains > PartSize then lBufferSize := PartSize else lBufferSize := lRemains;
  fs.Seek(lOffset, soFromBeginning);
  Result.CopyFrom(fs, lBufferSize);
  fs.Free;
end;

The client

The TDownloadThread class represents a thread that is responsible for downloading a single file chunk. See the Execute method, which is actually executed on the background thread. Note that we use a separate channel and message for each thread because those components are not multithread aware.

procedure TDownloadThread.Execute;
var
  ch: TROWinInetHTTPChannel;
  msg: TROBinMessage;
  svc: TExtendedFileTransferService_Proxy;
begin
  // Logging here requires GUI access, that's why it is synchronized
  WriteLogSynchronized('Thread #' + IntToStr(fThreadNumber) +
    ': downloading chunk ' + IntToStr(fPartNumber));

  ch := TROWinInetHTTPChannel.Create(nil);
  ch.TargetURL := fURL;
  msg := TROBinMessage.Create;
  svc := TExtendedFileTransferService_Proxy.Create(msg, ch);

  try
    fPart := (svc as IExtendedFileTransferService).DownloadFilePart(
      {$IFDEF UNICODE}WideStringToAnsiString{$ENDIF}(fFileName),
      fPartNumber, fPartSize);
  finally
    ch.Free;
    msg.Free;
  end;

  WriteLogSynchronized('Thread #' + IntToStr(fThreadNumber) +
    ' download finished, writing to disk');
  
  // Accessing the disk file is synchronized as well
  Synchronize(WritePartSynchronized);
  
  WriteLogSynchronized('Thread #' + IntToStr(fThreadNumber) +
    ' completed.');

  fPart.Free;
  fDone := true;
end;

• Examine the bDownload_Click handler, it is responsible for setting up the entire download process. Here is how it works:
  • The number of chunks is calculated depending on the file size and the selected chunk size.
  • The thread pool is formed, the number of threads in the pool is determined by the user's choice.
  • The main thread continuously scans the thread pool to find a free thread to use for downloading the next chunk in the queue until all chunks are processed.
  • After all chunks are processed, the main thread waits for all threads in the pool to finish.