Extended File Transfer sample (.NET)

Overview

This sample shows how the client can download files exposed by the server in chunks using multiple threads.
Notice that this sample is cross-platform compatible with Delphi and Cocoa versions.

How it works

The approach is quite simple, instead of downloading a complete file at once, it is split into small chunks to be downloaded separately one after another or simultaneously using multiple threads and connections. On the client side, those chunks are combined back into a single file.

This sample provides a RemObject server that exposes the content of a folder (see the GetFilesList method in the ExtendedFileTransferService implementation) and a client that can connect to the server to obtain a list of the files available for downloading. The server also provides another method for downloading a chunk of a file called DownloadFilePart. This method has three parameters:

  • FileName (AnsiString) – Name of the file to be downloaded.
  • PartNo (int32) – The chunk part that should be downloaded starting with 1.
  • PartSize (int32) – Size of the chunk that should be downloaded. DownloadFilePart reads the section of the file with offset PartSize*PartNo and returns the data as a Binary stream to the client. When we want to download a file we can specify the size of the chunk (below the Chunk Size combo box you will see the calculated total count of chunks). We can also specify the number of threads we want to use for the downloading operation. Since each partial chunk download is independent, using threads is quite attractive in this scenario.

Getting Started

  • Compile and run the server and client applications.
  • Prepare the server side:
    • Put one or more files into the shared folder (you can easily locate and open this folder when you click on the link label on the server main form). For the best experience, the file size should be several times larger than the chunk size.

  • Prepare the client side:
    • Get the available files on the client side by clicking the GetFiles button.
    • Adjust the chunk size and thread count on the client.
  • Try to download the selected file. At the end of the downloading process, you can open this file to check its integrity and ensure that downloading was successful.

Examine the code

The server

  • See the implementation of the GetFilesList method in the ExtendedFileTransferService class. Notice how the file information is packed into the array of TFileInfo structures, both the structure and the array type are defined in the RODL.
public virtual TFileInfo[] GetFilesList()
{
    DirectoryInfo dirinfo = new DirectoryInfo(fSharedFolder);
    // Get Files ...
    FileInfo[] files = dirinfo.GetFiles("*.*");
    int itemsCount = files.GetLength(0);
    int i;

    // 
    TFileInfo[] result = new TFileInfo[itemsCount];
    for (i = 0; i < itemsCount; i++)
    {
        result[i] = new TFileInfo();
        result[i].TypeName = files[i].Extension;
        result[i].FileName = files[i].Name;
        result[i].Size = (int)files[i].Length;
    }
            
    return result;
}
  • See how the file chunk is requested from the server via the DownloadFilePart method in the ExtendedFileTransferService class. Notice how we can read a chunk of the file, store it into the Binary instance and return back to the client.
public virtual Binary DownloadFilePart(string FileName, int PartNo, int PartSize)
{
    Binary responsedata = new Binary();
    String lLocalFileName = fSharedFolder + @"\" + FileName;

    using (FileStream file = new FileStream(lLocalFileName, FileMode.Open, FileAccess.Read, FileShare.Read))
    {
        Int32 lOffset = PartSize*(PartNo-1);
        Int32 lRemains = ((Int32)(file.Length - lOffset));
        Int32 lBufferSize = (lRemains > PartSize) ? PartSize : lRemains;
        byte[] lBuffer = new byte[lBufferSize];
        file.Position = lOffset;
        file.Read(lBuffer, 0, lBuffer.Length);
        responsedata.Write(lBuffer, lBuffer.Length);
                
        return responsedata;
    }
}

The client

  • See the DownloadPart method on the client side. This is the method that is executed on each background thread we run to download the selected file. Note that we use a separate channel and message for each thread because those components are not multithread aware. 
void DownloadPart(object aState)
{
    DownloadThreadState lState = aState as DownloadThreadState;
    Binary lFilePartData = null;
    DateTime lStartTime = DateTime.Now;

    String lMessage = String.Format("{0}: Downloading chunk#{1} has been started...", lStartTime.ToString("HH:mm:ss:ffff"), lState.ChunkNo);
    this.Invoke(new AddToLogCallback(AddToLog), new object[] { lMessage, false, false });

    using (BinMessage lMsg = new RemObjects.SDK.BinMessage())
    {
        using (IpHttpClientChannel lChannel = new RemObjects.SDK.IpHttpClientChannel())
        {

            lChannel.TargetUrl = this.TargetUrl;
            lChannel.KeepAlive = false;

            IExtendedFileTransferService lFileTransferService =
                CoExtendedFileTransferService.Create(lMsg, lChannel);

            lFilePartData =
                        lFileTransferService.DownloadFilePart(
                            lState.FileName,
                            lState.ChunkNo,
                            lState.ChunkSize);
        }
    }

    byte[] lBuffer = new byte[lFilePartData.Length];
    lFilePartData.Read(lBuffer, lBuffer.Length);

    lock (MyLock)
    {
        lState.FileStream.Position = lState.ChunkSize * (lState.ChunkNo - 1);
        lState.FileStream.Write(lBuffer, 0, lBuffer.Length);
    }
    
    // ...
}
  • Examine the bDownload_Click handler; it is responsible for setting up the entire download process. Here is how it works:
    • The thread pool is limited to the number of threads by the user's choice.
    • The number of chunks is calculated depending on the file size and the selected chunk size.
    • The job state object DownloadThreadState is created for each chunk.
    • All chunk jobs are queued in the .NET thread pool.
private void bDownload_Click(object sender, EventArgs e)
{
    // ...
    System.Threading.ThreadPool.SetMaxThreads(Convert.ToInt32(nudThreadCount.Value), 1000);

    Int32 lChunkCount = this.ChunkCount;

    txtLog.AppendText(String.Format("Start downloading file '{0}'" + Environment.NewLine, lFileName));
    txtLog.AppendText(String.Format("{0} threads in pool:" + Environment.NewLine, nudThreadCount.Value.ToString()));

    for (int i = 1; i <= lChunkCount; i++)
    {
        DownloadThreadState lState = new DownloadThreadState(
                    this.TargetUrl,
                    lFileName,
                    lDownloadDir,
                    writeStream,
                    this.ChunkSize,
                    i);

        System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(DownloadPart), lState);
    }
}